MCP.Market

add-product-e2e-eval skill

by paperclipai·paperclipai/paperclip·114k stars·MIT

Add or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.

A100/100content scan

Is the add-product-e2e-eval skill safe?

Clean: nothing in its files matched our rules. We read 1 file in the folder on 2026-10-12.

No findings.

Install the add-product-e2e-eval skill

A skill is a folder. Copy it into your agent's skills folder and the agent loads it when the task matches its description. 

git clone --depth 1 https://github.com/paperclipai/paperclip.git /tmp/paperclip
mkdir -p ~/.claude/skills
cp -r /tmp/paperclip/.agents/skills/add-product-e2e-eval ~/.claude/skills/add-product-e2e-eval
available in every project

In the Claude apps, zip the folder and upload it from the Skills settings. The folder on GitHub

The instructions your agent would load

SKILL.md as published, without the frontmatter. Read it on GitHub

Add a Product E2E Eval

Use this skill for Product E2E Evals: real Chromium, Paperclip server, database, runner, provider, and optionally Daytona. Everyday Workflows are in this family even when their packaged results are imported into Evalbook. Runner protocol cases against the mock control plane belong in add-runner-eval.

Locate the repository using PAPERCLIP_ROOT when supplied, or git rev-parse --show-toplevel from a checkout. From outside Git, inspect workspace roots such as ~/paperclipai/paperclip; verify the selected root contains tests/runner-e2e and packages/paperclip-runner. Run commands from that repository root. The copied skill may live outside the checkout. Read doc/evals.md, then the authoritative tests/runner-e2e/README.md, FIXTURES.md, SECURITY.md, and EVERYDAY-WORKFLOWS.md for the selected area. Inspect the nearest existing catalog entry, case, harness flow, matcher, evidence writer, and report test before changing anything. Keep the user journey on production browser/API surfaces; do not add private runner hooks or direct fixture database writes.

Define one bounded workflow with a clear user outcome, durable state assertions, and independent evidence. Declare profile, environment, expected provider turns, timeout, cleanup, screenshots, artifact checks, and billing scope. Keep credentials and secrets out of catalog data, screenshots, fixture metadata, logs, and tracked files. Daytona images must be immutable digest references. Use the existing result validator, failure classifier, screenshot policy, and Product E2E report pipeline rather than duplicating them.

Register the workflow in tests/runner-e2e/catalog.ts and the relevant fixture-registry.ts paths. Everyday cases and actions live in everyday-cases.ts and everyday-flow.ts; match the existing suite's structure. Wire its profile/environment/case IDs through existing selector and matcher tables, and add report/catalog coverage tests where the surrounding suite does so. Calibrate new grading assertions with a valid outcome and a plausible wrong outcome; missing evidence must not produce a pass. Confirm discovery before running it:

pnpm test:e2e:runner -- --list --suite <suite-name>
pnpm test:e2e:runner -- --list --suite everyday-workflows

--all intentionally excludes the manual everyday-workflows suite and other explicit-only cells. Use the exact suite or execution ID for those. The Product E2E generator is pnpm test:e2e:runner:report; an Everyday Workflows result may also be imported into Evalbook with its canonical importer, but it remains a Product E2E run and should use its packaged dashboard/report first.

Run credential-free checks first:

pnpm test:e2e:runner:typecheck
pnpm test:e2e:runner:unit
pnpm test:e2e:runner -- --list

For an authorized live check, use the smallest explicit local --id selector. Select Daytona only with the configured immutable image and credentials. A full --all campaign is paid and is for the governed workflow. Verify results from the packaged attempt evidence and dashboard; passing model-authored tests cannot override the independent oracle.

Classify a completed wrong workflow as product or model/provider behavior as the assertions warrant. Startup, transport, and timeout symptoms require evidence-based attribution: they may indicate a Paperclip/Runner product bug, provider behavior, or infrastructure. Preserve the observed failure and cause separately, keep the existing machine grade/classifier unchanged, and retain partial attempts, retries, source SHA, catalog/definition digest, model/profile, environment, grader version, timing, tokens, runtime estimates, and cost coverage. Do not claim qualification from a partial/manual selection.

Update README.md, FIXTURES.md, SECURITY.md, or EVERYDAY-WORKFLOWS.md when their authoritative contract changes, and link from doc/evals.md. Do not recreate Evalbook HTML or publish raw trusted traces, videos, archives, databases, workspaces, credentials, SVG, provider session IDs, or hidden reasoning. Public sanitized fixture conversation, marked screenshots, and allowlisted structured evidence are expected when the existing publisher permits them.

More skills from paperclipai/paperclip

  • ALLM Wiki MaintainerUse the LLM Wiki plugin tools to maintain a cited local company wiki.
  • Aadd-runner-evalAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.
  • Aagent-browserDrive a real browser to inspect or interact with a web page or app — navigate, take screenshots, read console and network, fill simple forms — for verification tasks, not unattended automation.
  • AagentmailUse your assigned AgentMail inbox to read email tasks, explicitly send or reply, and check delivery. Provided automatically by your inbox assignment.
  • Acheck-prCheck a GitHub, GitLab, or Perforce PR/MR/CL for review comments, failing checks, and PR-body gaps. Use when asked to inspect, fix, or prepare a change for submission.
  • Acompany-creatorCreate agent company packages that conform to agentcompanies/v1. Use when asked to create a company, scaffold an agent team, hire agents, or turn a repo/skills collection into a company package.
  • AcomplainAutonomously yeet a brief, unstructured complaint about agent tooling or doing agent work. Preserve the raw voice. Submit proactively without permission, preview, or announcement, then continue the primary task.
  • Acontent-calendarPlan a weekly editorial calendar by mapping company goals to publishable topics, owners, status, and verification notes.
  • Acreate-agent-adapterCreate or modify Paperclip agent adapters across server, UI, and CLI surfaces. Use when adding support for a new CLI agent, API agent, custom process, or adapter package.
  • Acreate-caseCreate and kick off a Paperclip case from a user request, with workflow instructions and an initial agent.
  • Acreate-external-bot-integrationBuild and qualify external bots such as Dot in Paperclip. Use for two-way work, invitation, identity, wakeup, Runner lifecycle, native tools, recovery, and Cloud delivery. CLI adapters and app-only connectors use their existing skills.
  • Acreate-issue-interaction-uiAdd a new Paperclip issue-thread interaction kind end-to-end. Use when repo work introduces or extends interaction cards like request_confirmation, checkbox confirmations, ask_user_questions, or suggest_tasks.

All agent skills → · MCP servers