perf-compare skill

perf-compare: one way to measure performance

Every agent that tunes speed or stability measures with this recipe, so results are comparable across agents and noise is not reported as a win. Sentry is the store: org significant-gravitas, API https://us.sentry.io, tracing at 100%. scripts/perf_compare.py (Python 3.8+, standard library only) runs the queries and the statistics.

S=$(git rev-parse --show-toplevel)/.claude/skills/perf-compare/scripts/perf_compare.py
python3 "$S" --help

When to use it

• Before claiming a change made something faster or slower, on a preview or after merge.
• After merge, to check a win holds on dev and then prod (release against release).
• To check a change did not regress other transactions (run without --transaction).
• For any numeric field: span.duration (default), web vitals on builder pageloads (measurements.lcp), LLM token counts (genai.usage.inputtokens), or custom attributes (tags[name,number]).

It is not for debugging one slow request (open the trace) or for error triage.

Token

The script reads SENTRYAUTHTOKEN from the environment and nowhere else. It needs a read-only token with exactly org:read, project:read and event:read: either a user auth token (Settings > Account > API > Auth Tokens) or an internal integration with Read on Organization, Project and Issue & Event. Organization auth tokens (sntrys_...) carry only org:ci and get HTTP 403 on these endpoints. A missing token exits with code 5 and these instructions. --print-queries needs no token.

Where things are in Sentry

Check a selector has data before you measure:

search_events  projectSlug=<project>  dataset=spans  query="is_transaction:true"
               fields=["environment","release","count()"]  sort="-count()"  period="7d"

If release comes back null for a project, select by time window instead of release. If autogpt-server-previews has no pr- events, that preview's backend is reporting into autogpt-server as app:dev-behave:cloud, mixed with dev traffic, and cannot be measured separately.

Choosing baseline and candidate

Both sides take KEY=VALUE pairs: project=, env=, release=, query= (extra Sentry filter), and time as window=START/END (repeatable), start=/end=, or period=24h. Shared defaults come from --project, --env, --query and --period/--start/--end/--window. A side's own project=, env= or query= wins over the shared value (env= with nothing after it clears it). Any time key on a side (window=, start=/end= or period=) replaces all of the shared time flags for that side. Times are UTC ISO 8601; without a time anywhere, both sides use the last 7 days.

1. After merge: release against release (dev or prod). Same project and environment, previous release against new release, in windows of equal length that cover the same hours of the day: the 24 hours before the deploy against the 24 hours after, or 7 days against 7 days for quiet routes. Always filter by release when there is one. Old releases keep sending events for days (open tabs, Vercel skew protection), so a time window alone mixes releases.

python3 "$S" --project builder --env app:prod-behave:cloud \
  --baseline release=OLD_SHA window=2026-10-10T14:00:00Z/2026-10-11T14:00:00Z \
  --candidate release=NEW_SHA window=2026-10-11T14:00:00Z/2026-10-12T14:00:00Z

2. On a preview: candidate preview against a control preview. Deploy the change with !deploy, and deploy a control (the dev commit the change branches from, with no changes) the same way. Each gets its own namespace and an empty branch database, so the two are alike. Drive the same load at both, either together in one window or interleaved A B B A (control, candidate, candidate, control; about 10 minutes each) so drift cancels:

python3 "$S" --project autogpt-server-previews --transaction '/api/store/agents' \
  --baseline env=pr-CONTROL window=T0/T1 window=T3/T4 \
  --candidate env=pr-CANDIDATE window=T1/T2 window=T2/T3

Measuring the candidate preview against app:dev-behave:cloud at dev HEAD in the same window is cheaper but weaker: dev has a full database, real traffic and different pods. Use it as a rough read, not as proof.

3. No release tag: before against after. Equal windows either side of the deploy. Other deploys inside those windows also change things, so say so in your report.

4. A/A check before trusting a win. Compare a selector with itself over two windows, for example the same release yesterday and today. It should say no detectable change. If it does not, your windows differ too much (hour of day, weekday, traffic mix), and so will your real comparison.

How much data

• The script gives no p50 or p95 verdict unless each side has --min-count samples (default 200) from --min-traces distinct traces (default 30). At 200, p95 rests on 10 values above it. p99 is shown but never judged. The verdict cell says which minimum was missed. The failure-rate test works from Sentry's event counts instead, and needs --min-count events per side.
• It samples up to --sample-cap values per side per transaction (default 1000, at most 20000), paging 100 at a time. A cap below --min-count is refused.
• For a claim about one transaction, name it with --transaction and raise --sample-cap to 2000–5000. Naming it also shrinks the multiple-comparison correction (see below).
• Correlated spans count once: the bootstrap resamples whole traces, and the trace minimum stops a few busy traces (a page polling an endpoint) from passing as a big sample. The table shows samples and distinct traces. The effective sample is closer to the trace count than the span count. In A/A tests on clustered data, resampling spans one at a time called 38–44% of identical pairs a change, against 2–5% when resampling by trace.
• Drive at least 1000 requests per side per transaction you care about. Wait a few minutes after load stops before measuring, so Sentry has the last events.

Driving load on a preview

A preview's backend is https://pr--server.agpt.co (REST under /api, health at /health), and its frontend is https://autogpt-pr-.vercel.app.

The k6 workload is in autogpt_platform/backend/load-tests/; its README.md covers the journeys, the profiles and every setting. Run it on your own machine: ./k6.sh runs k6 in Docker, so nothing else needs installing (the first run builds the image, about two minutes), and a local run costs nothing beyond the load on the target. Do not run it on Grafana Cloud k6. It bills every virtual-user hour, so ./k6.sh refuses cloud runs unless they are explicitly opted into, and the README reserves them for rare runs the team has agreed to.

tests/workload.js drives the user-facing mix at production's weights: copilot reads and streamed turns, account reads, store browsing, library reads and graph runs watched over the WebSocket. Every journey is an open-model scenario at a fixed arrival rate, so a slow build queues instead of sending fewer requests. LOAD_PROFILE picks smoke (a minute), baseline (one-minute warm-up, ten minutes steady: the comparison run) or stress (three times baseline):

cd autogpt_platform/backend/load-tests
export K6_ENVIRONMENT=PREVIEW PR_NUMBER=<N>
LOAD_PROFILE=baseline ./k6.sh run tests/workload.js

• Authentication needs no setup on a preview; graph runs and chat sessions need a Stripe test-mode key. setup() signs up throwaway @previews.agpt.co accounts named for the engine their copilot turns run on (loadtest-dummy--, loadtest-scripted--, and loadtest-- for the real model), seeds each with graphs, runs, chat sessions and a hired expert, and finds whatever store listings and expert templates the preview has. Previews paywall new accounts, so with STRIPETESTSECRETKEY unset the graph journey skips and the copilot reads have no session to open; with it, setup() subscribes each account in Stripe test mode first. The summary's coverage says which journeys ran in full. Dev has no throwaway sign-up, so it needs LOADTEST_ACCOUNTS (a JSON array of {"email", "password"}).
• Every run writes results/.json with the UTC window of its steady phase (window.steadystartedat / window.steadyendedat, after the warm-up and before teardown), the target and its Sentry selector, the rates it sent, what each journey covered, and the client-side numbers: per endpoint p50/p95, copilot time to first token and turn time, graph run time and WebSocket event latency. Use the steady window for the window= arguments above: it is what k6's thresholds judged.
• scripts/ab.sh runs the baseline profile at both previews interleaved (A B B A) with the same seed and prints the perf-compare command for the steady windows it produced. Runs that covered different journeys did not send the same load, so then it prints their coverage and no command. scripts/diff-summaries.js puts two summaries side by side; it is a quick read, not a verdict.
• Copilot turns run on the engine each account's address names. On a preview, a loadtest-dummy- account's turns run on the dummy engine and a loadtest-scripted- account's on the SDK engine against the preview's scripted model, at no model cost; everyone else on the preview keeps the real model (backend copilot/loadtest.py). setup() confirms both with a probe turn before it sends any, a probe that fails fails the run, and LOADTESTCOPILOTTURNS=off sends no turn. Real-model turns are opt-in (LOADTESTCOPILOTREALTURNS, 1 to 10, about $0.07 each). The backend records copilot.turn.ttftms, copilot.turn.queuewaitms and copilot.turn.durationms on its copilot.turn transaction, and copilot.loadtest names a load-test account's engine, so compare with --transaction copilot.turn --field 'tags[copilot.turn.ttftms,number]', adding copilot.loadtest:dummy to the query for one engine or !has:copilot.loadtest for people's turns only. tests/copilot-real-model.js sends a capped number of turns at a real model, opt-in, and prints what it cost.
• The repository's evals (backend/copilot/eval, backend/copilot/capabilities/eval, poetry run expert-style-eval, poetry run brain-dump-eval) are offline quality and cost harnesses. They send no traffic to a deploy.
• After merge, real traffic on dev and prod is the load.

Reading the verdict

The markdown output (default) or --json reports two things for each transaction.

• Sentry aggregates over every matching event: count, p50, p95, p99 and failure rate. These are exact for the windows.
• A verdict for p50 and p95:
• The interval is a bootstrap CI of candidate minus baseline from the raw samples, resampling whole traces.
• It is Bonferroni-adjusted so that every interval in the run holds together at --confidence (95%). Two transactions means four intervals at 98.75% each. The plain 95% interval is in the JSON as ci.
• Resamples are raised automatically until each tail of the adjusted interval holds 20 draws, up to 50000. For example, 4000 becomes 8000 for five transactions. At the 50000 cap a run still goes ahead with as few as 10 draws per tail. A run that would leave fewer than 10, because too many transactions are compared, is refused before it starts (exit 2). The JSON reports the number used as settings.resamples.
• faster or slower needs the adjusted interval to exclude zero, and both the sample change and Sentry's all-events change to be at least --min-effect (5%) in that direction. Anything else is no detectable change. Below --min-count samples or --min-traces traces it is not enough data.
• The transaction takes its worst percentile: slower if either is slower, else faster if either is faster.
• Failure rate: Fisher's exact test on the failure counts behind Sentry's failure_rate(). It is exact because a normal approximation is wrong with a handful of failures: 0 of 1000 against 4 of 1000 passes a z-test, but Fisher gives p = 0.12.
• The significance level is Bonferroni-adjusted over the K transactions that have --min-count events on both sides. That is one test per transaction, not two.
• higher needs significance and a rise of at least --min-failure-delta (0.1 percentage points). It counts as a regression even when the transaction got faster, because requests that fail fast look like a speedup.
• The test treats every event as independent, and --min-traces does not apply to it. When failures come from a few traces (one broken page retrying), it overstates significance. Check sample_traces and the failing traces in Sentry before acting on a failure-rate result alone.

no detectable change means "not distinguishable from noise with this much data". It does not mean "no change". Check the interval widths and trace counts before concluding anything.

A win counts only when all of these hold:

• you named the transaction with --transaction before measuring;
• the verdict is faster;
• the failure rate is not higher;
• it holds again on a second, independent window or run.

A run over the top transactions is for spotting regressions and leads, not for claiming wins.

Exit codes

With --json, the fields to branch on are:

• overall.verdict, overall.exit_code and overall.reasons;
• per transaction in transactions[]:
• verdict, note (why there is not enough data), regression;
• p50/p95: baseline, candidate, diff, rel, sentryrel, ci, ciadjusted, verdict;
• failurerate: baseline, candidate, diff, pvalue, verdict;
• baseline/candidate: count, p50, p95, p99, failurerate, samplen, sample_traces;
• settings, holding the values actually used: samplecap, resamples, comparisons, failurecomparisons, adjustedconfidence, mincount, mintraces, mineffect, seed.

Errors in --json mode print {"error": {"type", "message"}, "exit_code"}, where type is usage, auth, api or internal.

The exact queries

All requests are GET https://us.sentry.io/api/0/organizations/significant-gravitas/events/ with dataset=spans and project=. The id comes from GET /api/0/projects/significant-gravitas//. start and end (UTC) span the side's windows; the query's timestamp filter pins each window exactly. F is --field.

1. Aggregate per transaction, one request per side:

• field=transaction, count(), p50(F), p95(F), p99(F), failure_rate()
• sort=-count(), per_page=100

1. Raw sample, per side per compared transaction:

• field=F, trace, id
• sort=id, per_page=100, paged with the cursor from the Link header (rel="next"; results="true") until --sample-cap rows.
• Span ids are random, so the first N rows by id are a uniform random sample of the windows. Sorting by -timestamp instead returns the latest burst, often a few traces.

Both use this query string:

is_transaction:true environment:"ENV" release:"SHA" [has:F] [(EXTRA QUERY)]
  [transaction:"T" | (transaction:"A" OR transaction:"B")]
  timestamp:>=2026-10-11T14:00:00Z timestamp:<2026-10-12T14:00:00Z

Several windows become (timestamp:>=S1 timestamp:=S2 timestamp:<E2). has:F appears for any field other than span.duration.

python3 "$S" ... --print-queries prints the exact REST URL and MCP arguments for an invocation, without a token.

With only the Sentry MCP, use searchevents with dataset: "spans", or searchtraces without dataset:

• Pass projectSlug, the same query, fields and sort.
• Pass a period long enough to cover the windows, such as "50h"; the timestamp filter does the exact cut. The argument is period, not statsPeriod, and an unknown key silently falls back to 14 days.
• Pass limit ≤ 100.
• Confirm the "Executed Search" block in the reply shows your query unchanged.
• The MCP cannot page. For more than 100 samples, split each window into sub-windows, get count() for each, and take the first rows by id from each in proportion to its count.

Field names checked against live data

Checked through the S