pr-test skill

Manual E2E Test

Test a PR/branch end-to-end by building the full platform, interacting via browser and API, capturing screenshots, and reporting results.

Changelog 2.2.0 — auth flow updated for Better Auth (Supabase signup is gone), env-setup gaps closed, a proven Playwright fallback for agent-browser, a billing-test trap that produces false passes, safer process cleanup, and a mock-provider pattern for deterministic $0 testing. Learned on #14206 — see the evidence comment. 2.2.1 — corrects two claims from 2.2.0 that didn't survive live-stack verification (JWKS does not rotate on a frontend restart; the local Postgres port is 5432, not 54322) and hardens the auth setup (explicit password-length/allowlist/rate-limit failure modes, fail-fast on an empty token, password kept out of process args). 2.3.0 — screenshots are posted as GitHub comment attachments (gh pr comment --attach, needs gh >= 2.99.0) instead of being pushed to a test-screenshots/* branch, which accumulated one branch per tested PR and made every old report's images depend on that branch surviving.

Critical Requirements

These are NON-NEGOTIABLE. Every test run MUST satisfy ALL the following:

1. Screenshots at Every Step

• Take a screenshot at EVERY significant test step — not just at the end
• Every test scenario MUST have at least one BEFORE and one AFTER screenshot
• Name screenshots sequentially: {NN}-{action}-{state}.png (e.g., 01-credits-before.png, 02-credits-after.png)
• If a screenshot is missing for a scenario, the test is INCOMPLETE — go back and take it

2. Screenshots MUST Be Posted to PR

• Attach ALL screenshots to the PR comment with gh pr comment --attach (see Step 7). Never push image files to a repo branch — attachments live in GitHub's own asset store and leave nothing behind in the repo
• Post a PR comment with ALL screenshots embedded inline, each with its explanation
• This is NOT optional — every test run MUST end with a PR comment containing screenshots
• If the upload fails, retry. If it still fails, list failed files and require manual drag-and-drop/paste attachment in the PR comment, and mark the run INCOMPLETE

3. State Verification with Before/After Evidence

• For EVERY state-changing operation (API call, user action), capture the state BEFORE and AFTER
• Log the actual API response values (e.g., creditsbefore=100, creditsafter=95)
• Screenshot MUST show the relevant UI state change
• Compare expected vs actual values explicitly — do not just eyeball it

4. Negative Test Cases Are Mandatory

• Test at least ONE negative case per feature (e.g., insufficient credits, invalid input, unauthorized access)
• Verify error messages are user-friendly and accurate
• Verify the system state did NOT change after a rejected operation

5. Test Report Must Include Full Evidence

Each test scenario in the report MUST have:

• Steps: What was done (exact commands or UI actions)
• Expected: What should happen
• Actual: What actually happened
• API Evidence: Before/after API response values for state-changing operations
• Screenshot Evidence: Before/after screenshots with explanations

State Manipulation for Realistic Testing

Billing-test trap — this one produces a false pass, not a visible failure. LLM block cost filters key on the platform-owned credential id. A run made with the test user's own API key bills nothing by design, so a credits before/after assertion silently passes on zero deltas either way. Any test that verifies credit reconciliation MUST use the system credential, not a user-supplied key. Related: Ollama block entries are configured with an explicit $0 run-based cost (BlockCostType.RUN, costamount=0 in blockcost_config.py), not a token-metered one — so pre/post-flight cost deltas are always 0 for them regardless of credential. Never use an Ollama model to test credit reconciliation; pick any hosted model billed through the system credential instead.

When testing features that depend on specific states (rate limits, credits, quotas):

1. Use Redis CLI to set counters directly:

# Find the Redis container
   REDIS_CONTAINER=$(docker ps --format '{{.Names}}' | grep redis | head -1)
   # Set a key with expiry
   docker exec $REDIS_CONTAINER redis-cli SET key value EX ttl
   # Example: Set rate limit counter to near-limit
   docker exec $REDIS_CONTAINER redis-cli SET "rate_limit:user:$PR_TEST_USER_EMAIL" 99 EX 3600
   # Example: Check current value
   docker exec $REDIS_CONTAINER redis-cli GET "rate_limit:user:$PR_TEST_USER_EMAIL"

1. Use API calls to check before/after state:

# BEFORE: Record current state
   BEFORE=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/credits | jq '.credits')
   echo "Credits BEFORE: $BEFORE"

   # Perform the action...

   # AFTER: Record new state and compare
   AFTER=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/credits | jq '.credits')
   echo "Credits AFTER: $AFTER"
   echo "Delta: $(( BEFORE - AFTER ))"

1. Take screenshots BEFORE and AFTER state changes — the UI must reflect the backend state change

1. Never rely on mocked/injected browser state — always use real backend state. Do NOT use agent-browser eval to fake UI state. The backend must be the source of truth.

1. Use direct DB queries when needed:

# Query via Supabase's PostgREST or docker exec into the DB
   docker exec supabase-db psql -U supabase_admin -d postgres -c "SELECT credits FROM user_credits WHERE user_id = '...';"

1. After every API test, verify the state change actually persisted:

# Example: After a credits purchase, verify DB matches API
   API_CREDITS=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/credits | jq '.credits')
   DB_CREDITS=$(docker exec supabase-db psql -U supabase_admin -d postgres -t -c "SELECT credits FROM user_credits WHERE user_id = '...';" | tr -d ' ')
   [ "$API_CREDITS" = "$DB_CREDITS" ] && echo "CONSISTENT" || echo "MISMATCH: API=$API_CREDITS DB=$DB_CREDITS"

Arguments

• $ARGUMENTS — worktree path (e.g. $REPO_ROOT) or PR number
• If --fix flag is present, auto-fix bugs found and push fixes (like pr-address loop)

Step 0: Resolve the target

# If argument is a PR number, find its worktree
gh pr view {N} --json headRefName --jq '.headRefName'
# If argument is a path, use it directly

Determine:

• REPOROOT — the root repo directory: git -C "$WORKTREEPATH" worktree list | head -1 | awk '{print $1}' (or git rev-parse --show-toplevel if not a worktree)
• WORKTREE_PATH — the worktree directory
• PLATFORMDIR — $WORKTREEPATH/autogpt_platform
• BACKENDDIR — $PLATFORMDIR/backend
• FRONTENDDIR — $PLATFORMDIR/frontend
• PR_NUMBER — the PR number (from gh pr list --head $(git branch --show-current))
• PR_TITLE — the PR title, slugified (e.g. "Add copilot permissions" → "add-copilot-permissions")
• RESULTSDIR — $REPOROOT/test-results/PR-{PR_NUMBER}-{slugified-title}

Create the results directory:

PR_NUMBER=$(cd $WORKTREE_PATH && gh pr list --head $(git branch --show-current) --repo Significant-Gravitas/AutoGPT --json number --jq '.[0].number')
PR_TITLE=$(cd $WORKTREE_PATH && gh pr list --head $(git branch --show-current) --repo Significant-Gravitas/AutoGPT --json title --jq '.[0].title' | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | sed 's/^-//;s/-$//' | head -c 50)
RESULTS_DIR="$REPO_ROOT/test-results/PR-${PR_NUMBER}-${PR_TITLE}"
mkdir -p $RESULTS_DIR

Test user credentials — required to log into the UI or call authenticated APIs.

NEVER hardcode these in this SKILL, a PR comment, a screenshot, or any committed file. Sources, in priority order:

1. Env vars (CI / preconfigured local shell): $PRTESTUSEREMAIL + $PRTESTUSERPASSWORD. If both are set, use them.
2. Interactive prompt (everything else, including dev-preview runs): if the env vars are not set, ASK the user at the start of the run — e.g. "I need test-user credentials for this run; paste the email and password." Hold them in shell vars only for the duration of the run. Do not echo, log, or write them to disk.

Acquire the variables — env first, prompt if missing — and only then lock them in:

# 1. Prefer env vars (CI / preconfigured shell). Prompt only for the
#    specific var that is unset so an already-exported credential is
#    not overwritten by the prompt when only the other one is missing.
if [ -z "${PR_TEST_USER_EMAIL:-}" ] || [ -z "${PR_TEST_USER_PASSWORD:-}" ]; then
  echo "Test user credentials required for this run."
  if [ -z "${PR_TEST_USER_EMAIL:-}" ]; then
    read -r -p   "Email:    " PR_TEST_USER_EMAIL
  fi
  if [ -z "${PR_TEST_USER_PASSWORD:-}" ]; then
    read -r -s -p "Password: " PR_TEST_USER_PASSWORD
    echo
  fi
  export PR_TEST_USER_EMAIL PR_TEST_USER_PASSWORD
fi

# 2. Lock them in — fail loudly if either is STILL unset (e.g. the user
#    pressed Enter on an empty prompt). The error message names the var so
#    the agent / operator knows what to fix.
: "${PR_TEST_USER_EMAIL:?PR_TEST_USER_EMAIL is empty after env+prompt — supply a value before re-running}"
: "${PR_TEST_USER_PASSWORD:?PR_TEST_USER_PASSWORD is empty after env+prompt — supply a value before re-running}"

For local docker-compose runs, a fresh dev user is created on first call to the signup snippet below. For dev-preview runs, the test user lives in the project's hosted auth backend — ask the user for the current valid credentials each session (the previously-shared test@test.com test account was disabled on 2026-05-23 after its credentials leaked into this very SKILL — do NOT re-introduce a default). PRTESTUSERPASSWORD should always be a throwaway/test-only credential, never a real account's password** — the auth requests in 3h go over plain HTTP on localhost:3000 for local runs, which has no transport encryption. If a dev-preview run's target isn't on localhost, confirm it's HTTPS before sending credentials to it.

Step 1: Understand the PR

Before testing, understand what changed:

cd $WORKTREE_PATH

# Read PR description to understand the WHY
gh pr view {N} --json body --jq '.body'

git log --oneline dev..HEAD | head -20
git diff dev --stat

Read the PR description (Why / What / How) and changed files to understand:

1. Why does this PR exist? What problem does it solve?
2. What feature/fix does this PR implement?
3. How does it work? What's the approach?
4. What components are affected? (backend, frontend, copilot, executor, etc.)
5. What are the key user-facing behaviors to test?

Step 2: Write test scenarios

Based on the PR analysis, write a test plan to $RESULTS_DIR/test-plan.md:

# Test Plan: PR #{N} — {title}

## Scenarios
1. [Scenario name] — [what to verify]
2. ...

## API Tests (if applicable)
1. [Endpoint] — [expected behavior]
   - Before state: [what to check before]
   - After state: [what to verify changed]

## UI Tests (if applicable)
1. [Page/component] — [interaction to test]
   - Screenshot before: [what to capture]
   - Screenshot after: [what to capture]

## Negative Tests (REQUIRED — at least one per feature)
1. [What should NOT happen] — [how to trigger it]
   - Expected error: [what error message/code]
   - State unchanged: [what to verify did NOT change]

Be critical — include edge cases, error paths, and security checks. Every scenario MUST specify what screenshots to take and what state to verify.

Step 3.0: Claim the testing lock (coordinate parallel agents)

Multiple worktrees share the same host — Docker infra (postgres, redis, clamav), app ports (3000/8006/…), and the test user. Two agents running /pr-test concurrently will corrupt each other's state (connection-pool exhaustion, port binds failing silently, cross-test assertions). Use the root-worktree lock file to take turns.

Lock file contract

Path (always the root worktree so all siblings see it): $REPO_ROOT/.ign.testing.lock

Body (one key=value per line):

holder=<pr-XXXXX-purpose>
pid=<pid-or-"self">
started=<iso8601>
heartbeat=<iso8601, updated every ~2 min>
worktree=<full path>
branch=<branch name>
intent=<one-line description + rough duration>

Claim

LOCK=$REPO_ROOT/.ign.testing.lock
NOW=$(date -u +%Y-%m-%dT%H:%MZ)
STALE_AFTER_MIN=5

if [ -f "$LOCK" ]; then
  HB=$(grep '^heartbeat=' "$LOCK" | cut -d= -f2)
  HB_EPOCH=$(date -j -f '%Y-%m-%dT%H:%MZ' "$HB" +%s 2>/dev/null || date -d "$HB" +%s 2>/dev/null || echo 0)
  AGE_MIN=$(( ( $(date -u +%s) - HB_EPOCH ) / 60 ))
  if [ "$AGE_MIN" -gt "$STALE_AFTER_MIN" ]; then
    echo "WARN: stale lock (${AGE_MIN}m old) — reclaiming"
    cat "$LOCK" | sed 's/^/  stale: /'
  else
    echo "Another agent holds the lock:"; cat "$LOCK"
    echo "Wait until released or resume after $((STALE_AFTER_MIN - AGE_MIN))m."
    exit 1
  fi
fi

cat > "$LOCK" <<EOF
holder=pr-${PR_NUMBER}-e2e
pid=self
started=$NOW
heartbeat=$NOW
worktree=$WORKTREE_PATH
branch=$(cd $WORKTREE_PATH && git branch --show-current)
intent=E2E test PR #${PR_NUMBER}, native mode, ~60min
EOF
echo "Lock claimed"