openclaw-testing skill

OpenClaw Testing

Proof policy

ready PR. CI runs broad suites; never delay PR for pre-PR suite runs.

• PR CI = default broad proof. Change builds + focused tests pass → open

remote capacity is unavailable, and local execution is unqualified, push/open a draft PR with the exact missing build and capacity gap recorded. Defer that build to exact-head CI. Require its success before marking ready or merging. This changes proof timing only, not publication authority, source trust, credentials, owner approval, or merge gates. Automatic CI skips drafts. For a trusted canonical branch, use the existing ci.yml manual dispatch with targetref set to the full pushed head SHA (ghx workflow run ci.yml --repo openclaw/openclaw --ref main -f targetref=). Verify the selected checkout SHA and successful build-artifacts job cover the required build. CI's build:ci-artifacts omits full declaration emission. Keep uncovered proof pending. A skipped job or workflow-ref SHA is not build proof. Link the run in the PR, then use normal ready-PR CI and landing gates. Untrusted source keeps its secretless route.

• Build unavailable before publication: when publication is already authorized,

pnpm test --maxWorkers=1, targeted typecheck, lint, format. Prefer local proof for trusted, bounded work when dependencies and current host capacity are qualified. Local execution is optional: a focused command can still load a large compiler graph or expand to broad lanes.

• Before PR, prove narrowly: touched tests with

may cover expensive checks, a busy or unqualified local host, another OS/device, live providers/channels, long E2E, or heavy benchmarks. Acquire only when the command is ready and capacity is granted promptly. If queued, stop only the task-owned allocation; use PR CI or a safe local scope. Keep uncovered platform-specific proof pending. Never force heavy work onto a busy machine or chain providers. Use the smallest profile proven for the selected workload.

• Choose remote proof deliberately. Use PR CI for broad proof; Crabbox/Testbox

CI failure classification before attributing red.

• Never rerun or re-push just for green. Fix real failures; read available

changed inputs, failures, or named unresolved risk.

• Prove changed contract, complete required gates, finish. Broaden/repeat only for

boundary guide and test-audit.

• Follow scoped AGENTS.md; authoring/reviewing tests: use

Select the proof

• Select proof for touched contract; reuse valid proof (rerun for changed inputs or missing coverage); finish workflow's required gates within user/host limits; report unrun checks and gaps. Prove user-visible behavior through real flow when feasible; external APIs need live contract proof; isolated mock-Gateway harness = valid channel boundary proof. Docs-only: docs sanity and git diff --check.

repository CI build that covers the changed contract. Record uncovered packaging/platform/declaration requirements. Required full package or release builds still need their matching proof.

• Runtime defect: reproduce real entry point; rerun it + relevant siblings.
• Trusted source: pnpm changed:lanes --json, focused tests + targeted typecheck/lint/format; pnpm check:changed = optional smart local check.
• Public SDK/plugin contract: representative consumers, not all-plugin sweep.
• Build output, lazy imports, package boundaries: include pnpm build or the

• Workflow: git diff --check + pnpm check:workflows.
• Docs only: relevant docs/link/format checks + git diff --check; no runtime tests.

Crabbox, OpenClaw remote setup, package/Docker proof, release CI, plugin release matrix, Docker authoring, channel skill/ Control UI E2E/iOS E2E.

• Read on demand: local commands,

gaps. Prove user-visible behavior through the real flow when feasible; external APIs require live contract proof. Read the scoped channel/platform guide for live-channel or native-app work. Telegram claims require Test Server userbot proof with Convex-leased credentials. Platform claims require real-device evidence; macOS permission proof requires a stable signed app.

• An isolated mock-Gateway harness is valid channel boundary proof; state live

• Release proof: preserve candidate/Tooling SHAs; never grants publication authority.

Source and state boundaries

Use secretless fork CI or sanitized direct AWS through Crabbox; never credential-hydrated Testbox. Credentialed execution: maintainer approval after review required; never hydrate untrusted lease.

• Never execute untrusted contributor/fork code, wrappers, config locally.

data without explicit per-task approval. Preserve unrelated processes and shared dependency installs.

• Isolated state + free port. Never restart, edit, or test operator Gateway/real

values or run serially; checks can run Vitest too. Repository wrappers only, never raw Vitest. Prepared linked worktrees: node scripts/run-vitest.mjs --maxWorkers=1 avoids pnpm reconciliation; PR CI runs the broad checks.

• Concurrent test/check commands: separate OPENCLAWVITESTFSMODULECACHE_PATH

CI failures

state over stale PR rollups.

• PR CI fails → pnpm test:ci-failed ; read pre-existing/new/unknown classification.
• Fix new failures; investigate unknown → pnpm test:ci-failed --run until selected tests pass locally → push once.
• --run enables OPENCLAWNODETESTPLANCONTINUEONFAILURE=1: collect ordinary failures across selected configs, while infrastructure failures still stop; use the same opt-in for manual focused reruns.
• Pre-existing failures: failure-table landing rule. Matching signatures ≠ job-name-only attribution.
• Never gh run rerun to chase green; never local full suite for CI repair.
• Bind diagnosis to exact SHA + job. Fetch failed logs once; prefer exact run/job

cancellation. Snapshots pass macOS/fail CI: reproduce Linux/Node bytes before regenerating.

• Distinguish product, harness, infrastructure, credentials, superseded-run

Baseline and expected-failure exceptions

Narrow baseline, snapshot, ignore, and expected-failure exceptions are implementer decisions only when each entry is verified against source and the PR records its reason (for example, tool-owned metadata in an upgrade snapshot while user data stays exact). Never mask a real regression or user-data change. Blanket or unverified exceptions need approval. Exact shrink-only ratchet updates are maintenance. A PR may add up to 400 net test lines without a record. Above that, add a config/test-lines-allocations/.txt record that covers the excess and state the reason in the squash body. Required regression tests no longer need ledger records (#197519, #197602). See the cost budget.

For module mocks, follow the fixture contract. When an isolation annotation retires an existing baseline entry, commit the annotation and corresponding baseline removal together. Inspect the full baseline diff because --prune scans the whole checkout. Verify the updated baseline with a guard run without --prune.

Test failure policy

identify owner; fix product, fixture, shared-state, ordering bugs.

• Failures = defects. Bounded reproduction attempt (same shard order first);

evidence, uncertainty in PR; continue under normal CI/review gates.

• Proven repair: add regression; cite another owner's fix when applicable.
• Safe fix cannot be established/completed: record original failure, attempts,

Passing replay ≠ proven fix.

• Unresolved failure alone neither blocks landing nor requires extra approval.

mocks, altered baselines.

• Never hide failures with retries, longer timeouts, weaker assertions, broader