openclaw-qa-testing skill

OpenClaw QA Testing

Use this skill for qa-lab / qa-channel work. Repo-local QA only.

Read first

• docs/concepts/qa-e2e-automation.md
• docs/help/testing.md
• docs/channels/qa-channel.md
• qa/README.md
• qa/scenarios/index.yaml
• extensions/qa-lab/src/suite.ts
• extensions/qa-lab/src/character-eval.ts

Model policy

--fast does not mean fast is disabled; fast behavior is source-owned.

• Normal live suite runs rely on QA Lab source- and auth-aware defaults.
• Do not pass --model, --alt-model, or --fast by default. Omitted

and overrides generic default guidance, including when it requires other flags. Add explicit provider/model pins only when execution.config.requiredProvider or requiredModel requires them.

• For scenario-specific runs, the complete execution.summary is authoritative

Default workflow

1. Read the scenario pack and current suite implementation.
2. Decide lane:

• mock/dev: mock-openai
• real validation: live-frontier

1. For a normal live suite, use:

pnpm openclaw qa suite \
  --provider-mode live-frontier \
  --output-dir .artifacts/qa-e2e/run-all-live-frontier-<tag>

1. Watch outputs:

• summary: .artifacts/qa-e2e/run-all-live-frontier-/qa-suite-summary.json
• report: .artifacts/qa-e2e/run-all-live-frontier-/qa-suite-report.md

1. If the user wants to watch the live UI, find the current openclaw-qa listen port and report http://127.0.0.1:.
2. If a scenario fails, fix the product or harness root cause, then rerun the full lane.

OTEL smoke

For local QA-lab OpenTelemetry validation, use:

pnpm qa:otel:smoke

This starts a local OTLP/HTTP trace receiver, runs the otel-trace-smoke scenario through qa-channel, decodes the emitted protobuf spans, and verifies the exported trace names and privacy contract. It does not require Opik, Langfuse, or external collector credentials.

QA credentials and 1Password

• Use op only inside tmux for QA secret lookup in this repo.
• Quick auth check inside tmux:

op account list

Do not ask for secret values in chat or copy them into logs or proof artifacts.

• Resolve approved 1Password item and field references for:
• OPENCLAWQACONVEXSITEURL
• OPENCLAWQACONVEXSECRETMAINTAINER for the maintainer role
• OPENCLAWQACONVEXSECRETCI for the CI role
• If a required reference is missing or stale, ask the maintainer/operator for its current item and field reference.

The lease supplies the Test Server SUT bot, QA-user authorization, and test-group metadata defined by Telegram E2E userbot. Direct Telegram token variables do not replace the required Convex configuration.

• The npm Telegram Docker lane requires Convex credential leasing.

The preflight requires the site URL and matching role secret before the Docker run.

• Load the approved site URL and role-specific secret into the command's environment through the approved secret source.

• With the maintainer configuration loaded, run:

OPENCLAW_NPM_TELEGRAM_CREDENTIAL_SOURCE=convex \
OPENCLAW_NPM_TELEGRAM_CREDENTIAL_ROLE=maintainer \
OPENCLAW_NPM_TELEGRAM_PROVIDER_MODE=mock-openai \
OPENCLAW_NPM_TELEGRAM_PACKAGE_SPEC="openclaw@beta" \
pnpm test:docker:npm-telegram-live

A dispatch does not provision credentials or bypass a credential refusal.

• scripts/e2e/npm-telegram-live-runner.ts reads OPENCLAWNPMTELEGRAMPROVIDERMODE, not OPENCLAWQAPROVIDER_MODE.
• For authorized hosted proof, use the existing manual lane when its qa-live-shared environment has valid Convex CI configuration.

ghx workflow run "NPM Telegram Beta E2E" --repo openclaw/openclaw --ref main \
  -f package_spec=openclaw@YYYY.M.D-beta.N \
  -f package_label=openclaw@YYYY.M.D-beta.N \
  -f provider_mode=mock-openai

• Poll the exact run id from the dispatch URL. gh run view --json artifacts is not supported; list artifacts with:

gh api repos/openclaw/openclaw/actions/runs/<run-id>/artifacts

WhatsApp live credentials

Use this when setting up or replacing Convex kind=whatsapp credentials.

• Treat WhatsApp QA credentials as operator-owned live accounts, not generated fixtures.
• Use two dedicated WhatsApp-capable test numbers: one driver account and one SUT account. Do not use personal numbers or personal OpenClaw WhatsApp accounts in the shared pool.
• Register and link each account manually with WhatsApp or WhatsApp Business, storing Web auth only in isolated local auth dirs outside the repo.
• For group coverage, create a dedicated test group that includes both QA accounts and store its JID as groupJid; otherwise the group mention-gating scenario should be skipped by default and fail when explicitly requested.
• Package the two Baileys auth dirs into base64 .tgz payload fields and add a new active Convex credential row. Prefer adding a fresh row and disabling stale/broken rows over overwriting credentials in place.
• Expected payload fields: driverPhoneE164, sutPhoneE164, driverAuthArchiveBase64, sutAuthArchiveBase64, and optional groupJid.
• Keep credential material out of the repo, logs, PRs, and screenshots. Redact phone numbers unless the operator explicitly asks for local debugging.
• Validate with pnpm openclaw qa whatsapp --credential-source convex --credential-role maintainer --provider-mode mock-openai and preserve artifact paths plus redacted pass/fail summaries.
• If WhatsApp expires or invalidates a linked Web session, relink locally, package fresh auth archives, add a new Convex row, then disable the stale row.

Character evals

Use qa character-eval for style/persona/vibe checks across multiple live models.

pnpm openclaw qa character-eval \
  --output-dir .artifacts/qa-e2e/character-eval-<tag>

only). To exercise openclaw qa/qa-channel from a built dist, build with OPENCLAWBUILDPRIVATE_QA=1 pnpm build (emits dist/plugin-sdk/qa-lab.js, qa-runtime.js, dist/extensions/{qa-lab,qa-channel}) or run via pnpm dev.

• Runs local QA gateway child processes, not Docker.
• Packaged pnpm build omits QA Lab + qa-channel by design (source-checkout

judge, thinking, and fast defaults.

• With no model flags, character eval uses its current source-defined candidate,

or --judge-model ... only to replace the corresponding inventory explicitly.

• Repeat --model provider/model,thinking=[,fast|,no-fast|,fast=]

per-scenario *.yaml files with top-level title, scenario, and optional flow. Never add fenced qa-scenario / qa-flow Markdown files.

• Do not add new examples with separate --model-thinking; keep that flag as legacy compatibility only.
• Report includes judge ranking, run stats, durations, and full transcripts; do not include raw judge replies. Duration is benchmark context, not a grading signal.
• Candidate and judge concurrency default to 16. Use --concurrency and --judge-concurrency to override when local gateways or provider limits need a gentler lane.
• Scenario source is YAML-only under qa/scenarios/: use index.yaml and

• For isolated character/persona evals, write the persona into SOUL.md and blank IDENTITY.md in the scenario flow. Use SOUL.md + IDENTITY.md only when intentionally testing how the normal OpenClaw identity combines with the character.
• Keep prompts natural and task-shaped. The candidate model should receive character setup through SOUL.md, then normal user turns such as chat, workspace help, and small file tasks; do not ask "how would you react?" or tell the model it is in an eval.
• Prefer at least one real task, such as creating or editing a tiny workspace artifact, so the transcript captures character under normal tool use instead of pure roleplay.

Codex CLI model lane

Use model refs shaped like codex-cli/ whenever QA should exercise Codex as a model backend.

Examples:

pnpm openclaw qa suite \
  --provider-mode live-frontier \
  --model codex-cli/<codex-model> \
  --alt-model codex-cli/<codex-model> \
  --scenario <scenario-id> \
  --output-dir .artifacts/qa-e2e/codex-<tag>

pnpm openclaw qa manual \
  --model codex-cli/<codex-model> \
  --message "Reply exactly: CODEX_OK"

relevant secret-backed auth, and gateway child logs before changing scenario assertions.

• Treat the concrete Codex model name as user/config input; do not hardcode it in source, docs examples, or scenarios.
• Live QA preserves CODEXHOME so Codex CLI auth/config works while keeping HOME and OPENCLAWHOME sandboxed.
• Mock QA should scrub CODEX_HOME.
• If Codex returns fallback/auth text every turn, first check CODEX_HOME,

• For model comparison, include codex-cli/ as another candidate in qa character-eval; the report should label it as an opaque model name.