MCP.Market

ab-test-setup skill

by davila7·davila7/claude-code-templates·33k stars·MIT

When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," or "hypothesis." For tracking implementation, see analytics-tracking.

A100/100content scan

Is the ab-test-setup skill safe?

Clean: nothing in its files matched our rules. We read 1 file in the folder on 2026-11-09.

No findings.

Install the ab-test-setup skill

A skill is a folder. Copy it into your agent's skills folder and the agent loads it when the task matches its description. 

git clone --depth 1 https://github.com/davila7/claude-code-templates.git /tmp/claude-code-templates
mkdir -p ~/.claude/skills
cp -r /tmp/claude-code-templates/cli-tool/components/skills/business-marketing/ab-test-setup ~/.claude/skills/ab-test-setup
available in every project

In the Claude apps, zip the folder and upload it from the Skills settings. The folder on GitHub

The instructions your agent would load

SKILL.md as published, without the frontmatter. Read it on GitHub

A/B Test Setup

You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.

Initial Assessment

Before designing a test, understand:

  1. Test Context
  • What are you trying to improve?
  • What change are you considering?
  • What made you want to test this?
  1. Current State
  • Baseline conversion rate?
  • Current traffic volume?
  • Any historical test data?
  1. Constraints
  • Technical implementation complexity?
  • Timeline requirements?
  • Tools available?

Core Principles

1. Start with a Hypothesis

  • Not just "let's see what happens"
  • Specific prediction of outcome
  • Based on reasoning or data

2. Test One Thing

  • Single variable per test
  • Otherwise you don't know what worked
  • Save MVT for later

3. Statistical Rigor

  • Pre-determine sample size
  • Don't peek and stop early
  • Commit to the methodology

4. Measure What Matters

  • Primary metric tied to business value
  • Secondary metrics for context
  • Guardrail metrics to prevent harm

Hypothesis Framework

Structure

Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].

Examples

Weak hypothesis: "Changing the button color might increase clicks."

Strong hypothesis: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."

Good Hypotheses Include

  • Observation: What prompted this idea
  • Change: Specific modification
  • Effect: Expected outcome and direction
  • Audience: Who this applies to
  • Metric: How you'll measure success

Test Types

A/B Test (Split Test)

  • Two versions: Control (A) vs. Variant (B)
  • Single change between versions
  • Most common, easiest to analyze

A/B/n Test

  • Multiple variants (A vs. B vs. C...)
  • Requires more traffic
  • Good for testing several options

Multivariate Test (MVT)

  • Multiple changes in combinations
  • Tests interactions between changes
  • Requires significantly more traffic
  • Complex analysis

Split URL Test

  • Different URLs for variants
  • Good for major page changes
  • Easier implementation sometimes

Sample Size Calculation

Inputs Needed

  1. Baseline conversion rate: Your current rate
  2. Minimum detectable effect (MDE): Smallest change worth detecting
  3. Statistical significance level: Usually 95%
  4. Statistical power: Usually 80%

Quick Reference

Formula Resources

  • Evan Miller's calculator: https://www.evanmiller.org/ab-testing/sample-size.html
  • Optimizely's calculator: https://www.optimizely.com/sample-size-calculator/

Test Duration

Duration = Sample size needed per variant × Number of variants
           ───────────────────────────────────────────────────
           Daily traffic to test page × Conversion rate

Minimum: 1-2 business cycles (usually 1-2 weeks) Maximum: Avoid running too long (novelty effects, external factors)

Metrics Selection

Primary Metric

  • Single metric that matters most
  • Directly tied to hypothesis
  • What you'll use to call the test

Secondary Metrics

  • Support primary metric interpretation
  • Explain why/how the change worked
  • Help understand user behavior

Guardrail Metrics

  • Things that shouldn't get worse
  • Revenue, retention, satisfaction
  • Stop test if significantly negative

Metric Examples by Test Type

Homepage CTA test:

  • Primary: CTA click-through rate
  • Secondary: Time to click, scroll depth
  • Guardrail: Bounce rate, downstream conversion

Pricing page test:

  • Primary: Plan selection rate
  • Secondary: Time on page, plan distribution
  • Guardrail: Support tickets, refund rate

Signup flow test:

  • Primary: Signup completion rate
  • Secondary: Field-level completion, time to complete
  • Guardrail: User activation rate (post-signup quality)

Designing Variants

More skills from davila7/claude-code-templates

  • Aagent-evaluationTesting and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks Use when: agent testing, agent evaluation, benchmark agents, agent reliability, test agent.
  • Aagent-managementCreate, manage, and orchestrate AI agents using the AI Maestro CLI. Use when the user asks to "create agent", "list agents", "delete agent", "hibernate agent", "wake agent", "install plugin", "show agent", "restart agent", or any agent lifecycle management task.
  • Aagent-manager-skillManage multiple local CLI agents via tmux sessions (start/stop/monitor/assign) with cron-friendly scheduling.
  • Aagent-memory-mcpA hybrid memory system that provides persistent, searchable knowledge management for AI agents (Architecture, Patterns, Decisions).
  • Aagent-memory-systemsMemory is the cornerstone of intelligent agents. Without it, every interaction starts from zero. This skill covers the architecture of agent memory: short-term (context window), long-term (vector stores), and the cognitive architectures that organize them. Key insight: Memory isn't just storage - it's retrieval. A million stored facts mean nothing if you can't find the right one. Chunking, embedding, and retrieval strategies determine whether your agent remembers or forgets. The field is fragm
  • Aagent-messagingSend and receive cryptographically signed messages between AI agents using the Agent Messaging Protocol (AMP). Use when the user asks to "send a message to an agent", "check agent inbox", "message another agent", "reply to a message", "notify an agent", or any inter-agent communication task.
  • Aagent-tool-builderTools are how AI agents interact with the world. A well-designed tool is the difference between an agent that works and one that hallucinates, fails silently, or costs 10x more tokens than necessary. This skill covers tool design from schema to error handling. JSON Schema best practices, description writing that actually helps the LLM, validation, and the emerging MCP standard that's becoming the lingua franca for AI tools. Key insight: Tool descriptions are more important than tool implementa
  • Aagents-autogptAutonomous AI agent platform for building and deploying continuous agents. Use when creating visual workflow agents, deploying persistent autonomous agents, or building complex multi-step AI automation systems.
  • Aagents-crewaiMulti-agent orchestration framework for autonomous AI collaboration. Use when building teams of specialized agents working together on complex tasks, when you need role-based agent collaboration with memory, or for production workflows requiring sequential/hierarchical execution. Built without LangChain dependencies for lean, fast execution.
  • Aagents-langchainFramework for building LLM-powered applications with agents, chains, and RAG. Supports multiple providers (OpenAI, Anthropic, Google), 500+ integrations, ReAct agents, tool calling, memory management, and vector store retrieval. Use for building chatbots, question-answering systems, autonomous agents, or RAG applications. Best for rapid prototyping and production deployments.
  • Aagents-llamaindexData framework for building LLM applications with RAG. Specializes in document ingestion (300+ connectors), indexing, and querying. Features vector indices, query engines, agents, and multi-modal support. Use for document Q&A, chatbots, knowledge retrieval, or building RAG pipelines. Best for data-centric LLM applications.
  • Aagile-product-ownerAgile product ownership toolkit for Senior Product Owner including INVEST-compliant user story generation, sprint planning, backlog management, and velocity tracking. Use for story writing, sprint planning, stakeholder communication, and agile ceremonies.

All agent skills → · MCP servers