CRO SKILL
A/B test design
Design an A/B test that can be trusted, then run testing as a programme: hypothesis framework, sample-size table, metrics selection, allocation, the peeking problem, ICE prioritisation and a winners’ playbook.
No key, no account needed
Claude Code
Cursor
CodexWindsurfany agent that reads SKILL.md
npx skills add addition-labs/skills --skill ab-testingRun this in a terminal inside your project folder. Claude Code, Cursor and Codex then read the skill from where it lands.
- Step 01Open a terminal in your project folder
- Step 02Paste the command and press Enter
- Step 03Pick the agents to install to
- Step 04Ask the agent for the job; it reads the skill
ab-testingaddition-labs/skillsMIT licencev2.0.0updated 2026-09-05
The first half is one test done properly: a hypothesis with a because-clause, one variable, a pre-determined sample size from a quick-reference table (a 3% baseline needs about 47,000 per variant to detect a 10% lift), primary plus secondary plus guardrail metrics, and no peeking.
The second half is the programme: hypothesis sources, ICE scoring, velocity targets (four to eight experiments a month, 20 to 30% win rate), a playbook entry format for winners, and a weekly-biweekly-monthly-quarterly cadence.
It is the general-purpose testing skill. For the harder cases (ratio metrics, interaction effects, sequential testing, what not to test) the experiment-design skill beside it goes deeper.
When to use it
The phrases that trigger it
Say any of these and an agent that has the skill installed will usually load it; you can also call it by name. The phrases are the skill’s own, from its description.
- A/B test
- split test
- experiment
- statistical significance
- how long should I run this test
- experiment velocity
- ICE score
- experimentation program
What’s inside
The playbook, section by section
The hypothesis structure
Because [observation], we believe [change] will cause [outcome] for [audience]; we will know when [metric].
Sample-size quick reference
Baselines of 1, 3, 5 and 10% against lifts of 10, 20 and 50%, with links to Evan Miller’s and Optimizely’s calculators.
Three metric tiers
One primary that calls the test, secondaries that explain it, guardrails that stop it.
The peeking problem
Stopping early on an interim look inflates false positives; pre-commit to the sample size and trust the process.
The experiment loop and ICE
Generate, prioritise (impact, confidence, ease, averaged), run, analyse, promote winners to a playbook, repeat.
A playbook entry
Date, hypothesis, sample, result with confidence interval and p-value, guardrails, segment deltas, why it worked, the reusable pattern, where else to apply it.
Reference files the agent loads on demand
2 files in the skill folder, read only when an instruction calls for one.
sample-size-guide.md- How to size a test and set its duration from the baseline rate and the effect you want to detect.
test-templates.md- Templates for planning, documenting and analysing an experiment.
See it in action
One real run, in full
The request on the left went to the skill as installed. On the right is everything it returned, unedited; scroll inside the box to read it all.
We want to A/B test the hero headline on https://addition-labs.com/services/seo. Honest situation: we do not have traffic numbers yet (analytics goes live this month) and enquiries are a handful a month. Design the test if it can be designed, or tell us what it would take and what to do instead.
A/B Test Assessment: Hero headline, addition-labs.com/services/seo
Initial assessment
No product marketing context file was found (.agents/product-marketing.md, .claude/product-marketing.md, or product-marketing-context.md), so this works only from your request and the live page.
Test context. Change under consideration: the hero headline. The page today (fetched 2026-09-11) renders the H1 as a small label plus a statement:
> Search Engine Optimization > Pages people want to read, that search engines can read too.
Subhead: "We build pages a buyer enjoys using, structured so Google and AI systems read them the same way. Every page with one clear job." Primary hero CTA: "Send me AI & SEO Audit" (a free search and AI check form). You have not yet said *why* you want to change the headline or what the alternative would be, so there is no hypothesis to test yet (see Open questions).
Current state. Baseline conversion rate: unknown. Traffic to this page: unknown (analytics goes live this month). Enquiries: "a handful a month". Whether that handful is site-wide or from this page alone is not stated; if it is site-wide, the number for this page is smaller.
Constraints. Testing tool: not stated. Timeline: not stated.
Verdict: this test cannot be designed yet
A valid A/B test needs a pre-committed sample size, and the sample size depends on two numbers you do not have: the baseline conversion rate and the page's traffic. But the enquiry count alone is enough to settle the question, because a conversion test is limited by conversions, not by visitors.
What it takes to detect a headline effect (95% confidence, 80% power, two variants). These are the skill's standard inputs, calculated with the standard two-proportion formula; they hold at any baseline rate in the 1 to 10 percent range:
| Effect you want to detect | Enquiries needed in the control arm alone | Total enquiries across both arms, roughly |
|---|---|---|
| +10% (a typical headline win) | about 1,500 | about 3,000 |
| +20% | about 400 | about 900 |
| +50% | about 75 | about 190 |
| +100% (a doubling) | about 22 | about 65 |
Against "a handful a month". Take a handful as 3 to 8 enquiries a month for the whole page, so 1.5 to 4 per arm. The smallest detectable effect in the table, a doubling, needs 22 in the control arm: 6 to 15 months at that rate. A +50% effect needs 75: 19 to 50 months. The skill's maximum test duration is 4 to 8 weeks (novelty effects wear off, external factors intervene, and here the whole business changes in the meantime). Nothing on this table fits inside that window, and the realistic headline effect is the top row, not the bottom.
The same conclusion, expressed as the traffic you would need. To finish a two-arm test on the enquiry metric in 8 weeks:
| Baseline enquiry rate | Detect +50% | Detect +100% |
|---|---|---|
| 1% | about 280 visitors a day to this page | about 85 a day |
| 3% | about 90 a day | about 27 a day |
| 5% | about 55 a day | about 16 a day |
Once analytics is live you can check which row you are on. If the page is not near those daily numbers, the answer stays no.
One more reason not to run it now. With analytics launching this month there is no verified tracking on either variant, and "tracking verified" is a pre-launch gate in this skill, not an optional step. A test started before the tracking has been watched for a few weeks would be testing the instrumentation as much as the headline.
What to do instead (in order)
The sample size guide lists the options when requirements are too high: increase the MDE, reduce variants, combine traffic, test upstream, or don't test and decide from qualitative data. Applied to your situation:
1. Establish the baseline first (this month). Let analytics run for at least 4 weeks, ideally 8, before touching the headline. Record for this page: daily visitors, hero CTA clicks ("Send me AI & SEO Audit"), audit form starts, audit form submits, and enquiries. Those numbers decide everything below. Changing the headline while the baseline is being measured contaminates the baseline.
2. Move the metric upstream. Enquiries will never carry a headline test at this volume. The hero CTA click and the audit form submit are the same visitor decision, several steps earlier, and happen many times more often than an enquiry. A headline test measured on "hero CTA click rate" is the only version of this test that could ever reach sample size on this page. It is a weaker proxy (a click is not a client), so keep enquiries as a guardrail metric and check the direction agrees.
3. Combine traffic, if the hero pattern is shared. The page links to eleven sub-service pages (SEO Audit, SEO Content, Technical SEO, Link Building, and so on). If those use the same hero structure and the headline change is a pattern (for example, "buyer-outcome statement" vs "capability statement") rather than SEO-specific wording, run the same variant across all of them and pool the traffic. This is the skill's "combine traffic across similar pages" option and is usually the biggest single multiplier a low-traffic site has.
4. Decide qualitatively for now, and write the decision down. Until the numbers exist, choose the headline the way the skill's hypothesis framework demands: from an observation, not a preference. Sources that work at zero traffic: sales call notes (what prospects say they came for), the three "sentences we hear" already on the page, competitor hero lines, and five-second tests with people in the target role (show the hero for five seconds, ask "what does this company do and for whom"). Ship the better one as a plain change, log it in a hypothesis bank with the date, and treat the before/after in analytics as a signal, not a result.
5. Do not run a "small" test to see what happens. A test that cannot reach sample size will end in one of two ways: inconclusive (learned nothing, spent weeks) or a false winner from noise (learned something wrong). The skill's peeking rule exists for exactly this case.
The test, ready for when it becomes possible
Pre-filled from what is known; blanks are yours.
# A/B Test: SEO service page hero headline
## Hypothesis
Because [observation: what do prospects say, or what does the baseline show?],
we believe [the new headline, exact wording]
will cause [more hero CTA clicks / audit requests]
for [first-time visitors to /services/seo, from search / from ads / all].
We'll know this is true when hero CTA click rate rises by at least [MDE]%.
## Test design
| Element | Details |
|---|---|
| Test type | A/B (one variant; A/B/n raises the sample requirement about 1.5x) |
| What varies | The headline statement only. The "Search Engine Optimization" label, subhead, CTA and everything below stay identical. |
| Primary metric | Hero CTA click rate (clicks on "Send me AI & SEO Audit" / page views) |
| Secondary | Audit form submit rate; scroll depth to the "Hard problems" block |
| Guardrails | Enquiries per week (should not fall); bounce rate |
| Baseline | [from analytics, after 4 to 8 weeks] |
| MDE | [smallest lift worth the change; below 20% is unrealistic at this traffic] |
| Sample per variant | [from the skill's table at that baseline and MDE] |
| Duration | Sample x 2 / daily page visitors = [days]; minimum 2 full weeks (B2B weekday pattern), maximum 8 |
| Allocation | 50/50, sticky per visitor |
| Tool | [not stated; client-side tools such as PostHog or VWO fit a static marketing page; watch for flicker on the H1] |
## Success criteria
Winner: primary metric up by at least MDE at 95% confidence, guardrails flat.
Loser: primary metric down significantly; keep control, record why.
Inconclusive: sample reached, no significant difference; keep control, next test must be bolder.Open questions
The skill asks these before designing a test. Your message answers some; the rest are still open:
- Current conversion rate: unknown until analytics runs. What is the conversion you care about here, the enquiry or the audit request?
- Traffic to this page: unknown. When analytics is live, what is the daily visitor count to /services/seo, and where does it come from (organic, ads, direct)?
- What headline change are you considering, and what observation prompted it? Without this there is no hypothesis, only a preference.
- What is the smallest improvement that would justify changing the headline? (Anything under +20% will not be detectable here for a long time.)
- What tools do you have for testing? Is there a testing tool in place, or only analytics?
- Have you changed or tested this hero before, and what happened?
- Is "a handful of enquiries a month" for the whole site or for this page?
What this run could not do
It had no baseline conversion rate, page traffic, proposed variant, minimum detectable effect, testing tool or prior test history, so it gave the verdict conditionally and listed those six as open questions. "A handful a month" was read as 3 to 8 and labelled an interpretation.
Method behind it
Where we would differ, and why
Our A/B testing guide reproduces the sample-size arithmetic with worked store examples and shows why most ecommerce sites below a few thousand orders a month cannot detect the lifts they hope for. The skill designs the test; the guide tells you whether your traffic can carry it.
The analysis section reads 95% significance as "less than 5% chance the result is random". That is the common misreading of a p-value; the correct statement is the probability of data at least this extreme if there were no effect. The experiment-design skill states it more carefully.
The sample-size table in the skill body and the tables in its own reference file disagree by about 2.5x for the same inputs (1% baseline, 10% lift: 150k per variant in the body, 380k in the reference). The body matches the standard two-sided 95%/80% calculation; the reference does not say what inputs it used. Use the body, or a calculator.
Velocity targets of four to eight experiments a month assume traffic most stores do not have. Our compounding rule is one well-researched change a week; the number that matters is the research behind each, not the count.
SKILL.md
The file the agent reads
Written by Corey Haines and published under the MIT licence. Read the full file (1,664 words) on GitHub, at the commit we reviewed: coreyhaines31/marketingskills/skills/ab-testing/SKILL.md. The reference files are in the same folder.
Questions
Questions people ask before installing
What if I do not have enough traffic?
The skill’s table will show you: at a 1% baseline, a 10% lift needs 150,000 visitors per variant. Its advice for that case is a bolder change or more traffic; ours is to fix the obvious without a test and reserve testing for changes that could double the rate.
Client-side or server-side?
Client-side is fast and can flicker; server-side needs a developer and does not. The skill lists both with tools.
How is ICE different from PXL?
ICE averages impact, confidence and ease. PXL, which we use, weights page importance, ease and learning value. Either works if you re-score monthly.