Experiment design discipline

Experiment design: single "Book a call" hero vs two-button hero on service pages

Status: design drafted, sizing blocked on data. Analytics is not live yet, so there is no baseline enquiry rate and no exposure count. Everything below that depends on those two numbers is stated as a gap or as a conditional read of the sample-size tables, not as an estimate. Per this skill, a stated gap is a complete answer; a guessed baseline is not.

1. Hypothesis (cause, effect, magnitude, mechanism)

Written to Template 1 (conversion rate change). Slots you cannot fill yet are left in brackets on purpose.

> Replacing the two-button service-page hero (Book a call + Ask for an audit) with a single "Book a call" button will increase service-page-visitor-to-enquiry conversion by [magnitude, relative and absolute] (currently [baseline rate, unknown until analytics reports it], target [baseline plus MDE]) by removing the choice between two competing asks at the moment of highest intent, so visitors who would otherwise hesitate between "call" and "audit" take the one action available.

What each part is, and where it stands:

Two things about the metric definition that decide whether this test is honest:

1. The primary metric must count enquiries of both kinds, not "Book a call" clicks. The treatment removes the audit path entirely. If the primary metric were "call bookings," the treatment wins by construction (control's audit requesters are simply not counted). The claim you actually made is "will raise enquiries," so the metric is total enquiries per exposed visitor. If the audit path also exists elsewhere (footer, nav, a separate page), decide before launch whether audit requests from those routes count for both arms; they should, and they should be attributed to the service-page exposure.
2. The mechanism has a real counter-case. Some visitors are not ready for a call and were only willing to ask for an audit. Removing that option may convert them to nothing rather than to a call. That is exactly what makes the hypothesis falsifiable: if total enquiries fall or stay flat while call bookings rise, the mechanism was wrong and the test says so.

Falsifiability check (required by the skill before launch). Write down what result would make you keep the two-button hero. If the answer is "none, we prefer the single button anyway," do not run this test; ship the single button and skip the theater (see section 4).

Primary metric and guardrails (one primary, three to five guardrails)

• Primary: enquiry rate = unique service-page visitors exposed to the hero who submit an enquiry (any type) within [window]. One row per visitor, binary, so a standard proportions test is acceptable here; if you later switch to a per-session or per-pageview denominator, confirm the platform uses a ratio-aware variance estimator.
• Guardrails (proposed, thresholds to be set by you):

1. Qualified-enquiry rate (enquiries that pass whatever your first screen is). Guards against "more calls booked, worse calls."
2. Booked-call show rate. A single-CTA hero that produces more bookings that nobody attends is not a win.
3. Hero CTA click-through rate, both arms. Diagnostic only; tells you whether the effect is happening at the click or at the form.
4. Service-page bounce rate or scroll depth. Guards against the hero reading as pushier and losing the browse-first visitor.
5. Page load and error rate on the hero variant (standard).

Note on guardrail 1 and 2: you will not have a baseline for these either until analytics and the booking tool are wired together. If they cannot be measured by launch, say so in the pre-commitment rather than listing them and never reading them.

2. What traffic it would take

What is known, and what is not

• Baseline enquiry rate: unknown (analytics goes live this month).
• Weekly exposures to the service-page hero: unknown. Note the skill's warning: count exposure (visitors who actually render the hero on a service page), not site-wide sessions.
• Therefore no single sample-size figure can be given yet. What follows is the skill's own Table 1 plus a computed extension, read conditionally on the baseline you will observe.

Sample-size reference (two-sided, alpha 0.05, power 0.80, 50/50 split, totals across both arms)

From the skill's Table 1 (absolute MDE):

Service-page enquiry rates on agency sites are often below the table's lowest row, and at low baselines an absolute MDE stops being a sensible way to state the target (a 1 percent absolute lift on a 2 percent baseline is a 50 percent relative lift). So here is the same math for lower baselines, stated as relative MDE. These are computed with the standard two-proportion formula under the same assumptions as the skill's table (rounded to the nearest hundred; the skill's own table is described as approximate and my recomputation of its cells lands within roughly 10 to 15 percent of the printed figures):

How to turn this into a go/no-go once the numbers exist

1. After analytics has run for at least two full weekly cycles, read: (a) the service-page enquiry rate, (b) weekly unique service-page hero exposures.
2. Pick the MDE you would actually ship for. The skill's guidance: a change that only plausibly moves the metric by a small amount is either not worth a test or not bold enough. Removing one of two CTAs is a bold, visible change, so a +20 to +30 percent relative lift is a reasonable band to be powered for; if you would ship for less than that, the required sample grows fast (inverse square of the effect).
3. Duration = the longer of (required total ÷ weekly exposures) and 14 days, because this is a UI change and the skill sets a 14-day floor for novelty and primacy effects. Hard ceiling six weeks.
4. Worked reading of the table, not a prediction: if the observed baseline turns out to be 2 percent and you power for +30 percent relative, you need about 19,600 exposed visitors in total. That is about 1,400 per week over a 14-day run, or roughly 3,300 per week over six weeks. If weekly service-page exposures are below that, the test cannot answer the question at that MDE; see section 4.

Things that will bias the read on this specific surface

• Weekly cycle. B2B enquiry behaviour differs weekday to weekend; never analyse a partial week.
• Acquisition mix. If paid campaigns start, stop, or change targeting mid-test, the visitor mix changes. Coordinate the test window with whoever runs ads, or pre-register acquisition source as a stratification variable.
• Conversion window. Enquiries on agency sites can lag the first visit by days. Fix the window (for example 7 days from first exposure) before launch and make sure the last cohort has a full window before the analysis date.
• Peeking. If your analytics or testing tool does not do sequential testing, set the analysis date now and do not act on interim reads.

3. Duration and running rules (pre-commitment skeleton)

Fill this in and save it somewhere with a timestamp before the test starts. Blank slots are blank because the data does not exist yet.

• Primary metric: enquiry rate as defined in section 1, window = [ ] days.
• Baseline: [ ] (from analytics, measured over [dates]).
• MDE powered for: [ ] relative ( [ ] absolute ).
• Required total sample: [ ] (from the tables above or your platform's calculator).
• Expected weekly exposures: [ ].
• Duration: [ ] calendar days, not less than 14, not more than 42. Analysis date: [ ].
• Segments: recommend none for the first run. If you must, pre-register at most one (new vs returning visitor, or paid vs organic), and only if the segment alone would hit the sample above.
• Concurrent tests on service pages: [none / list]. Mutex if any.
• Decision rule:
• Lift at or above MDE, guardrails clean: ship the single-button hero.
• Total enquiries down, or a guardrail (qualified rate, show rate) breached: keep the two-button hero.
• Inconclusive: default is do not ship. Then choose one of: kill; re-run bolder (for example single CTA plus a stronger supporting line); or run a bigger version once traffic grows.
• Canary: 5 to 10 percent for a few days first, to confirm the treatment renders, both arms fire the enquiry event, and internal traffic is excluded.

4. Does it belong on the "what NOT to A/B test" list?

Checked against each item in the skill's list:

Verdict: not on the list by nature; provisionally on the list by traffic until analytics proves otherwise. Do not launch it until (a) you have at least two full weeks of baseline and exposure data, (b) the required sample at your chosen MDE fits inside six weeks of that traffic, and (c) you have written down what result would make you keep the two buttons.

Gaps, stated plainly

1. Baseline enquiry rate: not available. Magnitude slot in the hypothesis cannot be filled.
2. Weekly service-page hero exposures: not available. No duration or feasibility verdict is possible.
3. Enquiry attribution window and the definition of "enquiry" across both CTAs: not specified by you; needed before the metric is real.
4. Guardrail baselines (qualified rate, show rate): not available; may need the booking tool joined to analytics.
5. Whether a testing platform exists and whether it supports sequential testing: not stated; determines the peeking rule.

Once 1 and 2 exist, the sizing in section 2 resolves to a single number and a single go/no-go.