A test needs enough visitors to tell a real difference from noise, and the number depends mostly on how small a difference you want to catch. Work it out before launch, round up to full weeks, and write the stopping date down. Everything in this article that looks like a client figure is an illustrative example, not a result.
This is the first step of the testing loop we use, which follows the Conv.ca method: research and analysis, strategy development, implementation, then testing and refinement. Sample size belongs to strategy development, because it decides which tests are even worth building. See how we apply it in CRO experiments.
The four inputs that set your sample size
You need four numbers, and only one of them comes from your analytics. The rest are decisions.
- Baseline conversion rate. The share of visitors who call, text or request a quote today. Pull it from at least four recent full weeks.
- Minimum detectable effect (MDE). The smallest change you would act on. A 5% relative lift may be real, but if it would not change any business decision, there is no point paying to detect it.
- Significance level. The false-positive risk you accept, usually 5% (two-sided).
- Power. The chance of catching a real effect of your chosen size, usually 80%.
If your conversion tracking is not trustworthy yet, stop here. A sample size calculation on a broken baseline produces a confident wrong answer. Our measurement setup work exists for that reason.
The formula, in plain terms
For comparing two conversion rates, a standard approximation for visitors needed in each version is:
n ≈ (z₁ + z₂)² × [p₁(1 − p₁) + p₂(1 − p₂)] ÷ (p₁ − p₂)²
Here p₁ is the baseline rate, p₂ is the rate you hope the variant reaches, z₁ is 1.96 for 5% two-sided significance, and z₂ is 0.84 for 80% power. Those two add to 2.80, and 2.80 squared is 7.84, so the multiplier at the front is simply 7.84.
The key insight is in the bottom line. The gap between the two rates is squared, so halving the lift you want to detect roughly quadruples the visitors you need.
A worked example (illustrative numbers)
Suppose a furnace-replacement landing page converts at 4% and you want to detect a lift to 5%, a 25% relative improvement.
| Step | Calculation | Result |
|---|---|---|
| Variance terms | 0.04 × 0.96 + 0.05 × 0.95 | 0.0859 |
| Gap squared | (0.05 − 0.04)² | 0.0001 |
| Per version | 7.84 × 0.0859 ÷ 0.0001 | about 6,735 |
| Total | two versions | about 13,470 |
If that page gets 3,000 eligible sessions a week, the test needs about 4.5 weeks, which you round up to 5 full weeks.
Now say you only care about a bigger change, from 4% to 6%. The variance terms become 0.0948, the gap squared is 0.0004, and each version needs about 1,858 visitors, roughly 3,716 in total. At 3,000 sessions a week that is under two weeks, but we would still run two full weeks so that every weekday appears twice.
The lesson for an owner: small sites should test bold changes (a new offer, a different page structure, a shorter path to the phone number), not button colours.
Turning visitors into calendar time
Divide the total sample by the weekly traffic that actually reaches the tested page, then round up to whole weeks. Three adjustments matter for HVAC:
- Count only eligible traffic. If the test runs on the boiler page, sitewide sessions are irrelevant.
- Avoid weather breaks. A cold snap halfway through a two-week test changes who arrives. If one hits, extend rather than stop early. Our article on seasonality controls covers this in depth.
- Set a ceiling. If the maths says 14 weeks, the test is too small to run as designed. Pick a larger MDE or a different metric, as explained in testing with few calls.
How long is too long
Google's Search Central testing guidance says to run a test only as long as needed for a reliable result, then remove the test elements as soon as possible. It also asks for temporary (302) redirects when sending visitors to variant URLs, canonical links from variants back to the original, and no special version for Googlebot, which would count as cloaking.
In practice that means fixing the end date in advance, shipping the winner promptly and deleting the losing variant. A variant left running for months at 90% of traffic is not a test any more.
Common mistakes
- Peeking and stopping. Checking every morning and stopping on the first significant reading raises false positives sharply.
- Changing the page mid-test. Any edit resets the comparison.
- Using sessions when the decision is per person. If one homeowner visits five times before calling, count users or the result looks more certain than it is.
- Testing on a tracking setup nobody has audited. Fix measurement first.
Before you launch: a one-page checklist
- Baseline rate from at least four full weeks
- MDE agreed in writing, with the business reason
- Per-version and total sample, with the arithmetic shown
- Start date, end date and the rule for weather interruptions
- Primary metric and one guard-rail metric (for example, booked jobs per call)
When you have results, read them with an interval rather than a yes-or-no verdict; our guide to reading test results shows how. If you want a second pair of eyes on a test plan, call, text or email and we will check the maths with you. More method pieces are in the newsroom.
Frequently asked questions
Can I stop a test as soon as the tool says it is significant?
Not if you planned a fixed sample. Checking repeatedly and stopping at the first good reading inflates the false-positive rate well above the 5% you think you are accepting. Decide the sample in advance, or use a tool built for sequential monitoring.
Why run full weeks?
HVAC call behaviour differs by weekday. Homeowners often research on weekends and call on Monday. A test that covers only part of a week can mistake a calendar pattern for a winning page.
Does a long test hurt my Google rankings?
Google's Search Central guidance says to run tests only as long as needed for a reliable result and then remove the test elements. Using temporary redirects and canonical links back to the original page keeps the original indexed.