A test result should be read as a range: the interval of effects your data is consistent with. If the whole range sits above zero and the low end is still worth money, ship it. If it straddles zero, you have learned that the effect is small or uncertain, which is useful, but you have not found a winner. The figures below are an illustrative example, not a client result.
This is the testing-and-refinement end of the method we follow. The earlier steps are covered in sample size and duration, and the service itself is described under CRO experiments.
A worked example (illustrative numbers)
A heat-pump quote page is tested for four weeks. The original gets 3,000 visitors and 120 quote requests or calls (4.0%). The variant gets 3,000 visitors and 150 (5.0%). The headline says "+25%".
Here is how to check it yourself:
- Difference in rates: 5.0% − 4.0% = 1.0 percentage point.
- Standard error: the square root of (0.04 × 0.96 ÷ 3,000 + 0.05 × 0.95 ÷ 3,000), which is about 0.0054.
- 95% interval: 1.0 point ± 1.96 × 0.54 points, or roughly −0.05 to +2.05 points.
- Test statistic: 0.010 ÷ 0.0054 is about 1.87, a two-sided p-value of roughly 0.06.
So the variant might be twice as good as the headline suggests, or it might do nothing at all. The interval just touches zero. That is not a failure. It is a precise statement that four weeks of this traffic cannot separate a 25% lift from noise.
Statistical significance versus business significance
A result can be statistically clear and still not worth acting on, or uncertain and still worth a bet. Put a value on each end of the interval.
Suppose (illustratively) that 30% of quote requests on this page become booked jobs and a job is worth $9,000 in revenue. At 750 visitors a week:
- Low end, −0.05 points: about 0.4 fewer requests a week, roughly zero in practice.
- Middle, +1.0 point: about 7.5 more requests, or 2.25 more jobs a week.
- High end, +2.05 points: about 15 more requests, or 4.6 more jobs a week.
If the variant costs nothing extra to run and has no downside on lead quality, shipping it is a reasonable bet even at p ≈ 0.06. If it involves a costly offer, such as a discount that cuts margin, you want the low end of the interval clearly above break-even before you commit.
Four outcomes and what to do with each
| What the interval shows | What it means | Next step |
|---|---|---|
| Entirely above zero, low end worth money | A real, useful win | Ship, then confirm with a holdout period |
| Above zero, but low end too small to matter | Real but minor | Ship if free; otherwise move on |
| Straddles zero, wide | Underpowered | Rerun with a bolder change or a longer window |
| Straddles zero, narrow | Little or no effect | Stop testing this idea |
The narrow-and-flat case is undervalued. It tells you to stop spending time on an idea, which frees the next test slot.
Guard rails: the metric you are not optimizing
Every test needs a second metric you watch for harm. For HVAC lead pages, the usual guard rails are:
- Booked jobs per lead. A page that promises "free quotes in 10 minutes" can raise requests and lower quality.
- Call duration. A rise in very short calls often means confused callers.
- Service-area match. More leads from outside your area are not wins.
Google Ads makes a similar point in its custom experiments help: you can choose whether an auto-applied winner needs statistical significance or only a directional result. For HVAC budgets, we usually want significance plus a clean guard rail before anything is applied automatically.
Multiple variants and multiple metrics
The more comparisons you make, the more likely one looks good by chance. With three variants against an original, three comparisons are made. Dividing the 0.05 threshold by three (about 0.017) is a blunt but honest correction.
The same applies to metrics. If you check calls, texts, quote requests, scroll depth and time on page, one will usually move. Name one primary metric before launch and treat the rest as context.
Writing the result up
A good test write-up fits on one page:
- the hypothesis and why you believed it, linked to the conversion research that produced it
- sample sizes and dates
- the interval, not just the point estimate
- the guard-rail result
- the decision and the next test
Keep these in one folder. After a year, the archive of losing and flat tests is often the most valuable thing you own, because it stops the next agency from rerunning them. For low-volume situations where intervals stay wide, read testing with few calls. If you want us to read a result with you, call, text or email.
Frequently asked questions
What does a 95% confidence interval actually mean?
If you repeated the same test many times, about 95% of the intervals built this way would contain the true difference. In practice, treat it as the range of effects the data cannot rule out.
The variant converted 25% better. Why isn't that a win?
Because with modest traffic the interval around that 25% can stretch from nothing to something large. The headline lift is the middle of the range, and the range is what you are betting on.
What if I tested three variants against the original?
Each extra comparison adds another chance of a false positive. A simple fix is to divide your significance threshold by the number of comparisons, so 0.05 becomes about 0.017 for three.
Sources
- Google Ads Help: Set up a custom experiment (2026-10-23)