Holdout tests: measuring whether HVAC ads create jobs or only claim them

Three kinds of holdout

Geographic holdout

Pick service areas that behave alike (similar housing, similar call history) and split them into two groups. Ads keep running in one group and pause in the other. Compare the change in calls in each group against its own baseline.

This suits companies that serve several towns or a large metro, because you need enough comparable areas to split. A one-town business usually cannot run it.

Time-based holdout

Pause a campaign for a planned number of weeks, then restart it, ideally more than once. This is the switchback idea from our low-volume testing piece, applied to spend instead of pages. It works well for brand search, where the question is simple: if we stop paying for our own name, do the calls still arrive through the organic listing?

Audience holdout

For remarketing, exclude a random share of the audience from ads. Google Ads custom experiments can split traffic by cookie so each person sees only one version. The same page recommends audience lists of at least 10,000 users when using cookie splits with audiences, since smaller lists may give less accurate results. Many local HVAC remarketing lists are smaller than that, which is itself a useful finding.

The arithmetic of a geo holdout (illustrative)

Two groups of service areas are tracked for four weeks before and four weeks during the test.

A naive reading of the paused group alone says the ads were worth 20 calls. But demand in the "on" group rose by 20 over the same weeks, perhaps because the weather turned. The fair comparison is the difference in differences: (+20) − (−20) = 40 calls over four weeks attributable to the ads in the paused areas.

Then put it in money. If (illustratively) the paused campaigns would have spent $4,000 over those weeks and 30% of qualified calls book, 40 calls become about 12 jobs, or about $333 in ad spend per incremental job. Compare that with your margin per job and with what the attribution report claimed.

Planning around Google Ads experiment limits

If you run a campaign-level experiment instead of a manual holdout, Google's custom experiments page sets some boundaries worth knowing before you plan:

• only one experiment can run at a time, with up to five scheduled
• eligible campaigns must use Smart Bidding and include exact or phrase match keywords
• campaigns with a shared budget are not eligible
• the original campaign must stay active until the experiment's scheduled end
• changing the original or the experiment during a run can make results harder to read

That last point is the one HVAC accounts break most often, because someone raises budgets on the first cold morning. Agree a freeze for the test window.

Common mistakes

• Comparing against last year only. Weather and pricing change; use a concurrent control group.
• Too few areas. Two towns are not a sample. Aim for several per group, matched on history.
• Leaky geography. Location targeting is approximate, and commuters search from work. Leave a buffer between groups where you can.
• Counting form fills only. Most HVAC demand calls. Measure qualified calls, as set out in our call tracking guide.

When the answer is "not incremental"

If a campaign turns out to add little, that is good news: you have found budget to move. Reallocate it to the campaigns that passed, and retest once a year, because competitors and search results change. To check whether seasonal swings distorted your holdout, read seasonality controls. To plan one with us, see pricing and then call, text or email.