Holdout and incrementality testing for HVAC advertising

Who it is for

HVAC and heating companies spending meaningfully on advertising who suspect some of it is not pulling its weight. The usual suspects:

• Brand search. You pay for clicks on your own name. Would those homeowners have called through the organic listing anyway?
• Remarketing. Ads follow people who already visited. Were they going to call regardless?
• Lead platforms and directories. Some claim leads that also reach you directly.
• Broad display or video campaigns whose reported conversions seem high for the spend.

Generic searches like "furnace repair" from people who have never heard of you are more likely to be incremental, but they are worth testing too once the obvious candidates are done.

What is included

1. Candidate selection. We rank campaigns by spend and by how likely they are to be taking credit.
2. Design choice. Geographic, time-based or audience holdout, depending on your footprint and volume.
3. Matching. For geographic designs, service areas are grouped by call history and housing mix, with buffers where targeting overlaps.
4. Plan. Sample, duration, freeze rules and the exact metric, written before launch.
5. Execution support. Location exclusions, pause schedules or Google Ads custom experiments where the design fits its rules.
6. Readout. Difference-in-differences estimate, interval, and cost per incremental job.

Rules we plan around

Google's experiment tools have limits that shape designs: one experiment at a time, Smart Bidding and exact or phrase match keywords required, no shared budgets, and the original campaign active until the end. For audience splits by cookie, Google recommends lists of at least 10,000 users. Many local HVAC lists are smaller, so manual designs are often the honest choice.

A worked example of the decision (illustrative)

Suppose brand search costs $1,200 a month and Google Ads credits it with 40 calls. A time-based holdout pauses it for alternating weeks over two months. During paused weeks, total qualified calls barely change, because searchers click your organic listing or Business Profile instead. The measured incremental effect is a few calls a month, not 40.

The decision then depends on context. If competitors bid on your name and take the top spot when you pause, some brand spend may still be worth keeping as defence. If they don't, most of the $1,200 can move to campaigns that reach people who do not know you yet. Either way, the decision is made on measured effect rather than on credited conversions.

Timeline

A holdout usually needs a baseline period and a test period of similar length, both in full weeks, followed by a short wait for late-booking jobs to arrive. Seasonal timing matters: we avoid windows where demand is changing fast, as explained in seasonality controls. Your proposal sets the dates.

How results are measured

• Incremental qualified calls and booked jobs in the treated group versus the control
• Cost per incremental job, compared with your margin per job
• Comparison with attribution. How far the attribution report's claimed conversions differ from the measured incremental effect

Google's attribution overview describes the models GA4 uses to divide credit. A holdout gives you a second, independent reading to set beside them.

Mistakes to avoid

• Before-and-after with no control group. The weather will decide the answer.
• Too few areas per group. Two towns are anecdotes.
• Raising budgets mid-test because the first cold morning arrived.
• Reading the result too early. Replacement jobs often book weeks after the first call.
• Treating one test as permanent truth. Competitors and search results change; retest yearly.

Plans are on pricing, and real results from our HVAC work are on results. Call, text or email to discuss which campaign to test first.