Run one segment at a time or you'll never know what worked: how to measure cold email campaign performance

Sunny Ray · Jul 8, 2026 · 5 min read

The open-rate number you're optimizing is costing you the replies you actually want. And the second most common measurement mistake is running five segments at once and calling the blend a result.

On a July 6 kickoff for a franchise client's podcast campaign, with 91 mailboxes warming and about four to five days to launch, I told their team two things that surprised them. We weren't tracking opens. And we weren't launching all their segments at once.

How do you measure cold email campaign performance? Count replies and booked calls, not opens. Run one segment at a time so the lift belongs to someone. Change one thing per test. Everything else is noise you paid for.

The line from our kickoff

"We're not tracking the open rate, by the way, because that reduces the open rate. Ironically, there's all kinds of weird stuff like that that we need to be cognizant about."

That's the short version. An open-tracking pixel is a tiny image with a unique link. Spam filters look for exactly that kind of thing in cold mail. So the act of measuring opens makes fewer emails land in the inbox, which lowers the opens you were trying to measure. You pay for the number with the outcome.

And the number itself has gotten worse. A post I'd saved in April made the point well: opens are among the worst cold email metrics now, because pixels get preloaded by mail clients and privacy features, so an "open" may just be a machine fetching an image. A metric that's both costly and fake isn't a metric.

One public split test

I don't want to oversell this with one data point, but it's worth knowing. A practitioner published a split test of a single send: 4.8 percent replies with the pixel off, 0.8 percent with it on. Sample sizes weren't stated, so treat it as a story, not a law. But the direction matches what we see across our own campaigns, and it matches the logic.

Open tracking is part of a bigger set of first-touch rules we give every client. The same week I gave a different founder the list: under 80 words, no URLs, no calendar links, no tracking pixels in the first email. Little things move you into spam:

"So just little things like URL would get you in spam."

What we measure instead

Three numbers, in order of how much I care.

Booked calls per segment. This is the only number that connects to revenue. For a show-led campaign, a booked call is a pre-interview on the calendar.

Positive replies per segment. Someone who says yes, maybe, or tell me more. Count them separately from out-of-office and "remove me," which you should also count, because a spike in negatives tells you something about the list or the message.

Bounce rate per sending domain. Not a performance metric, a health metric. If it climbs, stop and fix it before anything else.

That's it. Sends are a cost, not a result. And a saved post from June made a point I agree with: reply rates have been falling for two years, and sending more into less interest only speeds up the decay. The answer to a weak reply rate is a better segment or a better offer, not more volume.

Run segments serially so you can read the lift

This was the second thing on that kickoff call, and it's the one most teams skip. We told the client we'd run campaigns serially, not in parallel, to isolate lift by segment.

Their primary list was senior, experienced people in their industry. The secondary list was people who'd left the industry. Both were reasonable bets. If you launch both on the same day with the same copy, and replies come in, you can't tell which list is carrying the campaign. Worse, you'll keep both running because the blended number looks fine, and one of them might be dead weight burning mailbox reputation.

So: launch the primary segment. Let it run long enough to read. Then launch the secondary as its own campaign, with its own numbers. Now each segment has a reply rate and a booking rate you can compare.

It's slower in week one and much faster by week six, because you stop feeding segments that don't work.

Change one variable at a time

The same discipline applies inside a segment. On that call the team was choosing between two names for their new show. Instead of debating it, the plan was to A/B test the two names at launch and let the reply data decide.

That's a clean test. Same list, same copy, same mailboxes, one variable. If you change the subject line, the opener and the offer at once, you'll get a different result and no idea why.

A simple log helps: date, segment, what changed, replies, bookings. Each campaign builds on the last one's lesson instead of starting from scratch.

The franchise team asked a fair follow-up: doesn't serial testing waste time? Only if you think of week one as the result. With 91 mailboxes warming and four or five days to launch, the first segment was going out into a clean system either way. The only question was whether the second segment would teach us something or blur the first.

What a good dashboard looks like

One row per segment. Columns for sends, positive replies, booked calls, bounce rate, and the one variable being tested. No open rate column at all.

If that looks thin, good. Thin dashboards force the right conversation: which segment books calls, and what do we test next. Fat dashboards full of opens and clicks let everyone feel busy while the calendar stays empty.

If you want to see the sending layer that runs these tests for you, see the sending layer.

Frequently asked questions