The first time I recorded my podcast after my summer break, I made a rookie mistake. Instead of saying, welcome to the retail media breakfast club, I said, welcome to Ecommerce Braintrust, the podcast I hosted and co-hosted for about 7 years while I ran my agency Bobsled Marketing and after I sold it to Acadia.
Fortunately my podcast editor picked it up and whisked it out. Old habits die hard, I guess. Its probably also because I still listen to that podcast today, now that its hosted by my former colleagues Julie Spear and Jordan Ripley, because these guys are in the trenches every day doing the things that I just talk about.
A recent episode about measurement grabbed me, because it really pushes on a creeping question that a lot of brands are asking today:
If every platform that we're advertising on says that it is winning, why don't our business results reflect that?
We add up all of these ad-attributed sales from the various platforms that we're advertising on, but somehow it ends up being much bigger than the actual sales that we brought in? Hmm....
So on this episode of the eCommerce Braintrust Podcast, the hosts, Julie Spear and Jordan Ripley, brought in two people who have to answer that question for a living: Sally Kazin, Acadia's head of analytics, and Ross Walker, their director of retail media.
Here are my personal highlights and takeaways.
Your advertisers can’t wait weeks for audiences.
GrowthLoop’s Composable Commerce Media solution helps media teams turn first-party data into high-value audiences, launch campaigns faster, and prove what’s working across every channel.
Learn how retail media leaders at Costco, Fanatics, and Gopuff use GrowthLoop to create highly segmented audiences and deliver stronger results for their brand partners.
A world of estimation
Sally talks about what's different for brands today versus three years ago.
She says that we've transitioned from a world of observation to a world of estimation, where platform conversions are modeled. But she says that the math, in many cases, simply hasn't caught up.
Three years ago, we were living in a more trackable and more investment efficient digital landscape, she says. "Today, we're trapped in the silo paradox. Brands are spending millions across Meta, Google, Amazon and others, and every one of those platforms is acting as its own judge. They're heavily incentivized to claim credit for every conversion they can possibly touch."
When you aggregate the revenue reported by those platform dashboards, it often exceeds what's actually hitting a brand's bank account. Sally calls this engine inflation: The gap between the dashboard numbers and the ground truth.
"You could end up celebrating platform success, but in the context of a lagging business, combined with record CPMs, [that] results in significantly wasted budget," she says.
My POV: every retailer is doing exactly what Sally describes. They claim every conversion they plausibly can. But hold that thought, because I don't think the retailers are the villains of this story.
Away with MTA
So what's the fix? Spoiler alert: Sally says multi-touch attribution ain't it.
The promise of MTA was beautiful, and we all wanted to believe it, Sally says. But today's reality means we aren't seeing a full picture anymore.
Between privacy restrictions and fragmentation, we're seeing less and less of that actual customer journey.
"It's like you're navigating a new city with a map that only shows every few streets, and filling in the gaps with guesswork and gut feelings," she says.
As a result, Acadia made the call to move away from MTA, she said. Instead, the agency's analytics practice attempts to triangulate the truth with a three-pillar framework:
- Modern media mix modeling (MMM). This helps with the strategy. She says this removes platform bias, and that shows what's actually moving results. Specifically, she calls out a Bayesian model that analyzes total spend versus total sales.
- Incrementality testing. This is the 'reality check,' where you can prove causation. Synthetic controls or lift tests are two options she calls out here.
- Platform optimization. The engines themselves. "This is where it gets tactical," she says. Insights are fed back into the bidding engines, so media buyers are optimizing for actual business growth.
This final part is where platform metrics like ROAS and CPA are still relevant, as long as they in service of maximizing platform performance and the business.
This 3-part approach makes sense as it doesn't rely on any one model. MMM outputs should not be the only inputs into big budget decisions.
An MMM doesn't necessarily have context of a business' broader goals. Perhaps the business wants to increase sell-through at a particular retailer despite their ad performance, or take a hit on a newly launched SKU.
We asked the model the wrong question
Director of retail media at Acadia, Ross Walker talked about how this played out for a brand account that he inherited at Acadia.
Acadia took over an account where the previous team had structured the measurement inputs by retail media network × ad type — Walmart sponsored product vs. Amazon sponsored product, and so on.
Ross's objection is that those two things aren't the same thing at all. What matters to incrementality isn't which retailer the dollar went to or what the ad unit was called; it's what the dollar was doing — defensive or offensive, branded or non-branded, upper funnel or lower funnel.
Ross worked with Sally's analytics team to rebuild the model around goals instead of formats: upper-funnel display against lower-funnel branded search against non-branded search, each one reporting its output distinctly. (He notes there's a limit to how far you can subdivide before it stops being useful.)
The failure mode he described is that MMMs come back saying "put more money into sponsored products," which was performing well just because that spend was defensive.
The model wasn't wrong, it was just fed the wrong question. It reported that sponsored products performed well — and they did. Defensive branded search converts quickly, because the customer had already decided.
So the failure sits one level above the metrics. The industry conversation is about better attribution and better dashboards. Ross's point is that a perfectly calibrated model fed the wrong taxonomy gives you a confident recommendation to double-down on an ad strategy that misses the bigger picture.
We got what we asked for
So we asked for this, right? Brands wanted accountability for every dollar. Retailers gave us that, and now they are claiming every conversion they plausibly can. We're in a prison of our own making.
Measurement models can only compare the buckets you give it. But the buckets are yours. When your analytics team builds the model, somebody has to decide how the spend gets grouped before it goes in — and that somebody works for you, not for a retailer.
Group it by retailer and ad type, and the model tells you sponsored products are working. Group it by what the money was trying to do, and you find out whether any of it brought you a customer you didn't already have.
That's kind of what I did when I recorded the podcast with the incorrect intro. I said the wrong name with total confidence, because it was the name I'd used for seven years, and nothing in my own head flagged it.
Mine got caught because somebody was listening for it. Who's listening to your model?
