In my last post I laid out five questions to ask an MMM firm before you sign the contract. Question #1 was about uneven data precision. Here’s what it costs when nobody asks it.

I built a simulated world where I know the ground truth: direct mail generates 30 leads per $1K, digital generates 25. DM is the stronger channel, by design.

One flaw in the data: DM spend is recorded when the invoice is submitted/paid, 3 to 5 weeks after the mail actually drops. Digital is recorded the week it runs. (If you’ve ever reconciled a media plan against accounting data, you know this isn’t hypothetical.)

Then I threw four models at it, from simple OLS all the way up to a full Robyn-style MMM: adstock, saturation, seasonality, ridge regularization, decay tuned by 5-fold cross-validation. The kind of model a skeptic can’t wave off.

Every single one estimated direct mail at or below ZERO and handed the credit to digital.

And the most sophisticated model, with the best fit of the four, gave the most confidently wrong advice. It predicted that shifting $20K from DM into digital would gain ~1,450 leads. The true outcome of that shift: a small loss. Wrong sign, sold as a big win.

Three takeaways for anyone buying or building an MMM

• Fit is the wrong success criterion. High R² means the model explains the past. It says nothing about what happens when you actually move spend. (Also why a vendor leading their pitch with R² is a red flag.)
• A rolling average can’t save you. Smoothing a signal that’s 4 weeks late gives you a smoother signal that’s still 4 weeks late.
• The fix lives in the data, not the model. Match spend to the date the media dropped, before a single line of code gets written. No amount of adstock or cross-validation substitutes for that.

The quiet tell was there the whole time, by the way: near-zero cross-validation R² on the spend variables. Plenty of teams would ship it anyway, because the in-sample fit chart looks beautiful.