Blocked Holdout Validation

Stage 35 is AMMM3’s out-of-sample time-series validation layer.

It answers a narrower and more useful question than “does the model fit the training data?”:

“If I refit the MMM on the earlier history only, can it still predict the last blocked window reasonably well?”

For weekly MMM, that is usually a better stress test than a random split because media carryover, seasonality, and trend all depend on time order.

Why AMMM3 uses a blocked tail holdout

AMMM3 reserves the final holdout_observations unique dates as the holdout window, then fits a fresh model on the earlier dates only. The validation pipeline does not reuse Stage 20 posterior state.

That means Stage 35 is checking whether the full model specification can generalize forward in time, not whether the same fitted posterior can explain the rows it already saw.

This is especially useful in MMM because:

  • adstock depends on lagged spend history
  • seasonality is time-ordered rather than exchangeable
  • marketing calendars often drift near the end of the sample
  • overfit specifications can look fine in-sample and fail on the final weeks

See the implementation in validation.py.

How Stage 35 works

Given a YAML block such as:

validation:
  enabled: true
  holdout_observations: 8
  include_last_observations: true
  coverage_levels: [0.5, 0.8, 0.94]
  sampler:
    draws: 500
    tune: 500
    chains: 2
    cores: 2
    random_seed: 42

AMMM3 does the following:

  1. Sort all unique model dates.
  2. Reserve the last holdout_observations dates as the holdout window.
  3. Fit a fresh model on the remaining earlier dates only.
  4. Sample posterior predictive draws for the holdout rows.
  5. Compute uncertainty-aware predictive metrics and residual diagnostics.

If include_last_observations: true, AMMM3 prepends the trailing lag history needed for adstock carryover internally, then trims those prepended rows back out of the returned holdout predictions. This matters whenever media effects have memory.

Why this stage exists when Stage 30 already exists

Stage 30 and Stage 35 answer different questions.

  • Stage 30 is an in-sample fit check. It uses the same rows the model was fit on.
  • Stage 35 is an out-of-sample blocked holdout check. It uses future dates the validation fit did not see.

If Stage 30 looks good and Stage 35 looks weak, that is a classic warning sign of overfit, misspecification, or regime change.

Main artefacts

Stage 35 writes the following files under results/<run_name>_<timestamp>/35_holdout_validation/:

  • validation_metadata.json
  • holdout_posterior_predictive.nc
  • holdout_predictive_summary.csv
  • holdout_predictive_report.json
  • holdout_observed.csv
  • holdout_fitted.csv
  • holdout_residuals.csv
  • holdout_timeseries.png
  • holdout_residuals_acf.png

See also Output Directory Schema.

How to interpret the main metrics

The headline table is holdout_predictive_summary.csv.

Point-error metrics

  • rmse: penalizes large misses more heavily than MAE. Lower is better.
  • mae: average absolute forecast error. Lower is better.
  • nrmse and nmae: normalized versions of RMSE and MAE. Lower is better and easier to compare across datasets with different target scales.

Use these as forecast-quality metrics, not causal-identification metrics.

Bias

  • bias: average signed error. Values near zero are better.

The implementation computes bias = mean(observed - predicted_mean).

  • Positive bias means the model underpredicts on average.
  • Negative bias means the model overpredicts on average.

Persistent bias on the holdout tail often points to a missing trend shift, pricing change, macro shock, or baseline misspecification.

CRPS

  • crps: continuous ranked probability score. Lower is better.

CRPS is usually more informative than RMSE alone because it rewards both:

  • accurate central predictions
  • calibrated predictive uncertainty

A model can have decent RMSE and still poor CRPS if its intervals are badly calibrated.

Coverage

AMMM3 reports fixed interval coverage as:

  • coverage_50
  • coverage_80
  • coverage_94

These should be read against their nominal values:

  • coverage_50 should be near 0.50
  • coverage_80 should be near 0.80
  • coverage_94 should be near 0.94

Low coverage can reflect narrow intervals, biased forecast centres or distribution shift. High coverage can reflect wide intervals or finite-sample variation. Inspect interval width and signed residuals together.

Report the number of scored observations. With eight scalar observations, coverage changes in steps of 1/8; nominal 94% coverage cannot be observed exactly. Panel cells and neighbouring dates can also be dependent. Account for this dependence and the limited holdout size before concluding that interval calibration is poor.

How to read the plots

holdout_timeseries.png

This is the first plot to inspect.

Look for:

  • whether the observed series generally stays inside the predictive intervals
  • whether misses are isolated or systematically one-sided
  • whether the model misses turning points or holiday spikes
  • whether the predictive band width looks plausible relative to the volatility of the target

Common interpretations:

  • repeated misses on the same side: likely bias
  • bands too narrow around volatile weeks: likely overconfidence
  • bands extremely wide but always covering: likely weak identification or an overly flexible baseline

holdout_residuals_acf.png

This checks whether the holdout residuals still contain serial structure.

Look for:

  • obvious positive autocorrelation across nearby lags
  • repeating seasonal patterns
  • long runs of same-sign residuals

If residual autocorrelation is strong, the model is usually still missing some time structure such as:

  • seasonality
  • holiday dynamics
  • delayed media effects
  • structural breaks or time-varying baseline behavior

Practical rules of thumb

These are pragmatic MMM heuristics, not hard pass/fail thresholds.

Choosing the holdout window

  • For weekly MMM with roughly 1 to 2 years of data, 6 to 12 weeks is a practical starting range.
  • 8 weeks is a sensible default when you want enough tail signal without throwing away too much history.
  • If the dataset is very short, a larger holdout can make the validation noisy and can leave too little training history for stable estimation.

AMMM3 itself enforces that blocked holdout validation must still leave enough training dates for the model to run; see validation.py.

Comparing Stage 30 and Stage 35

  • Expect Stage 35 to be worse than Stage 30. That is normal.
  • Worry when the degradation is large or the direction changes materially.
  • If Stage 30 is excellent and Stage 35 is weak, suspect overfit or misspecification before celebrating the in-sample fit.

Reading coverage

  • Coverage slightly below nominal can be acceptable.
  • Coverage far below nominal warrants investigation of bias, interval width and changes between the training and holdout periods.
  • Coverage far above nominal warrants a check of interval width and the amount of independent holdout information.

Reading bias

  • A small nonzero bias is normal.
  • Large consistent bias over the holdout tail is a stronger warning sign than a single noisy miss.
  • If bias retains the same sign across several model variants, inspect trend, holidays, and baseline structure before changing media priors.

Reading residual structure

  • White-noise-like residuals are what you want.
  • Visible residual runs or autocorrelation usually mean the model is still missing systematic time variation.
  • Do not treat a decent RMSE as sufficient if residuals still show structure.

What Stage 35 does not tell you

Blocked holdout validation is valuable, but it is not a causal guarantee.

It does not prove:

  • that channel attribution is identified
  • that ROAS is unbiased
  • that the chosen priors are correct
  • that the model is safe for large budget reallocation on its own

It does tell you whether the specification can forecast a held-out tail window coherently. That makes it an important diagnostic, but still only one part of MMM model assessment.

For most weekly MMM work:

  1. Run Stage 30 and Stage 35 together.
  2. Compare in-sample and holdout metrics before changing the specification.
  3. Use the same holdout window across candidate models so the comparison is fair.
  4. Prefer specifications that are stable across reasonable prior choices, not just the one that scores best on a single holdout.
  5. Treat Stage 35 as a forecasting sanity check alongside prior predictive checks, posterior predictive checks, and substantive business review.

Common mistakes

  • Using a random split instead of a blocked time split
  • Reading Stage 30 as out-of-sample validation
  • Ignoring coverage and focusing only on RMSE
  • Using a holdout that is too long for the amount of history available
  • Repeatedly tuning the spec to one holdout window until the score looks good