A Bayesian workflow for agency teams

AMMM3 supports a repeatable process for turning noisy marketing data into reviewable model evidence and conditional planning scenarios. The useful product is a recommendation whose assumptions, uncertainty and supporting artefacts another analyst can inspect and reproduce.

This guide defines the minimum evidentiary standard for teams adopting that process. It allows imperfect data and bounded conclusions. It requires an explicit reduction in the claim when the evidence cannot support the original decision. More elaborate modelling does not remove that obligation.

Apply the standard

Use the methodological benchmark protocol v1 to assess whether a configured tool and its operating procedure meet this standard. The protocol specifies evidence requirements, failure cases and a review record. It contains no comparative performance ranking.

In this section, must marks a requirement for claiming compliance with the workflow. Should marks a recommended practice that can be varied with a recorded reason. Requirements apply to the intended use and estimand, rather than to a model in isolation.

The modeller prepares the evidence. A named technical reviewer assesses it. The business decision owner accepts the commercial trade-offs within the reviewed scope. A reviewer should be someone other than the modeller. If a small team cannot separate those roles, it must record that limitation and must not describe the review as independent.

These are operating requirements. AMMM3 does not currently enforce the whole review process in software. The enforcement boundary below identifies what the library checks and what the team must control.

Why use a Bayesian workflow

Bayesian modelling makes assumptions explicit and represents uncertainty conditional on a likelihood, prior and model specification. Partial pooling and regularisation can stabilise estimates when the data contain limited information. Prior predictive checks expose assumptions that imply implausible outcomes before those assumptions are combined with observed data.

These benefits have conditions. A concentrated posterior can reflect a strong prior. Flexible baselines can absorb media variation. A restrictive baseline can force unrelated demand into media contributions. Neither a credible interval nor a plausible response curve proves that the fitted response is a causal effect. Non-Bayesian procedures must meet the same evidentiary standard; they need to explain what their uncertainty estimates represent.

The aim is a model simple enough to interrogate and rich enough for its intended use. A team does not need to explore every possible specification. It must investigate the plausible alternatives most likely to change the recommendation, and retain the unsuccessful attempts as well as the preferred fit.

Match each claim to its evidence

The following table sets the permitted interpretation of each claim class. These classes are not automatic stages of increasing approval.

A decision under uncertainty can still be useful without a verified causal point estimate. For example, a reversible learning experiment can be justified by uncertainty about channel effectiveness. A conditional allocation scenario must not be sold as a forecast of incremental sales unless the causal argument supports that interpretation.

Define the decision before fitting

The modeller and decision owner must record the outcome, population, unit of analysis, time horizon, baseline plan and proposed change. Distinguish total from marginal return, spend from exposure, and channel contribution from a change caused by an intervention. Define a practically important change in the outcome or plan before inspecting candidate results.

Document the source of variation that could distinguish competing explanations. A short causal diagram or an explicit list of causal assumptions is sufficient as a design record if it addresses the actual business process. The record itself is not verification of those assumptions. Consider demand-led budget setting, price and promotion, distribution changes, targeting, common seasonality and measurement changes. Explain why each control belongs in the model. Adjusting for a consequence of advertising can remove part of the effect of interest; adjusting for a collider can introduce bias.

A causal claim additionally needs a defensible account of confounding, intervention consistency, relevant support and interference or spillovers. An experiment is valuable evidence when its design is credible and its estimand matches the question. An observational identification argument is permitted, but its assumptions and unresolved threats must be explicit. Untestable assumptions must not be marked as verified by diagnostics.

Select an estimator within its supported contract. FE removes time-invariant unit intercepts from its slope likelihood; it does not remove time-varying confounding. CRE uses the declared between-unit adjustment and does not make that adjustment sufficient by construction. Consult the estimator selection guide.

Make messy data reviewable

Keep a fixed input snapshot and a record of its source, retrieval date, units, definitions and transformations. Distinguish missing observations from observed zeros. Record imputation, exclusions, aggregation, outlier treatment, currency conversion and changes in tracking or attribution. Reconcile the model outcome with the business measure that the recommendation concerns.

Inspect variation over the relevant time and geographic scales. Many rows do not guarantee many independent changes in media activity. Check shared campaign timing, channel substitutions and variation remaining after controls or within-unit transformations. Raw correlations and VIF are useful screens; they do not establish identification of a nonlinear MMM.

A material data defect requires repair, an explicit measurement assumption with sensitivity analysis, or a narrower use. Silent deletion and undocumented imputation fail the standard. Holdout construction must isolate evaluation outcomes from fitting and tuning. Record which future inputs are known at the forecast origin and which are assumed. Actual future spend can support a conditional prediction exercise, but changes the forecast task.

Specify and challenge assumptions

Record the likelihood, link, media transformation order, lag normalisation, maximum lag, baseline structure, controls, pooling and priors. Preserve the scaling needed to interpret every parameter. Sign restrictions and bounds are substantive assumptions and must have a reason.

Before interpreting a posterior, inspect prior predictive implications on the outcome and relevant response scales. Explain the practical meaning of the prior rather than relying only on its distribution name. A domain-informed prior can be useful even when precise external evidence is unavailable, but that uncertainty must be acknowledged and its decision consequences tested.

AMMM3 can write resolved prior-sensitivity scenarios. Writing those configurations is planning evidence. The team must fit the selected alternatives and compare their results before claiming stability.

Use calibration as evidence with a scope

For external experiments, retain the intervention, control condition, outcome, channel mapping, spend contrast, population, dates and uncertainty. Explain transport to the MMM population and period, including carryover and differences in exposure or spend. A calibration measurement must constrain the intended modelled contrast; matching a channel name is insufficient.

Record whether calibration enters through a prior, an additional likelihood or a penalty. Those mechanisms need not express the same statistical model. A cost-per-target penalty must not be described as independent experimental evidence unless its source supports that description. Check scale alignment and the graph contribution rather than accepting a narrower posterior as proof that calibration is correct.

Account for reused or overlapping information. Evidence used to calibrate a model cannot also be presented as an independent validation of that model. Conflicting experiments require investigation of design, estimand and transport, or a stated model for disagreement. Do not select only the preferred result.

See calibration for AMMM3’s current Python surface. Named FE and CRE presets do not support calibration.

Establish usable computation and adequate fit

For posterior sampling, retain chain diagnostics, effective sample sizes, Monte Carlo error where available, divergence and energy diagnostics, and any sampler warnings. Inspect the parameters and decision quantities that matter. A rounded summary value or a single maximum diagnostic is insufficient when chains disagree or warnings remain unexplained.

Freeze the applicable diagnostic profile before reviewing the preferred fit. AMMM3 records resolved gates in diagnostic_gates.resolved.yaml; the diagnostic guide explains the outputs. A failed computation gate blocks interpretation of affected posterior results until it is resolved. If a gate is demonstrably inapplicable, record a technical reason and a revised review scope. Do not weaken it simply to obtain a pass. Missing or non-finite required diagnostics cannot count as passing.

Separate numerical instability from statistical uncertainty. Repeat a material result with independent sampling initialisation and assess whether Monte Carlo variation is small relative to the decision tolerance. Extra draws can reduce Monte Carlo error; they cannot create missing identifying variation.

Use posterior predictive and residual checks to find material mismatch in the outcome patterns relevant to the task. Residual structure warrants investigation of omitted dynamics or measurement processes. It does not identify a causal mechanism by itself. When prediction is claimed, retain a relevant temporal or geographic evaluation and a simple reference prediction. AMMM3’s blocked holdout stage refits on the training period and can preserve trailing adstock history.

LOO and WAIC are predictive diagnostics under their likelihood and factorisation. Compare only compatible outcomes, observations and likelihood bases, and retain uncertainty in the comparison. A within-contrast likelihood and a level-space likelihood are not interchangeable comparison targets. Ordinary observation-level LOO does not directly assess future forecasting. Warnings and small holdouts limit conclusions; neither proves that the causal model is wrong, and a good score does not establish that it is right.

Assess identification and decision stability

Assess the estimand being used. Individual channel coefficients, total media contribution, marginal return at current spend and an allocation contrast can have different levels of support.

Weak identification means that the data distinguish relevant explanations poorly. Examine parameter dependence, likelihood-informed variation, plausible alternative specifications and prior sensitivity together. Marginal posterior-to-prior contraction alone cannot establish joint identification. A structural non-identifiability claim requires an applicable exact dependency or a demonstrated likelihood invariance. A failed raw-design screen is not such a proof for the full nonlinear model.

The minimum sensitivity set must target the largest plausible threats to the recommendation. These may concern priors, baseline flexibility, controls, lag structure, influential periods or calibration transport. Record the reference, each alternative, its reason, the result and the consequence for the decision. Change one factor at a time for diagnosis, then assess a combined alternative when interacting assumptions could invalidate that diagnosis.

Stability means that the intended conclusion remains acceptable within the predeclared practical tolerance across credible alternatives. It does not mean identical coefficients or intervals. If an interval includes zero, describe the sign as inconclusive. If it rules out effects of practical importance, state that narrower conclusion with its assumptions. Do not choose a model because it returns a preferred channel ranking.

When evidence remains weak, narrow the channel grouping or decision, present explicit conditional scenarios, collect relevant evidence, or stop. Any aggregation changes the estimand and requires its own review. Do not assume that an aggregate is identified merely because its components are uncertain.

Evaluate plans within the evidence

A planning comparison must specify the reference plan, total and per-period budget units, channel mapping, constraints, outcome basis, requested horizon and carryover horizon. Document assumptions about non-media inputs. Inspect extrapolation beyond observed spend and the support of calibration evidence.

Verify that the optimiser’s objective represents the intended decision. An optimum computed from a posterior mean response is a conditional optimisation result. It is not an uncertainty-aware decision rule simply because the model is Bayesian. Assess plausible outcome variation and relevant model alternatives for the chosen plan. Separate uncertainty in the response from variation in which allocation would be preferred. A narrow range across optimiser restarts does not measure either statistical uncertainty.

AMMM3’s scenario workflow retains typed plans and comparison outputs. Named FE and CRE support historical and manual scenarios within their fitted-unit contracts; fixed-budget optimisation is unavailable for those presets. Named RE remains gated.

Review, release and revisit

Before external decision use, retain the protocol review record with evidence links, reviewer, permitted claim, limitations and expiry or review trigger. The standard permits three case decisions.

A restriction cannot waive a failed requirement while retaining the original claim. Exploratory outputs can remain available internally with a clear label. Business urgency does not turn missing evidence into a technical pass.

Retain original inputs, resolved configuration, environment and code versions, fitted draws or fitted state, diagnostics, alternatives, calibration evidence, scenario specifications and review decisions. Use content hashes to identify artefacts and a controlled retained copy to anchor the review. A checksum proves identity against that reference; it does not prove truth, authorship or approval.

A second analyst must be able to reload the retained state and reproduce a material scenario within a declared tolerance. Separate that replay from a fresh stochastic refit and from replication on new data. A seed alone is not a reproducibility record. Reopen the review after material changes to inputs, measurement, specification, calibration, software or the proposed decision.

The AI advisor can explain evidence and propose tests. Its prose, referenced evidence and interpretation must be checked by the reviewer. It cannot supply missing identification, validate its own recommendations, or approve a model or business decision.

A worked agency judgement

Suppose television and paid social campaigns repeatedly run in the same weeks. A fitted model reproduces total sales reasonably well, but channel allocations change under plausible baseline and prior choices. This example is illustrative; it is not a measured result for any library.

The modeller first checks computation. If sampling is unreliable, channel interpretation stops. Once computation is usable, the modeller examines the transformed design and joint posterior, and compares the relevant alternatives. The reviewer distinguishes a weakly supported channel split from any separately assessed aggregate quantity. Good total prediction does not settle either channel’s causal contribution.

A tighter social prior may stabilise the split by imposing a stronger assumption. Without independent justification and sensitivity evidence, the team cannot describe that as learning more from the data. A relevant experiment may help, subject to its spend, timing and population differences.

The permitted output might be a set of conditional scenarios and a proposal for an experiment. The team can explain which assumption makes each scenario attractive and what new evidence would change the decision. It must withhold an unsupported claim that one channel produces a particular incremental return. That is a useful commercial result with an explicit evidentiary limit.

Enforcement boundary

The following table distinguishes current AMMM3 behaviour from obligations of a team claiming compliance. It is not a declaration that AMMM3 has passed the benchmark.

A completed run_manifest.json records execution status. Diagnostic failures reported as findings do not automatically prevent Stage 70 optimisation. An advisor decision state is not an agency release authorisation. The current advisor does not evaluate completed prior-sensitivity comparisons. Its global identification assessment remains unassessed, and computational checks do not establish causal identification.

Until end-to-end release controls exist in an application or team process, the technical reviewer must maintain the review record and stop unsupported external use. Source code, a fitted model or an attractive report alone cannot satisfy that requirement.

Continue the review

• Methodological benchmark protocol v1 defines the frozen acceptance procedure.
• Estimator selection defines current operation limits.
• Pipeline overview explains the execution record.
• Scenario comparison outputs explains retained planning evidence.