Model comparison for econometricians

Predictive model comparison assesses a specified prediction task. Similar predictive scores can coexist with substantially different media attribution. A predictive ranking does not identify a causal effect.

ELPD and leave-one-out prediction

Expected log pointwise predictive density (ELPD) evaluates predictive densities on a log scale. For leave-one-out (LOO) prediction, each unit is scored using a model fitted without that unit. The ArviZ elpd_loo result is the sum of those estimated log predictive densities. It is not their average.

Pareto-smoothed importance sampling (PSIS) approximates LOO from posterior draws. Its accuracy depends on the importance ratios and is not guaranteed by having a Bayesian model. Exact refitting or a different validation design may be necessary.

Higher ELPD is better when the scoring basis and prediction task match. Absolute values depend on the outcome scale and number of scored units. WAIC estimates a related predictive quantity and also has reliability limits. Both can be influenced by priors and likelihood specification.

Library outputs

Use mmm.diagnostics.bayesian_criteria_summary() for a table, or mmm.diagnostics.bayesian_criteria_report() for a typed report. The report contains LOO and WAIC estimates, standard errors, warnings, Pareto-k summaries and likelihood_basis. Its fields se_elpd_loo and se_elpd_waic correspond to the se field on the respective ArviZ result.

The runner retains bayesian_criteria_summary.csv, bayesian_criteria_report.json and the equivalent summary alias elpd_summary.csv under 50_diagnostics/.

Conditions for comparison

Require the same outcome, outcome scale, scored units and prediction task. Matching the input CSV alone is insufficient.

  • FE log likelihood uses within-contrast coordinates.
  • CRE log likelihood uses marginal unit blocks after integrating random unit intercepts.
  • Ordinary level-observation likelihoods score a different unit from either of those contracts.

The report labels FE as contrast_space and other models as level_space. That label alone does not establish matching observation units. Inspect the estimator contract and log-likelihood dimensions before comparing scores. Do not directly rank these different likelihood contracts by their raw ELPD.

Leaving out an observation or unit is also different from forecasting future dates. Use blocked holdout validation for the latter task, with matching holdout dates and outcomes across models.

Differences and uncertainty

For compatible scores, az.compare() reports ELPD differences and their standard errors. Inspect pointwise differences and influential units alongside the aggregate. Dependence and heavy tails can make a standard-error summary misleading.

A difference smaller than roughly two standard errors is often inconclusive. It does not establish predictive equivalence. A larger difference can support a predictive preference under the comparison assumptions; it is not a likelihood-ratio test or evidence for a causal attribution. Equivalence needs a stated practical margin and an assessment designed for that claim.

Pareto-k reliability

Inspect the good_k threshold returned by ArviZ. In ArviZ 0.23.4 it is min(1 - 1/log10(S), 0.7), where S is the posterior sample count used by LOO. Values above that threshold trigger a reliability warning. Values between 0.7 and 1 should not be endorsed as generally usable.

AMMM3 also reports counts above fixed 0.7 and 1.0 thresholds for screening. Those counts do not replace ArviZ’s sample-size-dependent warning. Investigate problematic units and consider exact refits or appropriate K-fold validation. ArviZ 0.23.4 does not expose the moment-matching option previously described in this guide.

Relation to classical criteria

AIC targets an asymptotic approximation to predictive performance under its regularity assumptions. BIC approximates a marginal-likelihood comparison under different assumptions. Bayes factors compare marginal likelihoods and can be sensitive to priors. ELPD addresses predictive performance and cannot replace every inferential role of those criteria.

Assess absolute fit with predictive checks as well as relative scores. Then assess causal assumptions, calibration evidence and attribution sensitivity for the business question. A small predictive difference does not settle those questions.