Post-remediation review and release gates
Date: 11 September 2026. Review baseline: e436b2b on main.
Scope: commits 7f469c9 and e436b2b, using the eight findings in the
10 September independent review as the requirements. This is a software
contract review; it does not establish estimator recovery or causal validity.
Execution plan
| Step | Owner | Deliverable and acceptance criterion |
|---|---|---|
| 1. Pin evidence | Primary agent | Record HEAD, clean/dirty state and hashes of the original report, verification record and reproduction scripts/results. Never run the original scripts in place. |
| 2. Forecast review | Independent forecast reviewer | Check actual public predictions against an independently calculated finite convolution and saturation result. Cover histories below/equal/above lag length, exact requested dates, one-date FE prediction and save/load. Report training-state mutations and retained-output differences separately. |
| 3. FE covariance review | Independent randomness reviewer | Hold model parameters fixed. Check conditional variance sigma^2 (1 + 1/T), within-unit cross-date covariance sigma^2/T, and zero cross-unit/cross-draw covariance. Use a bounded predictive oracle with declared numerical/Monte Carlo tolerances; check reproducibility separately from arbitrary chunking bit equality. |
| 4. Integrity review | Independent integrity reviewer and primary agent | Check labelled schedules, bounds, persistence and exclusive run ownership. Add separate tests for objective/gradient/allocation invariance, changed sources while still present, deleted sources, relocation and simultaneous collisions. Verify original artefact hashes remain unchanged. |
| 5. Reconcile findings | Primary agent | Distinguish confirmed defects, missing coverage and satisfied contracts. Report exact checks, expected failures and sampling limits. Passing broad tests cannot override a failing numerical oracle. |
| 6. Release decision | Primary agent, informed by independent reviews | Prepare a release only if required contracts pass and no unresolved result/evidence-integrity defect remains. Otherwise record a hold and a concrete repair backlog. Tagging and publishing are separate actions. |
| 7. Statistical validation scope | Primary agent | Write a separately approved study design with estimands, data-generating processes, replication precision and acceptance criteria before fitting or running recovery/coverage studies. |
Verification strategy
Use the existing Python environment. Add test_post_remediation_*.py tests;
keep original defect reproductions as historical evidence. Run each new test
file, then the affected existing suites and repository static checks. A final
default full-suite run is required if production code changes; tests and
documentation alone need the focused and relevant existing suites. Record
failures explicitly; do not weaken an oracle to match the implementation.
The reviewers may use fixed/prior parameter draws and bounded conditional predictive sampling. No full MCMC, estimator simulation study, dependency upgrade, CI change, version bump or publication belongs to this review.
Release decision
Software release preparation is approved; tagging and publication remain separate actions. Independent reviewers found residual optimiser and forecast-input defects, so the previous claim that F3–F8 were fully closed was too broad. The repairs below close those defects, and the required static, affected, default-suite and package checks pass.
Passing contracts: adjacent future TS/FE/CRE predictions match an independent geometric-convolution/logistic oracle for histories below/equal/above lag length; one-date FE prediction survives NetCDF reconstruction; fitted FE contrasts and the training minimum remain intact. Fixed-parameter FE predictions satisfy the joint conditional law and cross-draw independence under two chain/draw layouts. The old RNG restart, restored in memory, fails the new cross-draw oracle.
The primary-agent checks also verify schedule channel-label permutations at objective, gradient and allocation level; source mutation while files still exist, deletion and relocation; missing DataArray bounds rejected before SciPy; and two simultaneous run-directory reservations producing exactly one owner.
Initial review result: 18 new checks passed, 6 strict expected failures
exposed R1–R4/C1 below. With expected-failure handling disabled, those six
cases failed. 85 affected existing tests passed with five prior conversion
warnings. Ruff, formatting, configured MyPy and diff checks passed. Production
source was unchanged at that point. The independent reports, exact commands and
logs are saved separately under .planning/post-remediation-2026-09-11/;
SHA-256 comparison confirms all ten original review documents/scripts/results
remain unchanged.
Post-repair result: 34 post-remediation checks pass. They cover partial-mask label permutation, finite and canonical bounds, finite non-negative schedules, unambiguous low-level date coordinates, forward-only carry-over rejection, and the existing forecast, persistence, run-isolation and FE covariance contracts. The final affected regression set passed 347 tests with 79 existing warnings. The default suite passed 2,953 tests, skipped one, and reported 170 existing warnings under the required PyTensor configuration. The isolated package verifier built and installed the 3.1.0 source distribution and wheel.
Repaired input contracts
| ID | Observed defect or unresolved contract | Acceptance gate and owner |
|---|---|---|
| R1 | budgets_to_optimize remained positional: permuting labels changed which channel received spend. | The optimiser now requires exact unique labels and aligns masks to fitted budget coordinates before availability checks or array conversion. The partial-mask objective and gradient are invariant under channel-label permutation. |
| R2 | Dictionary bounds bypassed numeric validation. An actual solve with channel b bounded by (NaN, NaN) succeeded and assigned it 9 of 10 budget units. | Both bounds forms now use one finite-numeric and lower-not-greater-than-upper check before SciPy. DataArray bounds require ordered explicit labels: canonical ['lower', 'upper'] or the retained ['low', 'high'] spelling. |
| R3 | Schedule sums skipped NaNs, and negative entries could still sum to one. | Schedules now require finite, non-negative numeric values before sum validation and graph compilation. |
| R4 | Non-overlapping carry-over requests before the final fitted date were accepted. | Carry-over now requires every supplied date to be after the final fitted date before graph updates or predictive sampling. Retrospective and mixed windows fail at that boundary. |
| C1 | Schedule dates were length-only and bound-axis labels were checked only for length. | Low-level integer schedule dates must be canonical positions 0..N-1; all dates must be unique and ordered. Wrapper calendar-date validation remains exact against the requested scenario window. Bounds require ordered lower/upper labels. |
The strict expected-failure markers were removed after their repaired contracts passed. Static checks, the affected suite, the default suite and installed-wheel verification now pass.
Have independent reviewers inspect both repairs and the compatibility notes. Release preparation may then proceed only with no unresolved integrity finding, accurate notes on repeated-fit rejection, retained-model metadata, labelled optimiser inputs, forecast/RNG changes and collision errors, and recorded package verification. Version selection, tagging and publication follow that decision; none occurs during this review.
Separate statistical-validation design
Before execution, specify each released estimator and permitted operation. Freeze priors, transforms, likelihood, scaling and estimands: coefficients on their declared scale, fitted-unit conditional forecasts, and intervention contrasts only where the data-generating design identifies them. FE and CRE studies must retain balanced fitted-unit restrictions; named RE remains gated.
Use correctly specified synthetic time-series, FE and CRE designs plus explicit stress cases: weak within-unit variation, media collinearity, short histories, near-zero variance components, and correlated unit effects. Treat misspecified and confounded designs as separate sensitivity experiments, not tests expected to recover an unidentified effect. A causal study additionally needs a defined intervention, horizon, assignment mechanism and justified counterfactual.
Pre-register parameter values, sample sizes, seeds, replication count, fitting budget and failure handling. Choose replication count using binomial Monte Carlo precision for coverage, rather than a convenient round number. Specify simultaneous tolerances across the planned estimands before inspecting results.
Proposed acceptance gates, to finalise before execution:
- Recovery: absolute bias and root mean squared error below pre-agreed limits in the estimand’s units, with Monte Carlo uncertainty reported.
- Interval coverage: nominal coverage lies within the pre-specified Monte Carlo tolerance for correctly specified, identified designs; also report interval width and non-estimable cases.
- Convergence: per-fit R-hat, effective sample sizes, divergences and numerical failures satisfy declared thresholds. Count failed fits in the denominator; never report only successful replications.
- Forecasts: evaluate conditional means, joint covariance and interval coverage on held-out fitted-unit periods, with carry-over policy frozen.
- Causality: no causal claim follows from predictive or recovery checks alone. Require separately documented identifying assumptions and sensitivity tests.
Deliver a design, machine-readable configuration, complete replication ledger and uncertainty-qualified report. Study approval authorises the specific computational budget; this document does not authorise its execution.
Proposed initial study specification
Use two media channels and a Gaussian outcome with residual standard deviation
- Set shared response amplitudes to 0.5 and 1, geometric decay to 0.3 and 0.7, and logistic slope to 1. Generate positive media exposure with pre-specified cross-channel correlations 0 and 0.8. Use 52 and 104 equally spaced periods; FE/CRE panels use 8 and 24 units. Keep the first study’s transform order, normalisation, priors and scaling equal to a frozen model configuration, and report truths in the same units as the fitted parameters.
For FE, include fixed unit intercepts and strictly exogenous innovations. For CRE, generate unit intercepts with a declared linear dependence on unit exposure summaries plus independent Gaussian variation; test population standard deviations 0.1 and 0.5. Ensure the generated summary definition matches the model’s transformed-summary contract. If that DGP cannot be specified without ambiguity, resolve it before launching that cell. Analyse time-series, FE and CRE separately rather than pooling their coverage rates.
Primary estimands are each channel’s response amplitude and decay, the conditional mean for a two-period fitted-unit forecast, and its 95% predictive interval. Secondary intervention estimands require a separately fixed spend change and horizon with explicit media assignment/exogeneity; no observational causal claim is made by this initial correctly specified study.
Propose 1,000 replications per approved design cell. For a single nominal 95% coverage rate this gives Monte Carlo standard error about 0.0069. Pre-specify the finite family of primary coverage checks and a simultaneous binomial acceptance band before execution. Proposed recovery gate: absolute bias less than 0.1 empirical sampling standard deviations for identified primary parameters, with Monte Carlo uncertainty. Report RMSE and interval width even when coverage passes; a coverage pass alone is insufficient.
Proposed computational gate for each analysed fit: R-hat at most 1.01, bulk and tail effective sample sizes at least 400 for primary scalar estimands, and no divergent transitions. Report all fits that fail these gates and keep them in the replication ledger. Freeze the sampler budget and any retry policy before execution. These are proposed study acceptance rules, not claims that the current estimators meet them or that the study budget has been authorised.