AI diagnostics advisor

The advisor reviews retained evidence and proposes controlled tests. It does not fit a model, change priors, approve an estimator release or establish causality. Model equations and FE, CRE and named RE operation gates remain unchanged.

Review an existing run

Run a review from the repository root. Replace results/my_run with an existing retained run directory.

python3 -m abacus.ai.review results/my_run

This command enables LLM calls by default. It creates a separate directory under results/my_run/advisor_review/ for each review. It preserves the fitted model, previous reviews and the run manifest. Read the paths printed in the terminal.

The command uses the provider configured in the retained config.original.yaml. The command uses config.resolved.yaml when the original configuration is absent. Add --no-llm to generate local reports without a provider request.

python3 -m abacus.ai.review results/my_run --no-llm

New demo runs use openai/gpt-5.6-terra. Existing runs retain their saved model. Use --model to select a different model for a review without editing the saved configuration.

python3 -m abacus.ai.review results/my_run --model openai/gpt-5.6-terra

A process environment variable supplies the API key. A .env file in the current working directory also works. Keep .env out of Git. The shipped demos also set llm_enabled: true. Saved-run reviews enable LLM calls even when the retained configuration has llm_enabled: false; use --no-llm to disable them.

The Python entry point has the same behaviour.

from pathlib import Path
from abacus.ai.review import review_run

review_dir = review_run(Path("results/my_run"))
print(review_dir / "advisor_review.md")

Pass llm_enabled=False for a local-only Python review. Pass model="openai/gpt-5.6-terra" to override the retained OpenRouter model.

Read the reports

Stage 08 writes deterministic preparation evidence and rules only. It never calls a provider. Stage 90 writes the final review after response curves, optimisation and the Stage 80 interpretation inventory. It makes at most one provider request; an identical cached review makes none. The former Stage 55 report is no longer written in new runs. Historical run artefacts remain unchanged.

If a pipeline stage fails, the runner attempts a local incomplete-run review and preserves the original pipeline error. It does not call a provider during that failure handler. Use saved-run review to request LLM advice on the retained partial evidence. Missing and unfinished stages remain explicit.

Evidence and limits

The review includes configured prior paths and families, retained estimator contracts, design rank, diagnostic gates and predictive summaries. It reads parameter convergence summaries from 20_model_fit/posterior_summary.csv. Posterior and prior draws come from the corresponding groups in 20_model_fit/model.nc. Resolved prior specifications come from that file’s model_config attribute when a supported distribution can be matched to the parameter. Missing matches remain explicit.

Parameter screening covers at most 128 summary rows. Draw-based calculations use at most 1,000 evenly spaced draws per chain. The report records reviewed counts and reports incomplete coverage. Temporal variables are excluded from these calculations. The denominator uses an analytic prior SD for directly matched, scalar Normal, HalfNormal, Beta, Gamma, Exponential and Uniform specifications. Transformed, hierarchical and coordinate-specific specifications are excluded from this analytic calculation. Otherwise, a sampled prior SD is used only when its estimated relative Monte Carlo standard error is at most 5%. This precision budget applies to the denominator; it is not an identification threshold. The fourth-moment approximation assumes independent draws and a finite fourth moment. Unavailable or imprecise denominators leave contraction unassessed.

The evidence also records configured temporal terms, holiday-component presence, holdout calendar boundaries, prepared scenario counts, predictive-calibration scope, relative channel contribution shares, curve availability, numerical optimiser status and the interpretation inventory. It excludes raw outcome and spend values. Artefact presence does not establish adequacy. Curve shapes, optimisation stability and completed sensitivity comparisons remain unassessed by this reader.

Absolute posterior correlation of at least 0.9 raises a screening concern. Computation status is recorded separately; unresolved computation withholds interpretation without erasing the descriptive screen. This is a descriptive threshold, not an identification or release gate. Correlation can also reflect parameterisation or deterministic relationships. Marginal contraction can coexist with weak joint identification or dependence on prior assumptions.

FE and CRE reference-feature reports retain their reference-design scope. Numerical zero within variation is labelled non_estimable_reference. It does not prove global nonlinear non-identifiability. Raw-input rank, VIF and condition numbers also remain screening evidence.

The provider receives parameter aliases, recognised parameter families, selected prior specifications and relative diagnostics. The full local parameter lookup is excluded. Review diagnostics_evidence.json before sending sensitive work. These checks cannot certify the absence of every identifying detail.

Decision states

Missing or unusable required diagnostics produce incomplete_evidence. Diagnostic failures still take precedence and produce needs_model_rework. This retained state requires investigation and restricts interpretation. It does not establish that the model specification must change. Shared sampler failures produce one run-level finding. Few divergences and good convergence summaries cannot establish that affected estimates are reliable. Privacy or configuration failures block provider calls with do_not_use.

The current review does not assess completed prior-sensitivity comparisons. Passing available diagnostics therefore still requires needs_more_sensitivity when there are no missing required artefacts. A generated scenario plan does not satisfy that requirement. Global identification remains not_assessed; causal identification remains not_established.

An LLM cannot upgrade fit-review readiness to decision readiness or weaken the deterministic assessment. If reconciliation changes its decision state, the report replaces its summary with the deterministic assessment and withholds its narrative, recommendations and patch proposal so that optimistic instructions cannot contradict the assessment. The raw response remains available for inspection. Accepted recommendations still require human review.

Recommendations and provider handling

The advisor uses ADVISOR_SYSTEM_PROMPT in src/abacus/ai/prompts.py. OpenRouter receives it as a separate system message. The direct OpenAI client supplies it through the Responses API instructions field. Artefacts remain in the separate task payload and are treated as untrusted data.

The audience is marketing analysts. The prompt asks for 220 to 300 words across the summary, narrative and up to three recommendations. The main report explains what the results mean, what they imply for marketing decisions and what remains uncertain. It defines Bayesian terms in context and connects diagnostics to channel comparisons and planning decisions. Technical assessments are linked from the narrative. The separate parameter report retains its detailed tables.

Each LLM response must include non-empty narrative paragraphs and references to supplied evidence when evidence is available. These checks enforce structure and traceable references. Responses exceeding 350 prose words or three recommendations are rejected without an automatic retry. These limits exclude report headings and evidence references. The checks do not verify every prose claim. The system prompt forbids invented ROI, sales lift, channel rankings and causal claims from predictive fit alone.

Each recommendation must contain supporting evidence keys, a proposed test, an expected result, an acceptance criterion and an escalation criterion. Metadata such as channels cannot serve as evidence. Evidence validation checks supplied references; it does not prove that the LLM’s reasoning is correct.

Numerical prior advice requires a known parameter, scale, current prior and external justification. Narrower intervals alone do not justify tighter priors. A prior change must be assessed against plausible alternatives and the decision quantities of interest, with uncertainty.

The final advisor and saved-run review can write a validated prior patch proposal and a pending approval request. Stage 08 cannot propose patches. Invalid proposals produce config_patch_error.json. Approval and application remain separate from review. No review automatically refits.

OpenRouter requests require structured-output support, deny provider data collection and request zero data retention. Unsupported routing combinations fail visibly. See the OpenRouter structured-output documentation and provider routing documentation.

Requests are limited to 100,000 combined system and task characters and 3,000 output tokens. Truncated, refused or invalid responses are rejected. Returned usage is retained; reported cost is available only when supplied by the provider. Identical reviews reuse the run-local .advisor_cache keyed by provider, model, system prompt, task prompt and response schema. Cached usage describes the original request, not a new charge. Changed evidence or prompts invalidate the cache. No automatic retry loop spends further credits. A failed LLM request leaves the deterministic reports available. Received responses that fail validation are retained as <provider>_diagnostics_response.rejected.json. Their returned usage is recorded and llm_response_accepted is false. A received response can therefore have llm_call_performed: true without an accepted narrative.

Related pages

• Advisor configuration
• Prior sensitivity and advisor APIs
• Causal identification