Runner Overview

Use the pipeline runner when you want a full disk-backed PanelMMM run instead of only an in-memory fit.

The runner loads a YAML config and a CSV dataset, builds the model, executes a fixed stage sequence, and writes each stage’s artefacts into a structured run directory. When validation is enabled, the runner performs a second train-window fit for the blocked holdout stage, so the run takes longer than a pure full-sample fit.

If you want a quick first run, start with Quickstart: Pipeline Runner.

Public entry points

The public Python API is:

• abacus.pipeline.PipelineRunConfig
• abacus.pipeline.run_pipeline
• abacus.pipeline.PipelineRunResult

The thin CLI wraps the same code path:

python -m abacus.pipeline.runner --config path/to/config.yml

Terminal output

Both CLI entry points show a framed run summary, the model specification and a separate heading for each stage. An interactive terminal shows an activity indicator and elapsed stage time. This is elapsed time, rather than an estimate of the remaining sampling time. Completed stages show their duration and the number of retained artefact entries.

Warnings and failures appear in separate sections. The final summary lists the run directory, manifest and retained review files. Stage completion records execution status. It does not establish model adequacy or causal identification.

The display adapts to terminal width. Redirected output uses plain text without colour or animation. Set NO_COLOR=1 to disable terminal colours. Use --quiet to show the final outcome, warnings and output paths. Use --verbose to retain routine backend logs. The two flags are mutually exclusive.

The Python API accepts an optional reporter. To use the same display, pass a TerminalPipelineReporter from abacus.pipeline.terminal. Without a reporter, run_pipeline(...) retains its basic stage messages. Model graph construction logs the estimator summary at INFO level through abacus.mmm.panel; it does not print a duplicate summary to standard output.

Basic Python example

from pathlib import Path

from abacus.pipeline import PipelineRunConfig, run_pipeline

result = run_pipeline(
    PipelineRunConfig(
        config_path=Path("data/demo/geo_panel/config.yml"),
        output_dir=Path("results"),
        run_name="geo_panel_baseline",
        prior_samples=10,
        draws=500,
        tune=500,
        chains=2,
        cores=2,
        random_seed=42,
        curve_samples=100,
        curve_points=100,
    )
)

print(result.run_dir)
print(result.manifest_path)

PipelineRunResult contains:

What the runner does

run_pipeline(...) performs these steps:

1. Load the YAML config with load_yaml_config(...).
2. Load X and y from CSV using load_pipeline_data(...).
3. Merge CLI sampler overrides with YAML fit through build_model_kwargs(...).
4. Create the output directory tree and initialise run_manifest.json.
5. Run the retained stages in order, updating the manifest after every stage.

Stage 00 stores a PanelMMM instance in the shared PipelineContext. For unlabelled models, it builds the graph immediately. For named estimators, it prepares the model and defers graph construction until Stage 10. A callback after Stage 00 must therefore not assume that the graph exists.

Runner-only roots such as prior_sensitivity, ai_advisor, diagnostics, and validation stay on the pipeline context and are stripped before the public MMM builder validates the model YAML.

Stage order

The runner uses a fixed stage list.

The prior-sensitivity stage is marked skipped when the YAML config does not contain prior_sensitivity or it is disabled. The AI advisor stage follows the same convention for ai_advisor. The diagnostics advisor stage runs by default for enabled ai_advisor blocks and is marked skipped only when ai_advisor is absent, disabled, or has diagnostics_review_enabled: false. The validation stage is marked skipped when the YAML config does not contain validation or it is disabled. The optimisation stage is also optional; it returns None and is marked skipped when the YAML config does not contain an optimization block.

See Output Directory Schema for the stage folders and artefact layout.

Data and model assumptions

The retained runner is designed around PanelMMM.

• The flow-oriented public YAML is expected to describe a PanelMMM.
• The data loader reads CSV only.
• Later stages call PanelMMM plotting, summary, diagnostics, and optimisation methods directly.

If you need the exact YAML keys, see YAML Configuration.

PipelineRunConfig

PipelineRunConfig controls runtime settings that sit outside the YAML model specification.

Only sampler settings are merged into model construction. Other overrides are used by the runner itself during data loading, holiday resolution, diagnostics reporting, and output setup.

Run directory naming

The runner creates the run directory as:

<output_dir>/<effective_run_name>_<YYYYMMDD_HHMMSS>

The timestamp is generated in UTC.

All stage directories are created up front, even if a later stage is skipped or the run aborts.

Failure and skip behaviour

If a stage raises an exception:

• the current stage is marked failed
• the run manifest is marked failed
• all still-pending later stages are marked not_reached
• run_pipeline(...) re-raises the exception

If a stage returns None:

• the stage is marked skipped
• the manifest warning records that no configuration was supplied for that optional stage

Reporter hook

run_pipeline(...) accepts an optional reporter that implements the PipelineReporter protocol.

The reporter can observe:

• pipeline start
• stage start
• stage end
• pipeline end
• pipeline failure

See Extending the Runner for the callback contract.