Signed Experiment Campaigns

rle.reusable.campaigns turns a small declarative matrix into immutable, source-pinned execution units. It is designed for benchmark sweeps where a missing run must stay missing, two methods must use the same logical seeds, and an interrupted job must never look complete.

Start with the checked-in examples/reusable/campaigns/smoke.json definition, then materialize it from a clean checkout:

uv run python scripts/experiment_campaign.py materialize \
  examples/reusable/campaigns/smoke.json \
  --output exports/campaigns/smoke/resolved.json

The command records the current Git revision and refuses a dirty checkout by default. --allow-dirty is available for exploratory work and marks a dirty revision explicitly. A resolved manifest contains one signed run for every environment, algorithm, and logical-seed combination. Construction, learner, and evaluation seeds are derived in separate deterministic domains; algorithms on the same environment and logical seed receive the same domain seeds.

Running One Unit

Scheduler array tasks can select a run by ID:

uv run python scripts/experiment_campaign.py list exports/campaigns/smoke/resolved.json
uv run python scripts/experiment_campaign.py show \
  exports/campaigns/smoke/resolved.json \
  smoke--example_env--ippo--seed-00

The project-owned launcher then uses the lifecycle API:

from rle.reusable.campaigns import (
    campaign_suite_config,
    finalize_experiment_campaign_run,
    prepare_campaign_run,
    resolve_campaign_run,
)
from rle.reusable.experiments import run_experiment_suite

run = resolve_campaign_run(manifest, run_id)
state = prepare_campaign_run(run)
if state != "complete":
    result = run_experiment_suite(
        campaign_suite_config(run),
        project_algorithm_specs,
    )
    finalize_experiment_campaign_run(run, result)

prepare_campaign_run refuses unsigned files or metadata from a different resolved run. Resolved configuration mappings are deeply immutable, and the adapter rejects environment, common, algorithm, seed, or identity overrides. Campaign suites also replace AlgorithmSpec.config defaults with the signed algorithm configuration instead of merging undeclared defaults into the run. The run hook receives every declared domain through context.seeds; the learner-domain value remains available as context.seed.

Finalization hashes the run record, every tracked trainer artifact, and the complete per-run checkpoint directory. Status and reporting re-check the signature, source revision, file size, and SHA-256 digest. Artifact paths cannot escape the run root or traverse symbolic links, and an existing completion closure cannot be replaced with a smaller one.

For an intentional rerun, call prepare_campaign_run(run, restart=True). The whole completed run directory is renamed atomically beneath _archived_attempts/<run_id>/attempt_NNNN before a fresh run root is prepared; restart never removes only complete.json or resumes the old trainer record.

Reporting And Archival

Generate a row-complete status bundle with JSON, Markdown, and optional CSV:

uv run python scripts/experiment_campaign.py report resolved.json \
  --json-output status.json \
  --markdown-output status.md \
  --csv-output status.csv \
  --require-complete

uv run python scripts/experiment_campaign.py inventory resolved.json \
  --output artifact-inventory.json

--require-complete fails if any resolved unit is pending, partial, corrupt, or has a changed artifact. It never substitutes zero for a missing result. The artifact inventory is itself signed and is suitable for an archive index.

For paired comparisons, exact_paired_binomial_test covers binary outcomes, exact_paired_wilcoxon_test covers up to 24 non-zero paired differences, and holm_adjust_p_values applies family-wise error correction. Callers should pre-register the outcome, direction, exclusions, and family before using these generic calculations.

Scheduler Accounting

Write campaign_run_id=<signed-run-id> near the start of every retained Slurm or PBS log. The accounting adapters join that marker to scheduler output rather than guessing from job order:

uv run python scripts/experiment_campaign.py slurm-accounting resolved.json \
  --log-root logs \
  --accounting-input sacct.psv \
  --site cluster-a \
  --json-output accounting.json \
  --markdown-output accounting.md

Slurm input is sacct --parsable2 data. PBS input is qstat -f -F json data. Reports preserve attempt-level state, elapsed time, memory, CPUs, queue or partition, node list, and signed campaign identity.

Adapting To Your Project

Copy the campaigns package, the CLI, and one definition. Keep environment builders and algorithm registrations in the owning project, then translate a resolved unit with campaign_suite_config. Add new seed domains when another independent random stream matters; never reuse the learner seed implicitly. Treat schema or scientific-config changes as a new campaign_revision, retain the resolved manifest beside reports, and keep completion validation in the archive or publication pipeline.

The SHA-256 fields used here are deterministic content identities and corruption checks, not keyed authentication. Load pickle-like trainer payloads only from trusted campaigns.