Timings

RL Engine's shared timings helper lives in src/rle/reusable/timings and is exported from rle.reusable. It records durable stage and run durations in timings.json for experiments, notebooks, and small scripts.

As a reusable component, timings gives a new paper repo a simple convention for leaving evidence about how long stages took. It is deliberately a durable record helper, not a profiler.

Use it when a run should leave a compact timing record that can be loaded, summarized, and compared later.

Quick Start

from rle.reusable import load_timings, save_timings, timed

timings = load_timings("exports/run-001")

with timed(timings, "ippo_train", run_idx=0):
    train_policy()

save_timings(timings, "exports/run-001")

The canonical files are:

exports/run-001/timings.json
exports/run-001/timings_hardware.json

timings.json keeps durations and a short hardware_id on each run. timings_hardware.json stores each full hardware description once.

Public API

Every timing stage must contain a runs list, and every run must carry a hardware_id. Removed coarse RAM fields are rejected so malformed or outdated records cannot silently enter a current experiment.

load_timings() and save_timings() accept either a directory or a file-like artifact path. Passing exports/run-001/bundle.np still reads or writes exports/run-001/timings.json.

save_timings() also normalizes common notebook values into JSON-safe forms: Path, NumPy scalars and arrays, tuples, sets, and dictionaries with non-string keys. Unsupported values raise TypeError.

Concurrent Writes

Timing writes are safe for concurrent API callers. save_timings() and update_timings() acquire a sibling timings.json.lock file, write JSON to a temporary file in the same directory, then atomically replace timings.json. Readers see complete JSON snapshots.

Lock acquisition waits up to TIMINGS_FILE_LOCK_TIMEOUT_SECONDS, currently 15 minutes, before raising TimeoutError. This bounds wedged live writers without depending on stale lock-file cleanup: when a writer process exits or crashes, the operating system releases the underlying file lock automatically.

The default save_timings() behavior merges the incoming timing map with the current file while holding the lock. Run entries are de-duplicated by run_idx plus label, by label alone when no run_idx exists, and otherwise by canonical JSON content. Pass merge_existing=False for an exact locked replacement.

Use update_timings() when the write depends on the current file:

def add_elapsed(timings):
    record_timing(timings, "ippo_train", 1.25, run_idx=0)

timings = update_timings("exports/run-001", add_elapsed)

Entries without run_idx or label have no stable identity, so identical unindexed entries may collapse during a merge.

Recording

Use explicit records when a duration is already available:

record_timing(timings, "checkpoint_save", 0.42, run_idx=0)
record_timing(timings, "nashconv", 12.8, label="final")

You do not have to use with to add a timing entry. timed() is still the recommended path when RL Engine should measure a whole block for you; record_timing() is the direct path when you already have the elapsed seconds.

from time import perf_counter

from rle.reusable import record_timing

start = perf_counter()
train_policy()
record_timing(timings, "ippo_train", perf_counter() - start, run_idx=0)

Use timed() for a measured block:

with timed(timings, "alpharank"):
    compute_alpharank()

For callback-style APIs or notebook flows where a with block is awkward, timed() also returns a small manual timer:

timer = timed(timings, "ippo_train", run_idx=0)
timer.start()
train_policy()
elapsed = timer.stop()

stop() records the entry and returns the elapsed seconds. Manual timers are strict: stopping before starting, starting twice, or stopping twice raises RuntimeError rather than silently adding misleading records. Like record_timing(), manual timers update only the in-memory map; use save_timings() or update_timings() to persist JSON.

Recording rules:

• run_idx=None replaces the stage with a single entry
• run_idx=0 resets the stage and starts a run series
• later run_idx values append to the existing run series

timed() records in a finally clause. If the wrapped block raises, the timing entry is kept and the original exception is re-raised.

JSON Records

Timing files remain compact:

{
  "ippo_train": {
    "runs": [
      {
        "hardware_id": "hw_3d836072b1cf2b24",
        "run_idx": 0,
        "seconds": 1.25
      }
    ],
    "total_seconds": 1.25
  }
}

The sibling hardware file records the job-visible allocation:

{
  "hardware": {
    "hw_3d836072b1cf2b24": {
      "memory": {"limit_gib": 64.0, "limit_source": "cgroup"},
      "cpu": {
        "processor_name": "AMD EPYC 7742 64-Core Processor",
        "physical_cores_available": 4,
        "logical_threads_available": 8,
        "min_clock_mhz": 1500.0,
        "max_clock_mhz": 3400.0
      },
      "gpus": [
        {
          "count": 1,
          "memory_gib": 15.0,
          "name": "NVIDIA A16",
          "peak_memory_bandwidth_gb_s": 200.03
        }
      ]
    }
  },
  "schema_version": 1
}

Memory is the finite cgroup limit when available, with Slurm allocation variables as a fallback. It is not the physical node total. CPU counts come from the process affinity mask, so they describe accessible physical cores and logical threads rather than every processor on the node. Slurm and PBS metadata are supported, and CPU-only jobs use an empty gpus list. GPU records include the visible model, VRAM, bus width, maximum memory clock, and theoretical peak memory bandwidth when NVIDIA's management library exposes them.

Summary Notebook

Use notebooks/reusable/generic/timings_mo.py to inspect timing sidecars across export directories produced by the reusable experiment runner. It expects the canonical reusable layout:

exports/<group>/<algorithm_slug>/timings.json
exports/<group>/runs/<algorithm_slug>_<run_idx>.np

The notebook discovers groups from the parent export directory and algorithms from child directories that contain timings.json. When matching csh-compatible visualisation run archives exist, it uses their algo display label; otherwise it falls back to a prettified slug.

It reports <algorithm_slug>_run stages as mean ± 2 std s with run IDs, allocation memory, CPU topology and clocks, GPU details, file paths, and warnings when selected runs differ in hardware. Additional stages can be included with the Other stages checkbox or --include-other-stages true.

uv run marimo run notebooks/reusable/generic/timings_mo.py

For a quick script-mode report:

uv run notebooks/reusable/generic/timings_mo.py \
  --export-root exports \
  --groups all \
  --algorithms all \
  --latest-runs 3

Lock Status Notebook

Use notebooks/reusable/generic/lock_status_mo.py when you need to inspect lock files under an export root. The notebook scans matching *.lock files, groups them by the reusable export layout, and checks live status by attempting a non-blocking Unix flock compatible with RL Engine's internal lock helper. Lock-file text, when present, is shown only as diagnostic text; the file contents are not treated as the source of truth.

uv run marimo run notebooks/reusable/generic/lock_status_mo.py

For a terminal report of currently held timing locks:

uv run notebooks/reusable/generic/lock_status_mo.py \
  --export-root exports \
  --groups all \
  --algorithms all \
  --pattern timings.json.lock \
  --only-locked true

Copying To Another Project

Copy src/rle/reusable/timings and src/rle/internal/locks.py into the new package, then update the import path if the package namespace changes. GPU identity and VRAM use nvidia-smi; memory-bus and clock details use the system NVML library when available. Neither is required for CPU-only use.

The module writes ordinary JSON files named timings.json and timings_hardware.json. Project-specific notebooks and scripts can therefore share the timing convention without sharing algorithm code.

CPU ranges carry a clock_source. Physical nodes normally use linux_cpufreq. When ACPI CPPC exposes an autonomous boost ceiling above the cpufreq P-state maximum, the range is extended with the capability-derived maximum and uses linux_cpufreq_cppc. A virtual machine without either physical interface may use proc_cpuinfo_virtual_nominal only when all visible vCPUs advertise a known hypervisor TSC frequency and agree within 0.1 MHz; the fixed guest clock is then reported with equal minimum and maximum values. Varying or bare-metal /proc/cpuinfo readings are not treated as limits. CPU names remain the guest-visible model and may therefore be generic rather than the physical host SKU.

Adapting To Your Project

Copy src/rle/reusable/timings together with the internal lock helper. The timing helper is useful even without the reusable experiment runner, and the generic timing notebook expects timing sidecars plus optional csh-compatible visualisation archives:

exports/<group>/<algorithm_slug>/timings.json
exports/<group>/runs/<algorithm_slug>_<run_idx>.np

If you keep that shape, notebooks/reusable/generic/timings_mo.py can discover groups and algorithms automatically. The only project-specific choices are the export root and any algorithm filters passed through the UI or CLI:

uv run notebooks/reusable/generic/timings_mo.py \
  --export-root exports \
  --groups my_env/main \
  --algorithms ippo,icpo

The reusable experiment runner already writes one default stage per algorithm:

<algorithm_slug>_run

Add extra timing stages from project code when they are meaningful enough to compare later. Common examples are contract_synthesis, world_model_load, checkpoint_save, evaluation, or plot_export. Use update_timings(...) when several processes or notebook cells may touch the same file:

from my_project.reusable import record_timing, update_timings

def add_eval_timing(timings):
    record_timing(timings, "evaluation", elapsed, run_idx=run_idx)

update_timings(context.algorithm_dir, add_eval_timing)

The blocks most likely to change in a project are:

• stage names, which should be stable enough for notebooks and CI checks to recognize;
• where timing files are written, usually the algorithm export directory;
• whether additional stages are recorded inside AlgorithmSpec.prepare, inside the seeded run hook, or in post-processing notebooks;
• whether the timing notebook defaults to a project-specific export_root.

Do not put trainer-specific profiling data into timings.json. Keep this file coarse and durable: stage names, elapsed seconds, run indices, labels, and hardware IDs. Detailed profiler traces should remain project-owned.