Evaluation Workflows

FHOPS exposes deterministic playback tooling so you can inspect shift/day activity, idle capacity, mobilisation costs, and sequencing signals without leaving the CLI. This guide shows how to run the fhops eval-playback command and interpret its outputs.

Running deterministic playback

The playback command requires two inputs:

  • --scenario — path to the scenario YAML.

  • --assignments — CSV with machine_id, block_id, day, and optional shift_id and production columns. Any schedule exported by fhops solve-mip or fhops solve-heur is already in the expected format.

Example (building on the regression fixtures):

$ fhops solve-heur tests/fixtures/regression/regression.yaml --out tmp/regression_sa.csv
$ fhops eval-playback tests/fixtures/regression/regression.yaml \
    --assignments tmp/regression_sa.csv \
    --shift-out tmp/regression_shift.csv \
    --day-out tmp/regression_day.csv

The command prints two tables:

  • Shift Playback Summary — one row per machine/day/shift. Columns include production (m³), worked hours, idle hours (when --include-idle is used), mobilisation cost, and sequencing violation counts gathered during playback.

  • Day Playback Summary — day-level aggregation with production, total/idle hours, mobilisation totals, completed block count, and sequencing conflicts.

If you pass --shift-out or --day-out the same metrics are written to CSV files. The output schema matches the in-memory ShiftSummary and DaySummary dataclasses.

Optional flags

--include-idle emits rows for machine/shift combinations that were available but never assigned. This is useful when you want to inspect under-utilisation alongside productive shifts. Without the flag, only shifts that perform work are listed.

--shift-out and --day-out accept CSV paths. Folders are created automatically if they do not exist.

--kpi-mode toggles between basic and extended KPI summaries in the CLI output. The basic view focuses on production/mobilisation; the extended view includes utilisation, downtime, and weather metrics derived from the playback summaries.

Reporting templates

The repository ships with lightweight templates under docs/templates/ that you can use to stage KPI snapshots in Markdown/CSV reports. For example docs/templates/kpi_summary.md is a simple Markdown table containing placeholders such as {{ total_production }}, {{ uptime_ratio_mean_day }}, and {{ downtime_hours_by_machine }}.

Populate the template with the output of compute_kpis (or the CLI telemetry payload) to produce a shareable summary:

import pathlib
import re

import pandas as pd

from fhops.evaluation import compute_kpis
from fhops.scenario.contract import Problem
from fhops.scenario.io import load_scenario


def render(template: str, values: dict) -> str:
    """Replace ``{{ key }}`` placeholders (unknown keys become ``-``)."""
    return re.sub(r"\{\{\s*(\w+)\s*\}\}", lambda m: str(values.get(m.group(1), "-")), template)


pb = Problem.from_scenario(load_scenario("examples/tiny7/scenario.yaml"))
assignments = pd.read_csv("tests/fixtures/playback/tiny7_assignments.csv")
kpi_data = compute_kpis(pb, assignments).to_dict()

template = pathlib.Path("docs/templates/kpi_summary.md").read_text(encoding="utf-8")
pathlib.Path("tmp/tiny7_kpi_summary.md").write_text(render(template, kpi_data), encoding="utf-8")

You can embed the generated Markdown as-is in docs/notebooks or adapt the template to match your reporting format (CSV, HTML, etc.). A CSV variant lives alongside the Markdown template, so you can generate spreadsheet-friendly snapshots just as easily:

csv_template = pathlib.Path("docs/templates/kpi_summary.csv").read_text(encoding="utf-8")
pathlib.Path("tmp/tiny7_kpi_summary.csv").write_text(
    render(csv_template, kpi_data), encoding="utf-8"
)

Parquet and Markdown exports

Set --shift-parquet / --day-parquet to emit Parquet artefacts. These require pyarrow or fastparquet. The command fails early with a helpful message if neither backend is installed.

Use --summary-md to generate a Markdown digest containing topline metrics (sample count, total production, average utilisation) plus a preview table of the first 10 day-level rows. This is handy for dropping rich summaries into release notes or retrospective documents.

Quickstart example

$ fhops eval-playback examples/tiny7/scenario.yaml \
    --assignments tests/fixtures/playback/tiny7_assignments.csv \
    --samples 5 \
    --downtime-prob 0.1 \
    --weather-prob 0.2 \
    --landing-prob 0.3 \
    --shift-out tmp/tiny7_shift.csv \
    --day-out tmp/tiny7_day.csv \
    --shift-parquet tmp/tiny7_shift.parquet \
    --day-parquet tmp/tiny7_day.parquet \
    --summary-md tmp/tiny7_summary.md \
    --telemetry-log tmp/tiny7_playback.jsonl

The command prints rich tables to the terminal, writes CSV/Parquet/Markdown artefacts, and captures a JSONL telemetry record containing the same aggregate metrics written to disk.

Load the Parquet file, compute machine utilisation, and sanity-check totals:

import pandas as pd
from fhops.evaluation import machine_utilisation_summary, playback_summary_metrics

shift_df = pd.read_parquet("tmp/tiny7_shift.parquet")
day_df = pd.read_parquet("tmp/tiny7_day.parquet")

utilisation = machine_utilisation_summary(shift_df)
print(utilisation.filter(["machine_id", "total_hours", "utilisation_ratio"]).head())

metrics = playback_summary_metrics(shift_df, day_df)
print(f"Samples captured: {metrics['samples']}")
print(f"Total production units: {metrics['total_production']:.1f}")

The Markdown summary (tmp/tiny7_summary.md) contains topline metrics and preview tables. Open it in any Markdown viewer or drop it directly into release notes.

When you need a quick textual snapshot without leaving the CLI, pass --kpi-mode to the solver commands:

$ fhops solve-heur examples/tiny7/scenario.yaml --out tmp/tiny7_sa.csv --kpi-mode basic

The basic mode prints only production/mobilisation KPIs. Switch to --kpi-mode extended to include utilisation, downtime, and weather metrics in the CLI output.

Telemetry JSONL records can be ingested by automation scripts or dashboards. Each entry includes the scenario, sampling configuration, export paths, and summary metrics so playback runs are traceable.

Aggregation helper reference

The helper functions in fhops.evaluation.playback.aggregates expose stable DataFrame schemas that mirror the CLI exports:

  • shift_dataframe(result) — converts a deterministic PlaybackResult into a DataFrame with sample_id, availability, idle, mobilisation, and sequencing fields.

  • day_dataframe(result) — day-level aggregation with consistent column ordering.

  • shift_dataframe_from_ensemble(ensemble) / day_dataframe_from_ensemble(ensemble) — accept a stochastic EnsembleResult and stitch all samples (including the base result when requested) into a single DataFrame while preserving sample_id.

  • machine_utilisation_summary(shift_df) — groups shift-level data by machine/sample and reports total/available hours, production, mobilisation, and computed utilisation ratios. This is the fastest way to build custom utilisation charts.

  • export_playback(shift_df, day_df, ...) — shared serializer used by the CLI and telemetry code; it writes CSV/Parquet/Markdown outputs and returns the same summary metrics recorded in telemetry.

  • compute_kpis(...) returns a fhops.evaluation.KPIResult, a mapping that exposes scalar KPI totals while optionally attaching the canonical shift/day calendars. Use to_dict() when you need a JSON-serialisable payload or the helper with_calendars to bundle playback DataFrames.

  • compute_utilisation_metrics(shift_df, day_df) — helper under fhops.evaluation.metrics.aggregates that produces mean/weighted utilisation values plus per-machine and per-role breakdowns.

  • compute_makespan_metrics(problem, shift_df) — derives the latest productive day/shift (makespan) according to the scenario’s shift ordering; accepts fallback day/shift sets for deterministic/stochastic blends.

KPI formulas & required signals

The current KPI bundle includes:

  • total_production — delivered volume (m³): the part of Block.work_required delivered by each block’s terminal role during playback. production_units in the shift/day summaries is in the same units (m³, like ProductionRate.rate) but counts every role’s output.

  • completed_blocks — count of blocks whose remaining volume (work_required minus delivered m³) is zero after playback.

  • mobilisation_cost — total mobilisation spend accumulated in playback record metadata.

  • mobilisation_cost_by_machine / mobilisation_cost_by_landing — JSON mappings that expose cumulative mobilisation outlay by machine and landing.

  • sequencing_violation_* (when harvest systems are present) — counts and breakdowns derived from the heuristic/MIP sequencing checks captured during playback.

  • utilisation_ratio_mean_* / utilisation_ratio_weighted_* — average and weighted utilisation taken from the shift/day calendars, with optional breakdowns by machine or role.

  • makespan_day / makespan_shift — latest day/shift containing productive assignments according to the scenario’s shift definition order.

  • downtime_hours_total / downtime_event_count / downtime_hours_by_machine — aggregate downtime exposure derived from the sampled downtime durations (zero for deterministic runs).

  • downtime_production_loss_est — volume (m³) removed by downtime, summed per record from the volume the downtime event took out of each hit machine-shift (see below).

  • weather_severity_total / weather_severity_by_machine — cumulative weather intensity applied during stochastic playback, useful for correlating production drops with weather samples.

  • weather_hours_est / weather_production_loss_est — hours (severity × shift hours) and volume (m³) removed by weather, summed per record (see below).

Empty and partial plans

An empty assignment table (no rows, or only assigned = 0 rows — e.g. a MILP run that found no solution) is evaluated like any other plan, so it never looks complete:

  • total_production = 0; remaining_work_total and staged_production equal sum(Block.work_required) of the evaluated scenario (the carried-forward volume for a rolling window); completed_blocks = 0; makespan_day = 0 and makespan_shift = "N/A".

  • Day-level utilisation is 0 (every available day is idle); the shift, machine, and role utilisation keys are absent, as are the mobilisation, downtime, and weather keys.

  • Sequencing counts are 0 and sequencing_clean_blocks counts every harvest-system block.

For any plan (empty, partial, or complete) total_production is the playback delivered volume and total_production + remaining_work_total = sum(Block.work_required). Deterministic and stochastic playback (fhops.evaluation.run_playback(), fhops.evaluation.run_stochastic_playback()) report delivered_total = 0 and remaining_work_total = sum(Block.work_required) for an empty plan. FHOPS 1.0.0 reported the full scenario volume as delivered for an empty frame (#108). For rolling-horizon comparisons see Empty rolling plans and baselines.

Weather & downtime cost assumptions

The loss KPIs are emitted when compute_kpis() evaluates an assignment frame that carries stochastic event columns (for example a sample frame built by run_stochastic_playback()). Since 1.0.1 (#116) they are record-level sums:

  • downtime_hours_total sums the sampled downtime hours of every hit machine-shift (cancelled shifts count their full shift hours).

  • downtime_production_loss_est sums, over hit machine-shifts, the proposed volume the downtime event removed: the whole proposed volume for a cancelled shift, volume × d / shift_hours for a partial loss (the event records it in the private _downtime_lost column). Frames without that column fall back to rate(machine, block) × min(downtime_hours / shift_hours, 1).

  • weather_production_loss_est sums the proposed volume removed by weather (volume × severity, column _weather_lost; fallback production × severity / (1 − severity)), and weather_hours_est sums severity × shift_hours over weather-hit machine-shifts.

The losses are measured on each machine-shift’s proposed production before sequencing caps, so they include upstream (non-terminal) volume and do not equal the drop in total_production (which also includes knock-on sequencing effects and the other events). FHOPS 1.0.0 estimated both losses as hours × (delivered volume ÷ recorded hours of every role), which did not match the volume the events removed.

Shift hours (recorded hours_worked, shift availability used for utilisation, and the downtime fraction d / shift_hours) come from the matching timeline.shifts definition; without one, the machine’s daily_hours is divided by the number of shifts the machine has that day in the shift calendar (e.g. three shifts → 8 h each for a 24 h machine). Single-shift scenarios use daily_hours unchanged. FHOPS 1.0.0 recorded the full daily_hours for every shift of a multi-shift day.

Assignments must carry shift_id on scenarios with more than one shift per day; playback, KPIs, and fhops eval-playback raise an error instead of replaying every row as S1. Single-shift scenarios keep the S1 default.

Upcoming KPI extensions planned for Phase 3 will reuse the same shift/day summaries:

  • Weather/downtime penalties — additional cost categories driven by stochastic events.

  • Landing/system production breakdowns — richer summaries for dashboards/notebooks.

Before adding a new KPI ensure the required signal exists in either ShiftSummary or DaySummary. If a field is missing, extend the playback dataclasses first so both deterministic and stochastic flows emit the same schema and downstream KPIs remain reproducible.

These helpers are safe to use in notebooks, KPI pipelines, or automation scripts. The schemas are covered by regression tests so future changes will not silently break downstream consumers.

Stochastic playback toggles

The command also exposes stochastic options mirroring the API:

  • --samples — number of stochastic samples to evaluate (defaults to 1 for deterministic playback).

  • --downtime-prob / --downtime-max / --downtime-mean / --downtime-std — probability that a machine-shift is hit by downtime (or an exact number of hits per day with --downtime-max) and the Normal duration distribution in hours (defaults 4.0 / 1.5).

  • --weather-prob / --weather-severity / --weather-window — frequency, severity, and duration of weather-induced production reductions.

  • --landing-prob / --landing-mult-min / --landing-mult-max / --landing-duration — daily probability that a congestion shock starts at each landing, the multiplier range, and the shock length in days.

By default these probabilities are 0.0 so the command behaves deterministically unless you turn them on. Each sample’s shift/day summaries are concatenated in the exported CSVs, making it easy to aggregate or visualise variability across runs.

Stochastic event semantics

fhops.evaluation.run_stochastic_playback() copies the deterministic assignments for every sample, applies the enabled events in the order downtime → weather → landing shocks, and re-runs deterministic playback so sequencing caps still apply. Event effects compose multiplicatively on each assignment’s production (m³). Sample i uses numpy.random.default_rng(base_seed + i), shared by the events in that order, so a given seed and configuration always reproduce the same ensemble.

  • Downtime (DowntimeEventConfig). Days are visited in ascending order. On each day every eligible assignment (assigned > 0, optional target_machine_roles) is hit with probability; with max_concurrent set, exactly min(max_concurrent, n) of the day’s n eligible assignments are hit instead. Each hit samples a duration d ~ Normal(mean_duration_hours, std_duration_hours) clipped to [0, shift_hours] and multiplies the assignment’s production by 1 - d / shift_hours. A full-shift loss (d == shift_hours) cancels the assignment (assigned = 0, production 0). shift_hours is the matching timeline.shifts definition, otherwise the machine’s daily_hours divided by its number of shifts that day — the same hours deterministic playback records. The shift summaries report downtime_hours (the sampled hours, including cancelled shifts) and total_hours = shift_hours - d for affected shifts, so downtime_hours_total and utilisation reflect the sampled durations.

  • Weather (WeatherEventConfig). For each assignment day a spell starts with day_probability, picks a severity level, and covers impact_window_days consecutive days (overlaps keep the highest severity). Affected production is multiplied by 1 - severity. correlated_days is deprecated since 1.0.1: it never had an effect, and explicitly setting it to a non-default value (False) emits a DeprecationWarning (the default True, e.g. in older synthetic metadata.yaml files, does not warn). Use impact_window_days to model multi-day spells.

  • Landing shocks (LandingShockConfig). For each landing and each calendar day of the horizon a shock starts with probability, draws a multiplier uniformly from capacity_multiplier_range, and lasts duration_days calendar days from its start day. Every assignment on a block served by that landing has its production scaled by the multiplier on the affected days; overlapping shocks use the minimum multiplier. The expected fraction of landing-days under a shock is

    \[f = 1 - (1 - p)^{D}\]

    with p = probability and D = duration_days (days d < D at the start of the horizon use d instead of D). The default SamplingConfig (p = 0.1, D = 1, multipliers uniform on 0.4–0.8) shocks 10 % of landing-days with a mean multiplier of 0.6, i.e. an expected landing throughput loss of about 4 % over the horizon. The synthetic tier presets (fhops.scenario.synthetic.SAMPLING_PRESETS) were recalibrated for these semantics in 1.0.1: medium uses p = 0.025, D = 2 (f ≈ 4.9 %) and large uses p = 0.035, D = 3 (f ≈ 10.1 %); the 1.0.0 values (0.18/0.25) would shock 32.8 % / 57.8 % of landing-days. metadata.yaml files of the shipped examples/synthetic bundles record the presets in force when they were generated.

  • Event seeds. seed_offset (on every event config) is deprecated since 1.0.1 and ignored — it was never used. All events of sample i draw from default_rng(base_seed + i) in the order above; setting a non-zero seed_offset emits a DeprecationWarning.

Note

FHOPS 1.0.0 applied landing shocks to the first assignment rows of a shocked landing rather than to calendar days, always removed a whole shift on downtime (ignoring the duration settings), and dropped downtime-cancelled shifts from the summaries (so downtime_hours_total stayed at zero). Stochastic playback results from 1.0.1 onwards therefore differ from 1.0.0 for the same seed.

Relationship to KPI evaluation

fhops evaluate (existing command) still computes aggregate KPIs such as mobilisation cost and sequencing violations. fhops eval-playback complements it by surfacing the raw shift/day data used to compute those metrics. In future iterations the playback output will feed notebooks, stochastic sampling, and new KPI calculators documented here.