Data Contract Guide
This guide summarises the structured inputs FHOPS expects when authoring scenarios. It
builds on the examples/tiny7 and tests/fixtures/regression assets and reflects
recent extensions (mobilisation, geo metadata, crew assignments).
Reference dataset generator
All bundled scenarios (examples/{tiny7,small21,med42,large84}) now originate from a single
deterministic generator: scripts/rebuild_reference_datasets.py. The script samples Lahrsen-style
stand attributes, derives ADV6N7 skidding distances from block geometry, and emits the full CSV + YAML
bundle (blocks, machines, calendar, production rates, landings, mobilisation distances, README). Run:
python scripts/rebuild_reference_datasets.py tiny7 # or small21 / med42 / large84
Use --seed to reproduce specific variants. Regenerate the dataset before updating docs/fixtures so
the CSVs, README stats, and MILP bundles stay in sync across the ladder.
Core Tables (CSV)
Each scenario references a set of CSV files. Required columns and notes:
schema_version: Contract version tag (currently 1.0.0).
Table |
Required Columns |
Notes |
|---|---|---|
|
|
|
|
|
Optional: |
|
|
|
|
|
|
|
|
|
Cross References & Validators
The Pydantic models enforce consistency:
Blocks reference known landings. A block’s
harvest_system_idmust be a key of the scenario’sharvest_systemssection or of the default registry (fhops.scheduling.systems.default_system_registry(); see Harvest System Registry); scenario systems are overlaid on the defaults, so a scenario that defines its own systems can still use default ones. Enforced since 1.0.1 (FHOPS 1.0.0 never ran this check, see below).Calendar and production rates must reference defined machines/blocks and lie within the scenario horizon.
Mobilisation distances must reference known blocks; mobilisation parameters must reference known machines.
Crew assignments (optional) require unique crew IDs and valid machine IDs.
Locked assignments reference known machines/blocks, lie inside the horizon and inside the block’s
earliest_start/latest_finishwindow, avoid blackouts, and use known shift labels; on blocks with aharvest_system_idthe locked machine’s role must be one of that system’s roles. A machine may have one day-level lock or distinct shift-level locks per day.initial_stateentries reference known blocks/machines, carry non-negative values, use roles of the block’s harvest system, keep everyrole_remainingvalue at or below the block’swork_required, and pointlast_block_idat scenario blocks.
See Compatibility with FHOPS 1.0.0 for the inputs FHOPS 1.0.1 rejects that 1.0.0 accepted.
Note
Harvest systems the fleet cannot staff. A block with harvest_system_id can only be
worked by machines whose role is one of that system’s roles, and its volume is delivered only by
a machine with a terminal role (e.g. the loader). If no machine has any of the system’s roles,
no solver assigns a machine to the block (since 1.0.1, #158; earlier 1.0.1 pre-releases let
any machine work it in the heuristics); if no machine has a terminal role, upstream roles may
work but nothing is delivered. Both are valid scenarios, not errors: fhops validate prints a
Warning: block … line for each such block and exits 0, and the block’s volume counts as
leftover. Add machines with the missing roles or assign another harvest system.
Optional Extras
Recent helpers enable richer metadata:
MobilisationConfig— per-machine mobilisation costs and block distance matrices.GeoMetadata— optional GeoJSON paths and CRS tags for blocks/landings.CrewAssignment— map crew identifiers to machines/roles for downstream planners.TimelineConfig— shift definitions and blackout windows controlling daily availability.ScheduleLock— pre-assign specific machine/block/day (optionally day/shift) combinations (enforced by the operational MILP, the legacy MIP builder, and the heuristics).ScenarioInitialState— optional carried-in state (staged inventory, remaining role output, head-start shift counts, last block per machine) for resuming mid-operation; see Initial State (Resuming Mid-Operation).ObjectiveWeights— tweak solver objective weighting (production, mobilisation penalties, transition counts, optional landing-cap slack penalties).Round-the-clock operations — unless you override
machines.csvor the calendar, FHOPS expects continuous availability (24 h/day) and only pauses work when you add blackouts, downtime sampling, or calendar gaps.
ObjectiveWeights fields are optional; omit any you do not need. For example:
objective_weights:
production: 1.0
mobilisation: 0.5
transitions: 2.0
landing_surplus: 3.0
This configuration maximises production while penalising moves between blocks and soft landing
capacity violations. Setting a weight to 0 reverts to the default hard behaviour.
Reference tests/test_contract_validations.py for examples that exercise these validators.
Timeline Example
Add a top-level timeline block in your scenario YAML to describe shifts and blackouts:
timeline:
shifts:
- name: day
hours: 10
shifts_per_day: 1
blackouts:
- start_day: 5
end_day: 7
reason: wildfire risk
days_per_week: 5
The loader converts this into a TimelineConfig instance available via scenario.timeline.
Blackout windows are fleet-wide and inclusive: on every day from start_day to end_day no
machine may work any slot of that day (every slot of the problem’s shift grid on that day plus the
machine’s own shift_calendar shifts, also for machines without shift-calendar entries; days
without grid slots fall back to every timeline.shifts name, otherwise S1). The operational MILP treats these slots as unavailable
(availability A_{m,s} = 0), the heuristics never assign them, and locks may not fall inside
them. Playback does not cancel work scheduled in a blackout (for example, from an external plan);
it flags the record with blackout_hit.
Shift Calendars & Defaults
When you need per-machine shift availability (for example, night crews that only run certain
machines), add a shift_calendar CSV and reference it under data.shift_calendar:
machine_id,day,shift_id,available
H1,1,day,1
H1,1,night,0
H2,1,day,1
H2,1,night,1
shift_id values must match names declared in the timeline’s shifts section; the loader
rejects unknown labels or days outside num_days. Supplying multiple rows for the same
machine/day lets you model overlapping shifts with different availability flags, and you can
leave individual rows at available=0 to block a specific crew without editing the base
timeline. Day entries and shift entries combine with AND: a machine must be available in both
the day calendar and the shift calendar. When a shift_calendar is omitted entirely, FHOPS
synthesises a single shift named S1 for every day so legacy day-indexed scenarios continue to
work without edits.
Run fhops validate to confirm the calendar files and shift labels line up before solving:
fhops validate examples/tiny7/scenario.yaml
The CLI surfaces precise validation errors (missing columns, bad shift IDs, horizon overruns) so you can correct the dataset without spelunking through Pydantic stack traces.
Schedule Locking
Lock a machine to a block on a given day by adding locked_assignments:
locked_assignments:
- machine_id: YARDER1
block_id: B12
day: 5
Any attempt to reassign that machine/day is blocked in the operational MILP
(solve-mip-operational), the legacy MIP builder, the heuristics (SA/ILS/Tabu), and every
rolling-horizon window that covers the day (fhops plan rolling). A lock
without shift_id pins every available shift of that day to the block (all other blocks are
fixed to zero for that machine). Add shift_id to lock a single slot in multi-shift scenarios:
locked_assignments:
- machine_id: YARDER1
block_id: B12
day: 5
shift_id: night
shift_id must be one of the scenario’s shift labels, as returned by
fhops.scenario.contract.Scenario.shift_labels(): the shift_calendar labels when a shift
calendar is present, otherwise the timeline shift names, otherwise {"S1"} (the single synthetic
shift of day-indexed scenarios). Duplicate (machine, day, shift) locks and mixing a day-level
lock with shift-level locks for the same machine/day are rejected. Locks must reference known
machines and blocks, fall within 1..num_days and within the block’s
[earliest_start, latest_finish] window (latest_finish defaults to num_days), and avoid
timeline blackout days. On a block with harvest_system_id the locked machine must have a role
of that harvest system (a machine without a role is rejected too, because the operational MILP
cannot assign it to such a block); blocks without harvest_system_id accept any machine.
load_scenario applies exactly the same checks to YAML locked_assignments as to locks
passed to fhops.scenario.contract.Scenario in Python (FHOPS 1.0.0 skipped them for YAML
input).
Note
Scenario files that use shift_id on a lock require fhops>=1.0.1. FHOPS 1.0.0 silently
drops the field and treats the entry as a whole-day lock.
Initial State (Resuming Mid-Operation)
initial_state is an optional top-level YAML mapping (or the
fhops.scenario.contract.ScenarioInitialState model) that describes where operations stand
at day 1 of the horizon. Typical uses are rolling-horizon windows and re-plans after part of the
work is done; the rolling planner builds it for every window from the locked plan so far (see
Rolling-Horizon Planning). When it is omitted, the horizon starts from the same initial conditions as
in FHOPS 1.0.0: no staged inventory, zero role shift counts, each role may output the block’s full
work_required, and machines have no prior position (their first move is free). Results are not
guaranteed to match 1.0.0 even then, because 1.0.1 also fixes MILP/playback sequencing, blackouts in
the MILP, stochastic playback events, empty-plan KPIs, and rolling-horizon state (see
docs/releases/v1.0.1.md).
Note
Scenario files that use initial_state require fhops>=1.0.1. FHOPS 1.0.0 silently ignores
the section and plans from an empty state; schema_version stays 1.0.0.
initial_state:
blocks:
- block_id: B01
role_remaining: # m³ the role may still output on this block
feller_buncher: 0.0
grapple_skidder: 150.0
staged_inventory: # m³ output by the role, not yet consumed downstream
feller_buncher: 150.0
grapple_skidder: 60.0
role_shift_counts: # shifts already worked (informational)
feller_buncher: 4
grapple_skidder: 2
machines:
- machine_id: H3
last_block_id: B02 # block occupied in the last worked slot
Semantics:
Terminal volume still to deliver is
Block.work_requireditself (set it to the remaining volume; finished blocks can stay in the scenario withwork_required: 0so their ids remain valid for mobilisation distances andlast_block_id).staged_inventoryis keyed by the role that produced the volume. A downstream role can consume, at the start of the horizon, the minimum staged volume over its upstream roles (the single upstream value for linear chains). The operational MILP uses that value as the first-slotinventory_startand in the head-start check; the heuristics and playback sequencing tracker start their per-role inventories from it and apply the same head-start check (staged volume at the start of the slot against the buffer volume).role_remainingcaps each role’s cumulative output (the tracker, greedy seed, and repair use it as the initial remaining volume; the MILP enforcessum_s z[r,b,s] <= role_remaining). Omitted roles default towork_required.role_shift_countsis informational: it seeds the tracker’s shift counters (reported by the rolling-horizon carry-forward) but does not affect sequencing. Head-start buffers (role_headstart_shifts) are enforced as staged volume in every solver and in playback (see Harvest System Sequencing How-to).last_block_idmakes the first move away from that block cost mobilisation (and a transition) in the MILP objective (first slot), the heuristic score, and playback mobilisation KPIs.Role keys are normalised like machine roles (
Feller-Buncher→feller_buncher) and must be roles of the block’s harvest system; role-keyed state therefore requiresharvest_system_id.role_remainingvalues may not exceed the block’swork_required(tolerance 1e-6 m³): an upstream role has always output at least as much as the terminal role, so it cannot have more volume left than the block still has to deliver.
Note that the operational MILP charges the boundary move only when the machine works in the first
shift slot (consistent with its slot-to-slot transition variables), whereas the heuristics and
playback charge it on the machine’s first worked slot. The deprecated legacy builder
(fhops.optimization.mip.builder.build_model()) does not model initial_state and warns
when one is present; fhops solve-mip runs the operational MILP since 1.0.1.
Compatibility with FHOPS 1.0.0
schema_version is unchanged (1.0.0) and every 1.0.1 addition is optional, but files are
not interchangeable in both directions:
Files that need 1.0.1. Scenarios that use
initial_stateorlocked_assignments[].shift_idrequirefhops>=1.0.1. FHOPS 1.0.0 does not reject them: it ignoresinitial_stateand reads a shift lock as a whole-day lock, so it silently plans something else.Inputs 1.0.1 rejects at load. FHOPS 1.0.0’s
load_scenarioattached optional YAML sections without cross-validation, so it accepted these inputs; most 1.0.0 commands then rejected the invalid locks, duplicate crew ids, and superset distance tables later, when they built the internalProblem, whilefhops plan rollingran to completion with them (it dropped user locks). FHOPS 1.0.0 had no window, role, harvest-system, or initial-state checks at all. In 1.0.1load_scenario(and therefore every CLI command, includingfhops validate, the solvers,eval-playback, andfhops plan rolling) raises aValidationErrorat load for:locks on a timeline blackout day;
locks that reference an unknown machine or block, or a day outside
1..num_days;duplicate locks for the same machine/day (or day/shift), and a day-level lock combined with shift-level locks for the same machine/day;
locks with an unknown
shift_id(seeshift_labels());locks outside the block’s
earliest_start/latest_finishwindow (also rejected for scenarios built in Python);locks on a block with
harvest_system_idwhose machine has no role or a role outside that harvest system (also rejected in Python);mobilisation distance tables (
distance_csv) that mention blocks not in the scenario — a distance CSV covering a superset of the scenario’s blocks must be trimmed;crew assignments or mobilisation
machine_paramsthat reference unknown machines, and duplicate crew ids;invalid
initial_stateentries (unknown ids, roles outside the block’s harvest system,role_remainingabovework_required);blocks whose
harvest_system_idis neither in the scenario’sharvest_systemsnor in the default registry (also rejected in Python). FHOPS 1.0.0 had this check, but it never ran (Pydantic validatedblocksbeforeharvest_systemswas available), so such blocks loaded and the solvers treated them inconsistently: the operational MILP gave the unknown system no roles and could not harvest the block, while the heuristics and playback did not sequence it. Register the system underharvest_systems(inline mapping or adata.harvest_systemsYAML file) or use a registry id.
The 1.0.1 release was checked against every scenario in this repository and the companion BC case study scenarios: all load to the same validated
Scenarioas under 1.0.0, except the repository’s own regression fixture (tests/fixtures/regression/regression.yaml), whose customground_sequencesystem (feller → processor) was never registered; it is now defined in the fixture’sharvest_systemssection.
GeoJSON Ingestion & Distances
When supplying block or landing geometries:
Provide GeoJSON files with a
FeatureCollectioncontaining polygon features.Each feature must include an
idorproperties.idmatching the block/landing IDs defined in the CSV tables.Use a projected CRS suitable for distance calculations; FHOPS defaults to
EPSG:3005(BC Albers) but any metre-based CRS is acceptable when specified viageo.crs.Store relative paths in
GeoMetadata.block_geojson/landing_geojsonso the CLI tooling can locate the files.
To generate mobilisation distances from geometries, run:
fhops geo distances blocks.geojson --out mobilisation_distances.csv
The command computes centroid-to-centroid distances (in metres) respecting the CRS. The
resulting CSV aligns with the MobilisationConfig distance format and can be referenced
under scenario.data.mobilisation_distances.
Machine-Rate Defaults & Costing Helper
FHOPS includes a BC-focused rental-rate catalogue under data/machine_rates.json to keep
costing consistent across scenarios:
Each entry records
machine_name,role, owning/operating costs ($/SMH), default utilisation, move-in allowance, citation, and notes. Baseline values combine Dodson et al. (2015) Montana machine-rate data (converted at ~1.33 CAD/USD with diesel ≈ CAD 1.80/L) and Hartley & Han (2007) coastal grapple-yarders to match BC market conditions.Repair and maintenance allowances come from FPInnovations Advantage Vol. 4 No. 23 (2003). The report’s 2002 CAD regression averages are escalated to 2024 CAD using the Statistics Canada Machinery & Equipment CPI (Table 18-10-0005-01), giving a cumulative multiplier of ≈1.56. FPInnovations reports the allowances at cumulative usage classes of 0–5k, 0–10k, 0–15k, 0–20k, and 0–25k operating hours; we treat the 10 000 h bucket as the default because every machine class has data at that point and it best represents a “typical” fleet age. Newer machines (≤5 000 h) and end-of-life machines (≥15 000 h) sit in the neighbouring buckets, so supplying
repair_usage_hourslets you scale the allowance up or down using the published FPInnovations ratios when a scenario deviates from the baseline. These allowances remain optional per role.The CLI exposes the defaults via
fhops dataset estimate-cost:# Deterministic Lahrsen productivity with table defaults fhops dataset estimate-cost --machine-role grapple_yarder \ --avg-stem-size 0.45 --volume-per-ha 320 --stem-density 750 --ground-slope 35
The command prints the chosen role, source, owning/operating/repair split, utilisation, productivity (m³/PMH15), and $/m³. Add
--include-repair/--exclude-repairto toggle the FPInnovations allowance or--usage-hoursto select a different usage-class multiplier (nearest 5 000 h bucket from Table 2). Supplying--dataset <scenario.yaml> --machine <ID>auto-loads the machine’s role,operating_cost(when already populated), andrepair_usage_hoursso recurring reports always respect the dataset defaults. Override components directly via:fhops dataset estimate-cost --machine-role feller_buncher \ --owning-rate 95 --operating-rate 120 --repair-rate 40 \ --productivity 30 --utilisation 0.8
When you also need a road/subgrade allowance, append the TR-28 helper directly to the same command:
fhops dataset estimate-cost --machine-role grapple_yarder \ --productivity 22 --utilisation 0.85 \ --road-machine caterpillar_235_hydraulic_backhoe --road-length-m 150
Supplying
--road-machine(slug or name fromfhops dataset tr28-subgrade) together with--road-length-mprints a CPI-adjusted “TR-28 Road Cost Estimate” table after the machine-cost summary. Use--road-exclude-mobilisationwhen movement is covered elsewhere—the CLI warns in either case and cites the soil-protection guidance from FNRB3 (Cat D7H vs. D7G trial) and ADV4N7 (compaction thresholds). When the scenario already lists road jobs (see below),fhops.dataset estimate-cost --dataset …auto-selects the only row or lets you pick via--road-job-id; attach additional soil warnings by referencingdata/reference/soil_protection_profiles.jsonthrough--road-soil-profile(or via the CSV).
Road Construction Tables & Validation
To keep road-building metadata alongside the rest of the scenario, add an optional road_construction table:
id,machine_slug,road_length_m,include_mobilisation,soil_profile_ids RC1,caterpillar_235_hydraulic_backhoe,150,True,fnrb3_d7h|adv4n7_compactionReference the file under
data.road_constructioninscenario.yaml. Each row requires a uniqueid, a TR-28 machine slug (seefhops dataset tr28-subgrade), the road/subgrade length in metres, and whether the published mobilisation charge should be included by default.soil_profile_ids(pipe- or comma-separated) tie intodata/reference/soil_protection_profiles.jsonso the CLI can print structured reminders (ground-pressure multipliers, compaction thresholds, recommended mitigation). When a scenario contains exactly one row,estimate-cost --dataset ...pulls that entry automatically; specify--road-job-id RC1if multiple road jobs exist or pass--road-machine/--road-length-mto override everything from the command line.
Harvest-system templates now infer these rows from the skyline/cable presets they assign: any block that references a preset with
defined road defaults (see fhops.scheduling.systems.SYSTEM_ROAD_DEFAULTS) gains the matching TR-28 slug/length automatically,
and the synthetic dataset generator writes data/road_construction.csv alongside the other tables so downstream tooling picks up
the entries without manual edits. Run fhops validate after editing the table to catch duplicate IDs or malformed slugs before
passing the scenario to solvers or costing helpers.
Supplying
--rental-ratebypasses the lookup for bespoke studies, butmachines.csvrows should normally use the curated rates (or CLI recomputed totals) so costing/evaluation tools stay aligned.fhops dataset inspect-machine prints the machine metadata and the same default owning/operating/repair breakdown (honouring
repair_usage_hours) so you can audit scenario inputs without running a full cost estimate. You can also inspect the rental table directly via--machine-role(e.g.,fhops dataset inspect-machine --machine-role loader_barko450dumps the TN-46 Barko 450 live-heel loader rate). Historical Appendix II presets such asloader_cat966c_tr45,skidder_tr45, andbulldozer_tr45are exposed the same way so you can compare CPI-normalized 1979 CAD rates against modern entries. When you pair the command with a harvest system such asground_fb_loader_liveheel, the loader job automatically swaps its cost role toloader_barko450so budgeting outputs stay aligned with the TN-46 preset. Add--json-out machine.jsonto capture the same payload for automated QA pipelines. Solver telemetry (solve-heur,solve-ils,solve-tabu,eval-playback) automatically embeds thismachine_costsbundle whenever--telemetry-logis used, so KPI histories and dashboards can trace the assumed repair buckets.
Overriding the defaults follows two common paths:
Quick sensitivity checks – keep the built-in role but pass overrides to the CLI, e.g.
fhops dataset estimate-cost --machine-role grapple_yarder \ --owning-rate 185 --operating-rate 260 --repair-rate 50 \ --avg-stem-size 0.5 --volume-per-ha 280 --stem-density 700 --ground-slope 40
This keeps the source/role metadata but swaps in your costs for the report/run only.
Persistent dataset changes – clone
data/machine_rates.jsonor edit your scenario’smachines.csv. For the JSON table, add a new entry (e.g.,"role": "custom_grapple") and pointmachines.csv.roleat that slug; the loader will backfill operating_cost from your entry. Alternatively, write the full $/SMH intomachines.csv.operating_costfor the specific machine if the rate is project-specific. Both approaches play nicely with the CLI helpers and solver defaults.
When authoring datasets set machines.csv.operating_cost to the all-in rental rate ($/SMH)
for each machine/system. Use the CLI helper (deterministic or distribution-based) to turn
Lahrsen stand descriptors and machine rates into comparable $/m³ for QA, reporting, or solver
inputs.
Skyline Stand Profiles
Arnvik (2024) Appendix 5 stand metadata is bundled under
data/reference/arnvik/appendix5_stands.json and exposed via the CLI:
fhops dataset appendix5-standslists authors/species and the parsed slope/ground descriptors (filter with--author). The parser converts ranges (e.g.,23 (5-55)%) and qualitative labels (Level - Erasmus) into approximate slope percentages.fhops dataset estimate-cable-skiddingevaluates the Ünver-Okan (2020) uphill skidding regressions (spruce stands in north-east Turkey). Supply either--slope-percentor a stand--profile(author name) to pull slope defaults directly from Appendix 5, and treat the results as non-BC placeholders unless you calibrate them.fhops dataset estimate-skyline-productivitywraps the Lee et al. (2018), TR-125, and TR-127 regressions, with optional--tr119-treatmentmultipliers (strip cut / retention levels) so partial-cut scenarios automatically apply the published BC productivity and cost offsets. Lee et al. (2018) regressions are small-scale tethered yarder studies from South Korea (HAM300), while TR-127 block-specific models (tr127-block1 … block6) use the Appendix VII coefficients from the FPInnovations case studies in northwestern BC. TR-112 only publishes descriptive productivity tables, so no regression helper is available for that report yet.Programmatically,
fhops.reference.arnvik_appendix5exposes dataclasses and helper functions so skyline helpers and costing workflows can bind stand descriptors to the new cable productivity models.
Log Hauling Extension
schema_version: "1.1.0" may add an optional hauling: section with log sorts, contractors,
truck types and trucks, depots, sort yards, supply, demand, haul arcs, loader bindings, and a
driver policy. The section can be inline or referenced from data.hauling. Existing 1.0.0
scenarios remain valid and have no hauling layer; a 1.0.0 file that contains hauling: is
refused so older readers cannot silently ignore it.
See Log Hauling for the entity rules, units, and a complete example.
LidarBC Products
Scenarios may optionally pin FEMIC-compiled LidarBC bundles and record aggregated block/landing/ road support evidence. Both sections are optional; existing scenarios without them remain valid. FHOPS consumes the FEMIC bundle and never discovers or materializes source tiles. See FEMIC LidarBC Products for the bundle, pin, support, aggregation, and OGL-BC attribution contract.
Tree Inventories
Scenarios may optionally reference a validated individual-tree inventory in the
top-level tree_inventory section. The inventory is referenced, not copied;
the pin records the CSV and metadata checksums, CRS, review state, producer,
method, tree count, and block memberships. See
Tree Inventory Contract for the contract, registration, block map,
and reconciliation rules.
Fixture Gallery
tests/data/minimal— smallest possible scenario for smoke-testing the loader.tests/data/typical— multi-block example with mobilisation distances and harvest system IDs.tests/data/invalid— intentionally malformed inputs that surface validation errors.
See also Quickstart for CLI commands and tests/fixtures/regression for a
mobilisation-aware example.