Synthetic Datasets
==================

FHOPS ships with a small library of synthetic scenarios that mirror the structure of our teaching
and benchmarking datasets. Each bundle is a self-contained directory under
``examples/synthetic/`` with a ``scenario.yaml`` and the CSV tables that back it. The bundles are
generated by :func:`fhops.scenario.synthetic.generate_random_dataset` so they follow the same data
contract validation as user-supplied scenarios.

Reference bundles
-----------------

Three ready-to-use bundles are versioned with the project. The table below summarises their scale;
see ``examples/synthetic/metadata.yaml`` for the full metadata (random seeds, blackout windows, and
shift calendars).

.. list-table:: Reference bundle statistics
   :header-rows: 1
   :widths: 15 12 12 12 12 12 25

   * - Bundle
     - Blocks
     - Machines
     - Landings
     - Days
     - Shift slots/day
     - Seed / Notes
   * - ``small``
     - 4
     - 2
     - 1
     - 112
     - 1
     - Seed ``101`` — no blackouts; fleet ``cable_micro_hi_skid`` (hand faller → Hi-Skid yarder).
   * - ``medium``
     - 8
     - 4
     - 2
     - 112
     - 1
     - Seed ``202`` — short blackout windows; fleet ``steep_tethered`` (tethered harvester →
       tethered shovel/skidder → processor → loader).
   * - ``large``
     - 16
     - 6
     - 3
     - 112
     - 1
     - Seed ``303`` — extended blackout periods; fleet of nine cable systems sharing six roles
       (faller, grapple yarder, processor, loader, hand faller, Hi-Skid yarder).

The bundles are written by ``fhops synth generate --tier <tier> --overwrite`` (tier presets in
``fhops.cli.synthetic.TIER_PRESETS``: 112 days, one ``S1`` slot per day). The same commands, with
the seeds above, produce the SoftwareX synthetic tiers under ``docs/softwarex/assets/data/``.

Fleet roles and harvest systems
-------------------------------

When harvest systems are supplied (``fhops synth`` passes the default registry),
:func:`~fhops.scenario.synthetic.generate_random_dataset` builds the fleet from them (since 1.0.1,
#158): the systems of the tier's ``system_mix`` are visited in a weighted random order and kept
while the distinct roles of the kept systems fit in the sampled machine count; machines take those
roles round-robin (upstream roles first) and each block draws its harvest system from the kept
systems by mix weight. Every block can therefore be worked and delivered (block windows can still
be too short for a long role chain). ``metadata.yaml`` records the choice as ``fleet_systems`` and
``fleet_roles``. Before #158 the machines kept the ``role_pool`` roles (``harvester`` /
``forwarder``), which no system of the tier mixes uses, so the synthetic tiers delivered no volume.
Without ``systems`` the generator keeps ``role_pool`` roles and blocks have no harvest system.
:func:`~fhops.scenario.synthetic.generate_with_systems` (the deterministic spec helper) still
assigns every registry system round-robin and gives machines roles from all registry systems, so
most of its blocks have no machine of their system's roles (``fhops validate`` warns about them);
use ``generate_random_dataset`` for deliverable system-aware datasets.

The example README (``examples/synthetic/README.md``) gives a short overview and links back to the
metadata file if you need exact counts or blackout windows for reproducibility.

Block enrichment, crews, and blackouts
--------------------------------------

Each bundle ships with richer signals to support evaluation/benchmarking:

* ``blocks.csv`` now contains ``terrain`` and ``prescription`` columns sampled from tier-aware pools
  (e.g., ``gentle``/``mixed`` for the small instance, ``steep``/``snow`` for the large one).
* ``machines.csv`` adds a ``crew`` column; the generated ``crew_assignments.csv`` table records the
  crew → machine mapping with capability notes so validators and scenario loaders can reconstruct
  assignments.
* Crew capabilities are sampled from the ``capability_pool`` while respecting the tier-specific span
  (single-skill crews in ``small``, multi-skill teams in ``large``).
* Blackout windows are produced with tier defaults: zero-probability for ``small`` bundles, short gaps
  for ``medium``, and multi-day outages for ``large``. Metadata captures the sampled windows so smoke
  tests can diff future changes.
* Weighted terrain/prescription distributions and blackout bias windows can be overridden via
  ``SyntheticDatasetConfig`` (see ``terrain_weights`` / ``prescription_weights`` and ``blackout_biases``)
  to concentrate sampling on seasonal or topographical hotspots. Harvest-system mixes are also recorded
  when bundled systems are supplied.

If you regenerate the library, the metadata YAML under each tier (and the aggregate metadata file in
``examples/synthetic/``) records the terrain/prescription mix, crew capabilities, blackout windows,
and the seed that produced the dataset.

CLI walkthrough
---------------

Use the bundles anywhere a normal scenario is accepted. Typical entry points:

* Validate the data contract:

  .. code-block:: bash

     fhops validate examples/synthetic/medium/scenario.yaml

* Run the simulated annealing baseline and capture the schedule:

  .. code-block:: bash

     fhops solve-heur examples/synthetic/medium/scenario.yaml --out /tmp/medium_sa.csv --seed 99

* Generate shift/day summaries with KPI exports:

  .. code-block:: bash

     fhops eval-playback examples/synthetic/medium/scenario.yaml \
                         --assignments /tmp/medium_sa.csv \
                         --shift-out /tmp/medium_shift.csv \
                         --day-out /tmp/medium_day.csv \
                         --summary-md /tmp/medium_playback.md

* Benchmark solver presets across the reference datasets (long-running: includes a 1800 s MILP
  solve; add ``--no-include-mip`` for heuristics only):

  .. code-block:: bash

     fhops bench suite --scenario examples/synthetic/large/scenario.yaml \
                       --operator-preset explore \
                       --out-dir tmp/bench_synth_large

* Generate a fresh bundle (preview or write to disk):

  .. code-block:: bash

     fhops synth generate tmp/synth_medium_custom --tier medium --seed 321 --blocks 10:14 --preview

  Drop ``--preview`` (and optionally add ``--overwrite``) to write the generated CSV/YAML bundle. The
  command accepts YAML/TOML configs via ``--config`` when you need full control over the
  ``SyntheticDatasetConfig`` fields.

  Common flags you can mix in:

  - ``--machines 6:8`` / ``--landings 2`` / ``--days 10:14`` to resample sizes.
  - ``--shifts-per-day 2`` to simulate multi-shift calendars.
  - ``--machine-daily-hours 20`` when you want the generated ``machines.csv`` to reflect partial-day
    availability (default is 24, matching the data-contract baseline).

  .. note::
     When you write directly to ``examples/synthetic/<tier>`` the CLI automatically refreshes the aggregate
     ``examples/synthetic/metadata.yaml`` so documentation and automation stay in sync.

* Create several bundles via a batch plan:

  .. code-block:: bash

     fhops synth batch docs/examples/synthetic_batch_plan.yaml --overwrite

  The plan file can be YAML/TOML/JSON and contains a list of entries, each mirroring the options
  accepted by ``fhops synth generate`` (``tier``, ``seed``, ``config``, overrides, etc.). This is handy
  when preparing datasets for experiments or CI warmups. See
  ``docs/examples/synthetic_batch_plan.yaml`` (and the referenced ``synthetic_batch_custom.yaml``) for
  a ready-to-run example.

These commands reuse the same CLI surfaces already documented in :doc:`evaluation` and
:doc:`../reference/cli`, but the synthetic bundles keep the inputs lightweight enough for quick
iteration and teaching exercises.

The loader automatically ingests ``crew_assignments.csv`` when present, so downstream analytics can
group KPIs by crew. Pair the bundle metadata with playback exports to slice utilisation or downtime
by terrain/prescription class.

Regenerating bundles
--------------------

The generator can be scripted to refresh or extend the library. Seeds are recorded in
``metadata.yaml`` so you can rebuild a bundle exactly:

.. code-block:: python

   from pathlib import Path

   from fhops.scenario.synthetic import SyntheticDatasetConfig, generate_random_dataset

   config = SyntheticDatasetConfig(
       name="synthetic-medium",
       tier="medium",
       num_blocks=(8, 10),
       num_days=12,
       num_machines=4,
       num_landings=2,
       shifts_per_day=1,
       shift_hours=(9.5, 10.5),
       terrain_pool=["rolling", "mixed", "steep"],
       crew_pool=["crew-alpha", "crew-beta", "crew-gamma"],
       capability_pool=["harvester", "forwarder", "processor"],
       crew_capability_span=(1, 2),
   )

   bundle = generate_random_dataset(config, seed=202)
   bundle.write(Path("examples/synthetic/medium"))

You can customise the config ranges (blocks, production rates, blackout probabilities, and role
assignments) or pass a ``systems`` dictionary to incorporate predefined harvest system templates via
``generate_random_dataset(..., systems=default_system_registry())``.

For small teaching examples, the deterministic helper :func:`fhops.scenario.synthetic.generate_basic`
creates a scenario with a single landing and no randomisation; for system-aware datasets use
:func:`fhops.scenario.synthetic.generate_with_systems`.

Where to go next
----------------

* ``tests/test_synthetic_dataset.py`` illustrates the expected invariants and is a good starting
  point if you extend the generator.
* The benchmarking how-to (:doc:`benchmarks`) explains how to wire new scenarios into the benchmarking
  harness once you are happy with the dataset characteristics.
