Thesis Evaluation Workflow (Chapter 2)
======================================

This guide translates Rosalia Jaffray’s MASc proposal (Chapter 2: “Does FHOPS work, and does it close
the gaps we identified in Chapter 1?”) into a repeatable workflow. Use it when preparing the thesis
case-study experiments: assemble operational datasets, run FHOPS solvers, evaluate KPIs, and document
trade-offs for Chapter 2 narratives.

Context & Goals
---------------

- Chapter 1 (literature review) catalogues BC operational-planning gaps (data accessibility, solver
  transparency, mobilisation awareness, shift-level sequencing).
- Chapter 2 must show that FHOPS can **replicate and improve** existing planning efforts via
  reproducible case studies on small-scale operations.
- Deliverables for each case:

  1. Validated FHOPS scenario (data contract-compliant, shift-aware, mobilisation-enabled).
  2. Baseline solver runs (MIP + heuristics) with traceable KPIs.
  3. Trade-off discussion (production vs mobilisation vs utilisation).
  4. Documentation package (commands, telemetry, plots) that can be cited in the thesis.

Pipeline Overview
-----------------

1. **Curate the Case Dataset**

   - Start from ``examples/med42`` or ``examples/large84`` as a template.
   - Replace ``data/*.csv`` with the case-study inventory (blocks, machines, landings, calendars,
     production rates, optional ``road_construction``).
   - If the case uses known harvest systems, add a ``harvest_system_id`` column per block using IDs from
     :doc:`../reference/harvest_systems`.
   - Record provenance notes (tenure, timeline, data sources) in ``README.md`` for the case folder.

2. **Validate the Scenario**

   .. code-block:: bash

      fhops validate case_study/scenario.yaml

   - Fix reported errors (missing references, shift IDs, schema version) before running solvers.
   - Use ``docs/howto/data_contract.rst`` if new columns/optional extras are required.

   .. seealso::

      :func:`fhops.cli.main.validate` – command reference for the scenario validator (includes all CLI options and schema checks).

3. **Run Baseline Solvers**

   - **Operational MILP (reference solution)**:

     .. code-block:: bash

        fhops solve-mip-operational case_study/scenario.yaml \
          --out case_study/out/mip_solution.csv \
          --time-limit 1800

     Add ``--solver gurobi`` when a Gurobi licence is available. (``fhops solve-mip`` is a
     deprecated alias that runs the same model since FHOPS 1.0.1.)

   - **Simulated Annealing (fast heuristic)**:

     .. code-block:: bash

        fhops solve-heur case_study/scenario.yaml \
          --out case_study/out/sa_solution.csv \
          --iters 12000 --seed 42 \
          --operator-preset explore \
          --telemetry-log case_study/out/sa_telemetry.jsonl \
          --show-operator-stats

   - Optional: add ``fhops solve-ils`` and ``fhops solve-tabu`` for comparative analysis.

   - Document runtime, objective values, and solver settings (Chapter 2 must highlight reproducibility).

   .. seealso::

      :func:`fhops.cli.main.solve_mip_operational_cmd`, :func:`fhops.cli.main.solve_heur_cmd`, :func:`fhops.cli.main.solve_ils_cmd`, and :func:`fhops.cli.main.solve_tabu_cmd` – each CLI entrypoint documents the complete option set, telemetry hooks, and solver-specific notes.

4. **Evaluate KPIs & Mobilisation Spend**

   .. code-block:: bash

      fhops eval-playback case_study/scenario.yaml \
        --assignments case_study/out/mip_solution.csv \
        --shift-out case_study/out/mip_shift.csv \
        --day-out case_study/out/mip_day.csv \
        --summary-md case_study/out/mip_summary.md

   - Collect:

     * ``total_production`` and ``completed_blocks``.
     * ``mobilisation_cost`` and ``mobilisation_cost_by_machine`` (Chapter 1 gap: no cost audit).
     * ``utilisation_ratio`` (shift/day), ``makespan``.
     * ``sequencing_violation_*`` counts (showing constraints hold).

   - Repeat for heuristic schedules. Compare KPI deltas in a table (include convergence rationale).

   .. seealso::

      :func:`fhops.cli.main.evaluate` for KPI-only summaries and :func:`fhops.cli.main.eval_playback` for shift/day playback exports (deterministic or stochastic).

5. **Benchmark Trade-offs**

   - Run the benchmark harness to quantify solver differences and generate plots:

     .. code-block:: bash

        fhops bench suite --scenario case_study/scenario.yaml \
          --include-ils --include-tabu --out-dir case_study/bench \
          --time-limit 900 --sa-iters 12000 --tabu-iters 8000 --ils-iters 400
        python scripts/render_benchmark_plots.py case_study/bench/summary.csv \
          --out-dir case_study/bench/plots

   - Use the summary CSV/JSON to extract:

     * Objective gap vs best heuristic (evidence of improved solution quality).
     * Runtime ratios (feasibility for small-scale operators).
     * Operator telemetry (link back to mobilization-aware operators when discussing Chapter 1 needs).

   .. seealso::

      :func:`fhops.cli.benchmarks.bench_suite` – benchmark CLI helper that powers ``fhops bench suite``.

6. **Synthesize Chapter 2 Materials**

   - Insert KPI tables, mobilisation spend charts, and sequencing status into the chapter draft.
   - Reference appendix artefacts: telemetry logs, shift/day CSVs, benchmark plots.
   - Describe how FHOPS addresses Chapter 1 gaps (e.g., distance-aware mobilisation, shift calendars,
     open-source reproducibility).

Worked Example (med42 Template)
-------------------------------

1. Copy ``examples/med42`` to ``case_study/``; replace ``data/`` with the case inventory.
2. Run validation + solvers as above. Capture commands and seeds for thesis appendices.
3. Highlight insights:

   - ``kpi_mobilisation_cost`` decreased by X % when switching from default to mobilisation-focused
     operator preset.
   - Sequencing violations remained zero, confirming registry accuracy for the target system.
   - Runtime remained under N minutes on lab hardware (relevant for small operations).

4. Discuss trade-offs (production vs mobilisation vs utilisation) with references to Chapter 1 gaps.

Tips & References
-----------------

- Maintain a ``case_study/log.md`` file capturing every command, seed, and data edit (supports Chapter 2 audit trail).
- Include small screenshots or plots (generated from ``case_study/bench/plots``) to visualise KPI movement.
- Cite the proposal folders under ``tmp/jaffray-rosalia-masc-*`` when referencing scope and motivation.
- Keep FHOPS docs updated (see ``notes/sphinx-documentation.md``) whenever new thesis-driven workflows appear.
