# Output Artifacts and Media – Conventions

This note records the current conventions for organizing and regenerating
output artifacts (eval JSONs, plots, and GIFs) used in docs and the README.
It is intentionally narrow and focused on the Qwen synthetic login benchmark
for now, but the patterns are intended to generalize.

## 1. High-level layout

We distinguish between:

- **Ephemeral outputs** – useful for debugging, not committed:
  - `logs/` – JSONL per-step eval logs.
  - `synthetic_*` – raw synthetic PNGs and session directories.
  - Most one-off eval JSONs at the repo root.
- **Canonical artifacts** – curated assets that we may commit and
  reference from docs/README:
  - `experiments/qwen_login/` – synthetic login benchmark assets:
    - `2b_dev/` – Qwen3-VL-2B hardened dev loop.
    - `8b_hero/` – Qwen3-VL-8B hardened benchmark.

Recommended structure under `experiments/qwen_login/`:

```text
experiments/qwen_login/
  2b_dev/
    eval/
      eval_qwen3_2b_base_login_hardened.json
      eval_qwen3_2b_ft_login_hardened.json
    plots/
      qwen3_2b_base_vs_ft_hardened.png
    media/
      qwen3_2b_login_demo.gif
  8b_hero/
    eval/
      eval_qwen3_8b_base_login_hardened.json
      eval_qwen3_8b_ft_login_hardened.json
    plots/
      qwen3_8b_base_vs_ft_hardened.png
    media/
      qwen3_8b_login_demo.gif  # optional
```

The intention is that:

- Ephemeral directories (`logs/`, `synthetic_*`, `checkpoints/`) remain
  gitignored or uncommitted by default.
- Only the curated eval JSONs, plots, and GIFs in `experiments/qwen_login/**`
  are referenced from `README.md` and `docs/`.

## 2. Plots

Plots for base vs fine-tuned Qwen3-VL models are generated by
`openadapt_ml/evals/plot_eval_metrics.py` and currently live under `plots/`.
For the Qwen login benchmark we recommend copying/moving the canonical
plots into `experiments/qwen_login/.../plots/` as shown above.

The plotting script now adds a small suptitle:

- "Qwen-VL base vs fine-tuned (FT = LoRA fine-tuned model)"

so plots are self-explanatory when embedded in docs.

## 3. GIFs – generic make_gif.py

For UI demos, we generate GIFs from per-step PNGs produced by the
synthetic generators. Rather than a login-specific tool, we plan to
implement a **generic** script:

- Location (planned): `openadapt_ml/scripts/make_gif.py`.
- Responsibilities:
  - Take a directory of PNG frames and a filename pattern.
  - Optionally overlay **deterministic captions** per frame based on
    episode metadata and actions.
  - Write an animated GIF to a specified output path.

Expected CLI shape (conceptual):

```bash
uv run python -m openadapt_ml.scripts.make_gif \
  --frames-dir synthetic_demo/session_0000 \
  --pattern "session_0000_login_step_*.png" \
  --output experiments/qwen_login/2b_dev/media/qwen3_2b_login_demo.gif \
  --duration-ms 1000
```

Captioning strategy is **deterministic**, derived from the synthetic
implementation, not from an external API:

- Use `Episode.goal` (e.g. "Log in with username 'user0' and password 'pass0123'").
- Use `Step.action` (`type`, `x`, `y`, `text`) and step index.
- For login, this yields captions like:
  - "Step 0: Initial login screen (WAIT)"
  - "Step 1: CLICK username field"
  - "Step 2: TYPE username 'user0'"
  - "Step 3: CLICK password field"
  - "Step 4: TYPE password (masked on screen)"
  - "Step 5: CLICK Login button"
  - "Step 6: DONE (logged in)".

This approach generalizes to other synthetic scenarios (settings panel,
browser UI, etc.) as long as generators expose enough semantic
information (e.g. which bbox corresponds to which control).

## 4. README and docs usage

The README section `4.3 Qwen3-VL synthetic login benchmark (typical usage)`
links to plots and (optionally) GIFs via relative paths under
`experiments/qwen_login/**`.

`docs/state_and_next_steps_qwen_login.md` §7 "Hardened benchmark results"
contains tables with the canonical 2B and 8B metrics and references the
corresponding plots. When GIFs are available, they can be embedded in the
same way:

```markdown
![Qwen3-VL-2B synthetic login demo](experiments/qwen_login/2b_dev/media/qwen3_2b_login_demo.gif)
```

The goal is that a small, well-defined set of artifacts under
`experiments/qwen_login/**` is sufficient to visually convey the
Qwen-based synthetic login benchmark, while the rest of the generated
files remain ephemeral.
