---
description: |
  [TOPIC] Lessons learned — project organization
  [DETAILS] The project-organization row of the append-only lessons log: hard-coded parameters vs `./config/`, missing `./config/`, missing top-level `Makefile`, the `SHELL := /bin/bash` footgun, the `PATH.yaml` outer-wrapper and missing-`f"..."` crashes, cross-stage `symlink_to`, silent no-op saves for unknown extensions, incomplete `@stx.session` INJECTED declarations, README-per-directory, the `./docs/` tree, `.gitkeep` for expected-empty dirs, cohort two-level layout, generic filenames, pre-created `tests/` mirror, and the single `GITIGNORED/` scratch dir. Split from [`99_lessons-learned.md`](99_lessons-learned.md).
tags: [scitex-scientific-lessons-learned]
---

# Lessons Learned — project organization

The project-organization section of the append-only lessons log
([`99_lessons-learned.md`](99_lessons-learned.md)).

## Project organization

| Mistake | Symptom | Fix → leaf |
|---|---|---|
| Hard-coded parameters scattered across experiment scripts (`N_TRIALS = 100`, `LR = 1e-3`, paths) | Re-running with different values means editing N files; no single source of truth; can't sweep by config | All parameters in `./config/PARAMS.yaml` (or `EXPERIMENT.yaml`); access via `CONFIG.PARAMS.<key>` injected by `@stx.session` ([`./07_config-and-parameters`](02_research-project_07_config-and-parameters.md)) |
| Project missing `./config/` entirely | Parameters live in code; reproducibility depends on git diff of `.py` files | Mandatory `./config/{PATH,PARAMS,EXPERIMENT,COLORS}.yaml` from project init ([`./03_config-and-data`](02_research-project_03_project-structure-config-and-data.md)) |
| Project missing top-level `Makefile` | `README.md` says "run `bash scripts/cohorts/a/dataset/download.sh && python scripts/...`"; new contributors run wrong command in wrong order; pipeline reproduction undocumented | Mandatory thin top-level `Makefile` dispatching to `scripts/makefile/` targets — `make download / extract / inventory / run-pipeline / repro / eval / clean` are the standard target names ([`./04_makefile`](02_research-project_04_project-structure-makefile.md)) |
| `SHELL := /bin/bash` at top of Makefile silently breaks `@stx.session` | Pipeline scripts run fine when invoked directly via `python3 ...`, but fail with `AttributeError: 'DotDict' object has no attribute '<KEY>'` when the same command is invoked via `make`. Burns hours debugging "the same command works in two places". | Drop the `SHELL :=` line entirely; let make use its default `/bin/sh`. Bash-specific features go inside `scripts/makefile/<target>.sh` with their own shebang ([`./04_makefile` § footgun](02_research-project_04_project-structure-makefile.md#footgun-do-not-set-shell--binbash-at-the-top-of-the-makefile)) |
| `config/PATH.yaml` wrapped in outer `PATH:` namespace | Every script accesses `CONFIG.PATH.<KEY>` and crashes — turns out keys are nested at `CONFIG.PATH.PATH.<KEY>` because the YAML's outer `PATH:` got loaded as a key, not stripped | Drop the outer `PATH:` wrapper in `PATH.yaml`. Filename gives the namespace ([`./03_config-and-data` § PATH.yaml](02_research-project_03_project-structure-config-and-data.md#pathyaml--single-source-of-truth-for-paths)) |
| Static paths in `PATH.yaml` written without `f"..."` prefix | `eval(CONFIG.PATH.X)` raises `SyntaxError: invalid syntax` because plain `"./data/foo"` is parsed as a Python expression, not a string literal | Always use `f"..."` form, even for static paths: `KEY: f"./data/foo"`. The `f` prefix makes it a valid Python expression ([`./03_config-and-data` § PATH.yaml](02_research-project_03_project-structure-config-and-data.md#pathyaml--single-source-of-truth-for-paths)) |
| Cross-stage pipelines fail because stage 2 can't find stage 1's `_out/` outputs | Each `stx.io.save(obj, "x.csv")` auto-routes to `<this-script>_out/`; stage 2's `stx.io.load("x.csv")` looks at cwd → `FileNotFoundError` | Use `stx.io.save(obj, "x.csv", symlink_to=eval(CONFIG.PATH.X))` — real bytes stay under the producer's `_out/`, but a symlink at the canonical PATH-yaml location makes them discoverable from anywhere ([`./03_config-and-data` § cross-stage I/O](02_research-project_03_project-structure-config-and-data.md#cross-stage-io-with-symlink_to)) |
| `stx.io.save(text, "x.mmd")` silently does nothing for unrecognised extensions | Log says "saved" but no file lands on disk; only common formats (csv/json/pkl/png/...) are dispatched | For unknown extensions, write directly: `Path(...).write_text(content)` |
| `@stx.session def main(CONFIG=stx.INJECTED, logger=stx.INJECTED):` — only 2 of 5 INJECTED params declared | scitex-linter STX-S006 warns; some agents reject the file | Declare all 5: `CONFIG, COLORS, logger, plt, rngg` (use `stx.session.INJECTED`, not the deprecated `stx.INJECTED`) |
| Sub-directories with no `README.md` | Reader has to guess what `data/cohort_b_bixbench/notebook_convert/` is for; new contributors open arbitrary `.py` files to reverse-engineer; rationale lost on hand-off | **Every directory has a `README.md`** explaining its purpose, what files belong here, what does NOT belong here, and links to sibling/parent READMEs. Even one paragraph beats nothing. Especially required: every cohort dir, every experiment dir, every `dataset/` dir. |
| Project lacks a `./docs/` tree (only top-level README) | Methodology, hypothesis history, dataset provenance, decision logs all crammed into README or scattered; reviewers / co-authors can't find context; can't link to specific topics | Mandatory `./docs/` with topic subdirs (`docs/{benchmark_datasets,methodology,decisions}/`). README links to `docs/`, not the other way around. Pre-create at project init even if empty (use `.gitkeep`). |
| Empty directory needed at runtime (e.g. `outputs/`, `cache/`, `logs/`) but git can't track empty dirs | `git clone` on a fresh machine; pipeline fails because `mkdir -p` was assumed but missing; or developer reorganises and forgets to recreate | **Place `.gitkeep` in every empty dir that the project's runtime expects to exist**. Distinguish from gitignored output dirs (`SDIR_OUT/`) which are NOT tracked. `.gitkeep` is for "this dir is part of the project structure and must exist on clone". |
| Cohort raw data + extractions + experiment outputs all in `data/<cohort>/` flat | No way to tell what's source vs. derived; accidental edits to upstream | Two-level: `data/<cohort>/{capsules,src}/` ([`./08_cohort-datasets`](02_research-project_08_cohort-datasets.md)) |
| Cohort-specific filenames (`download_corebench.sh`) in a flat scripts/ dir | Orchestrator needs lookup table; rename = 3× the work | Cohort goes in the **path**, filename is generic ([`./10_naming-and-numbering`](02_research-project_10_naming-and-numbering.md) rule 3) |
| `tests/` empty until tests written | Structure invisible in git; people forget the mirror | Pre-create with `.gitkeep` ([`./08_cohort-datasets`](02_research-project_08_cohort-datasets.md)) |
| Top-level project dirs created ad-hoc (`GITIGNORED/` tracked accidentally) | Reviewer notes / scratch in git history | Top-level scratch dirs MUST be in `.gitignore`; check `git ls-files` after creation |
| Scattering temporary/scratch files (TODOs, reviewer notes, PDF drafts, draft hypotheses, discussion logs) across the repo | Pollute git history; reviewers see WIP state; merge conflicts on personal notes | **Convention**: a single top-level `GITIGNORED/` dir per project, listed in `.gitignore` from day one. All temporary / per-machine / per-author files go there. Subdirs by type: `GITIGNORED/{TODO.md,REVIEWERS/,DISCUSSIONS/,DRAFTS/}` |
| Experiment outputs written into `data/<cohort>/src/` | Can't tell upstream raw from our derived; loses provenance | Outputs go to `SDIR_OUT/`; `src/` is read-only convention ([`./09_id-readability`](02_research-project_09_id-readability-and-data-immutability.md)) |
