PR 8c measurements -- the completion instrument against what this run actually produced
Head measured: origin/develop 33a3f6e (PR 8b merged). Date: 2026-10-01.
Every line below was produced by an executed instrument; the raw outputs sit beside this file.

1. Row set (instruments/pr8c_plan_issue_set.py -> pr8c_plan_issue_set.txt)
   Plan section 3, Fixes and Refs columns: 25 issues.
   #258-#272 (15, of which #270 is Refs only and #271 is PR 9), plus the Revision 6
   byproducts #277 #279 #281 #282 #284 #285 #286 #287 #288 #306.
   The archived manifest covers 15 of them; the 10 byproducts are not in it.

2. Proposition 6 as archived reads closedByPullRequestsReferences
   (instruments/pr8c_closure_evidence.py -> pr8c_closure_evidence.txt)
   - closedByPullRequestsReferences is [] for all 25 issues, and the closer of the
     last CLOSED event is null for all of them. Merges go to develop, which is not
     the default branch, so GitHub never auto-closes; every issue was closed by hand.
     As archived, proposition 6 fails for every closed issue: DC7.
   - 23 closed (22 COMPLETED, #286 NOT_PLANNED), #270 and #271 open.
   - For each of the 20 closed issues whose last comment before closedAt cites a PR
     number, every cited number that is a pull request (the others are issues) is
     MERGED into develop, its merge commit is an ancestor of
     origin/develop, and its body cites the issue (digit-bounded #N).
   - #258, #279, #282: the close comment names PR #292 by title, not number; PR #292's
     body says "Fixes #258", "Fixes #279", "Fixes #282".

3. Worktree import resolution (executed by hand, recorded here)
   - lizyml/_version.py is generated by hatch-vcs and ignored (.gitignore:12). A fresh
     worktree lacks it, so `import lizyml` raises ModuleNotFoundError there: every p2,
     p3 and p5 evaluation in a worktree fails as archived. DC7.
   - The venv's editable install adds /home/rem/repos/LizyML to sys.path. With
     cwd=<worktree>, both `python -c` and `python -m pytest` import the worktree's
     lizyml (verified: lizyml.__file__ under the worktree). Running a script FILE puts
     the script's directory first and imports the main checkout instead -- the after
     code -- so the runner must never execute derivations as a file. DC1 if it did.
   - Path(".venv/bin/python").resolve() follows the venv symlink to the base
     interpreter, which has no pytest ("No module named pytest" for all 23 rows, first
     run of pr8c_red_before.py). The runner must use an absolute, unresolved path.

4. Archived manifest rows against develop (scout, executed collection; see table in
   pr8c_acceptance_criteria.md section 2)
   - 4 of the 17 named test files do not exist. Of the population_test selectors in
     files that do exist, 9 select nothing (pytest exit 4): #258 #260 #261 #262 #263
     #266 #268 #269 #272; with the 4 selectors in absent files, every one of the 13 archived rows that carries a
     population_test has a dead selector (corrected in design review round 1, which re-ran them; this
     line first said 7). Declared cardinalities disagree with what shipped (#261 3 not 21, #263 19,
     #267 30 not 378, #272 16 not 4).
   - 9 issues have no node whose parametrisation is the population: #259 #264 #265
     #268 #269 #277 #281 #284 #288 (#270 is a whole-suite property).
   - Several populations are lists declared in the test, not derivable from code:
     #267 #272 #282 #285 #287. #266 is not one of them: its 7 sites come from the
     test's scan of the documents, so a new version declaration in a document changes
     the collected count (round 1 added one in scratch and got 8). This line first
     listed #266 here.
   - #262: 6 of 27 cells are strict xfail (#299); pytest exits 0, so "passes" holds.

5. Proposition 2: each row's shipped tests run in an older tree
   (instruments/pr8c_red_before.py; helpers absent from the older tree are staged:
   tests/_train_spy.py, tests/_ast_scan.py, tests/test_data/_hostile_arrays.py,
   tests/test_config/_knob_registry.py, tests/test_persistence/_load_census.py,
   tests/test_docs/__init__.py)

   a. At 5712f41 (the archived head_before) -> pr8c_red_at_5712f41.txt
      FAILED >= 1, no collection error: 13 rows (#258 #259 #260 #261 #262 #263 #266
        #267 #268 #269 #272 #281 #306; #268 is 1 failed + 80 errors)
      passes: #265 (decision-only)
      collection error only: 9 rows (#264 #270 #277 #279 #282 #284 #285 #287 #288)
      13 + 1 + 9 = 23 rows run (#271 has no test yet, #286 none by design)

   b. At the first parent of the fixing PR's merge commit -> pr8c_red_at_merge_parent.txt
      FAILED >= 1, no collection error: 16 rows (#258 #259 #260 #261 #262 #263 #266
        #267 #268 #269 #272 #279 #281 #282 #285 #306)
      passes before the fix: 3 rows -- #265 (decision-only, expected), #284 (2 passed),
        #287 (8 passed)
      collection error only: 4 rows -- #264, #288 (ImportError: check_duplicate_identities
        from lizyml.core._model_factories, added by #278), #277 (ModuleNotFoundError:
        lizyml.calibration._optimizer, added by #296), #270 (partial, not judged by p2)
      16 + 3 + 4 = 23

   c. Why #284 and #287 pass before their PR: results/pr2c_acceptance_criteria.md --
      "The original #287 symptom is historical and already fixed at this base. This PR
      records and tests the remaining policy decision"; #284 is a structural
      consolidation (one walk), and its cyclic-mapping test "failed on the first
      candidate", i.e. the PR's own intermediate code, not the base. Neither is a
      regression in the proposition-2 sense; both need a non-regression disposition
      with that justification.

7. The archived manifest was never committed
   `.gitignore:166` ignores `*.json`. `git ls-files .../instruments/deferred` lists README.md,
   check_derivations.py, phase3_gap.py and test_phase3_gap.py -- not phase3_manifest.json, which
   exists only on this machine's disk. The deferred README describes it as archived.
   `git add -f` is not a way round it: the repository's pre-commit hook refuses staged ignored
   files ("BLOCKED: gitignored files are staged"). This PR adds `!docs/audits/**/*.json` to
   `.gitignore`, so the shipped manifest and the two row files this PR's measurement reads
   (`results/pr8c_rows_draft.json`, `results/pr8c_rows_parent.json`) are ordinary tracked files;
   `git check-ignore` confirms `foo.json` at the root is still ignored. The shipped-manifest unit
   test is what fails in CI if the manifest is ever not tracked.

6. Proposition 4 with a growing population (#271)
   The archived #271 derivation counts HISTORY.md proposals; the archived value was 92
   at 5712f41 and the deferred README measured 93 with H-0092; the same derivation
   prints 110 at 33a3f6e. Growth is uneven, not one per PR (round 1 measured #278
   94 -> 97, #296 unchanged at 101, #300 101 -> 103). A literal measured at a fixed SHA
   can match there; what fails is a literal written before the population stops
   moving. So the row may carry a derivation instead of a literal, to avoid keeping a
   stale number, not because a literal is unsatisfiable. (This item first said every
   PR adds one and that a literal cannot match.) #271's test does not exist yet (PR 9).

--- Added after design review round 1 ---

8. Supplying only NEW package files to the before tree
   (instruments/pr8c_overlay_probe.py -> pr8c_overlay_probe.txt)
   Copies into the before tree only the lizyml/*.py files the after tree has and the
   before tree lacks (never a modified file), plus the row's tests and helpers.
   - #277 at 5ac725e: 3 new files staged -> 102 failed, 27 passed. A behavioural RED.
   - #264/#288 at ccae32b: 6 new files staged -> still a collection error:
     cannot import name 'check_duplicate_identities' from 'lizyml.core._model_factories'.
     The name was added to a module that already existed, so it cannot be supplied
     without supplying the fix. PR #278's commit list has the fix (b97e30c) before the
     first test commit (e255182), and PR #296's feat (1181e05) precedes its tests
     (33339cb): neither PR has a tests-only commit to run instead.
   - No dynamic module discovery: `pkgutil`, `import_module`, `iter_modules` and
     `entry_points` occur nowhere in lizyml/ at 33a3f6e, ccae32b or 5ac725e, so a new file
     the before code never imports cannot change its behaviour.

9. Closure association and executed outcomes (instruments/pr8c_round1_probe.py ->
   pr8c_round1_probe.txt; fixing PRs in results/pr8c_fixing_prs.json)
   - The last comment at or before closedAt names every fixing PR of all 23 closed rows:
     by digit-bounded #N for 20 rows, by the PR's exact title for #258/#279/#282 (#292).
   - A keyword-anchored body citation (fix/close/resolve/ref + #N) is missing for 5 pairs:
     #262<-#275, #272<-#310, #286<-#290, #287<-#291, #288<-#278. Requiring it would refuse
     correct records (DC7), so p6 requires a plain digit-bounded citation plus the close
     comment naming the PR.
   - Executed outcomes at the after tree (JUnit, per node): every row's tests are all
     passed, except #262 with 28 passed and 9 xfailed (strict, #299). No row has a skipped
     node -- on the input this probe read (results/pr8c_rows_parent.json), which gave #284
     only test_param_domain_cycles.py. The declared #284 row also has
     test_param_domain.py, which has 230 skipped nodes (corrected in item 10).

--- Added after design review round 2 ---

10. Mutation, closure-comment and staging checks for the round 2 redesign
    (instruments/pr8c_round2_probe.py -> pr8c_round2_probe.txt,
     instruments/pr8c_round2_checks.py -> pr8c_round2_checks.txt,
     instruments/pr8c_round2_staging.py -> pr8c_round2_staging.txt)
    - Reintroduction mutations at the after tree (33a3f6e): #264 (lizyml/core/model.py)
      96 failed, 133 passed; #288's round 2 mutation (lizyml/core/_model_factories.py,
      replaced after round 3 -- see item 11) 13 failed, 216 passed. Unmutated, both
      suites are 229 passed.
    - fix_text provenance: `override=params` (#264) and the round 2 #288 fix_text
      `return dict(base)` are `+` lines of #278's merge. The whole #264 `old` line is
      not, because #290 later added `validate_values=True` to it; and `canonical =`
      in the round 2 #288 `old` was not added by #278. So the manifest binds
      provenance to `fix_text`, not to `old`. (The current #288 fix_text,
      `overlay_params(provider, model_params, fixed)`, is a `+` line of #278's merge
      to lizyml/core/model.py; phase3_gap checks it on every run, trial 3.)
    - Closure comments: 22 comment ids pinned (every closed row with a fixing PR; #286
      is not-planned and has none). Each was written no earlier than every fixing
      PR's merge and no later than the close.
    - Staging (AST `new_module_references`, the check phase3_gap ships): for each of
      the 19 regression rows with a fixing PR, no before-tree module imports, by
      resolved module name or by a dotted-name string, any package file only the
      after tree has. 0 references in all 19 rows (13 distinct before commits), so
      staging is never refused at this head. The earlier string-based check in pr8c_round2_probe.py reported 2
      references for 20 rows (`lizyml/config/version.py`); every one was a false
      positive on the word `version`, which is why the shipped check reads the AST.
    - Helper identity: of the 25 staged tests/ helpers and package markers (the
      staging output's "26 helper files" also counts the unstaged
      tests/conftest.py; corrected after round 3, fact 9), every one
      the before tree already has is byte-identical to the after copy, for all 19
      rows (13 distinct before commits); the only differences are helpers absent from
      the before tree (created by staging, removed afterwards). `tests/conftest.py`, which is not
      staged, is identical in all 13 before commits too. So staging changes no existing
      before-tree test file.
    - Item 9's note and §2 measured the #284 outcomes on `test_param_domain_cycles.py`
      only. The declared #284 row also has `test_param_domain.py`. phase3_gap at
      33a3f6e over both files (pr8c_trial2.txt): 5,265 passed and 230 skipped, the 2
      cycles nodes included. The manifest declares that skip (`expected_nonpass`, bound
      to one test function, reason `holds a mapping`, count 230) and the row is
      COMPLETE with no unexplained node.
    - Full trial of phase3_gap at 33a3f6e (pr8c_trial2.txt, 229 s): COMPLETE 20,
      COMPLETE-RED-BY-MUTATION 2 (#264, #288), PARTIAL 1 (#270), NOT-PLANNED 1 (#286),
      INCOMPLETE 1 (#271: its test does not exist yet, PR 9), UNKNOWN 0; exit 1. Every
      before-tree row's p2 has at least one failed node; no row was refused staging.
      The first trial, before the round 2 redesign, is
      pr8c_trial1_before_round2_redesign.txt.
    - The digit-boundary lookbehind in pr8c_closure_evidence.py and
      pr8c_round1_probe.py was aligned with phase3_gap (`(?<![\w/])`, so `abc#12`
      and `repo/#12` do not cite #12). Re-run at 33a3f6e: the closure_evidence output
      is byte-identical, and so is the round 1 closure section.

--- Added after design review round 3 ---

11. #288's evidence, and the round 3 rule changes against the real trees
    (instruments/pr8c_round3_probe.py -> pr8c_round3_probe.txt,
     instruments/pr8c_round2_staging.py re-run -> pr8c_round3_staging.txt,
     pr8c_trial3.txt = pr8c_completion_33a3f6e.txt)
    - #288 reported a coverage gap, not a defect; its fix is test-only (22b11b3 in
      #278: tests/test_core/test_fit_params_override.py and the PR 2 criteria). The
      round 2 manifest mutation (overlay_params merging every seam by spelling) is no
      evidence for #288: the row's tests as they were at 22b11b3^ already fail 9 more
      nodes under it (10 failed against 1 unmutated), so the tests before the fix
      caught it. Seam by seam, at 33a3f6e, newly failed nodes:
        seam mutated (spelling merge)          22b11b3^ tests   shipped tests
        overlay_params, every seam (round 2)         9               13
        tuning result (model.py)                     1                4
        provider fixed (model.py)                    0                1
        trial overlay (_model_tuning.py)             0                3
        provider fixed (_model_tuning.py, tune)      0                0
      The manifest now declares the provider-fixed seam in model.py: the tests before
      the fix miss it, the shipped tests catch it
      (test_the_provider_fixed_seam_carries_one_spelling). The last row is the
      pre-study training-ownership check (`resolved_model` feeds only
      check_training_managed_overrides), not a merge that reaches lgb.train, so it is
      not one of #288's seams; no test catches its spelling merge.
    - Staging references with string literals read over-inclusively (absolute,
      leading-dot relative, and under the file's package and every enclosing
      package): still 0 in all 19 rows (13 distinct before commits)
      (pr8c_round3_staging.txt, whose helper line now reads "25 staged helper
      files + unstaged tests/conftest.py").
    - Full trial after the round 3 fixes (pr8c_trial3.txt, 220 s): the same split as
      trial 2 -- COMPLETE 20, COMPLETE-RED-BY-MUTATION 2, PARTIAL 1, NOT-PLANNED 1,
      INCOMPLETE 1 (#271), UNKNOWN 0, exit 1. With p3 now comparing node ids as a
      multiset, every real row's JUnit report matched its collection exactly. #288
      under its new mutation: 1 failed, 228 passed.
    - Criterion 10 (instruments/pr8c_mutations.py -> pr8c_mutations.txt, 90 s): 9 of
      9 mutations caught. (a)-(h) INCOMPLETE; (f) now by p2 ("no failed node in the
      before tree": the collection error is read as an error case, where before the
      round 3 fix it was UNKNOWN); (i) UNKNOWN (collection fails without
      `_version.py`), exit 1. Every worktree was clean afterwards.

--- Added after the acceptance review (round 4) ---

12. Literal imports combined within a call, against the real trees
    (instruments/pr8c_round2_staging.py -> pr8c_round4_staging.txt)
    - new_module_references now also resolves each string literal in one call's
      arguments against each other one taken as a package, and every relative
      literal against every `lizyml` literal in the file (round 4, finding 1:
      `import_module(".new", "lizyml.other")` and
      `__import__("lizyml.other", fromlist=["new"])` had produced a false RED).
      Still 0 references in all 19 rows (13 distinct before commits); the row
      lines of pr8c_round4_staging.txt equal pr8c_round3_staging.txt's.

--- Added after the acceptance review (round 5) ---

13. The staging guard read by what literals name, not by call shape
    (instruments/pr8c_round2_staging.py -> pr8c_round5_staging.txt)
    - Round 5 refuted the call-shape rule a third time
      (`__import__('other', globals(), None, ['new'], 1)`). The guard now reads
      import statements, a package `__init__.py`'s `__all__`, and, in a file that
      names the import machinery, every dotted-name or `.py` path literal's
      components and the import statements of any literal that parses as code.
    - A first version took every word of every literal in a machinery file, and
      took `__all__` in any module as machinery. At 33a3f6e it refused 16 of the
      19 rows: `lizyml/_version.py` (a module `__all__` listing 'version') and
      error-message prose containing the word 'version' (new file
      lizyml/config/version.py). Restricted to name-shaped literals and to a
      package's own `__all__`, it finds 0 references in all 19 rows (13 distinct
      before commits), pr8c_round5_staging.txt.
    - Before round 6, two IN-kind gaps of that guard were closed: a literal that
      parses as code is now read recursively for its own literals
      (`eval("__import__('lizyml.other.new')")`), and bytes literals count
      (`exec(b"import lizyml.other.new")`). Re-run (pr8c_round5b_staging.txt):
      still 0 references in all 19 rows. The files that name the import
      machinery are the same two in every before tree, lizyml/core/model.py and
      lizyml/estimators/lgbm/provider.py, both through `importlib.metadata`; none
      of their name-shaped literals is the last component of a staged module, so
      the margin is that set. A later file that names the machinery and holds
      such a literal would be refused: p2 INCOMPLETE, visible in the trial, never
      a false RED.
    - Full trial with the round 5 guard (pr8c_trial4.txt, 222 s; also saved as
      pr8c_completion_33a3f6e.txt): identical to trial 3 apart from the run-exclusive wrapper lines.

--- Added after the acceptance review (round 6) and the maintainer's option C ---

14. p2 without staging any package file (option C)
    (instruments/pr8c_round6_probe.py -> pr8c_round6_probe.txt; pr8c_trial5.txt =
     pr8c_completion_33a3f6e.txt; pr8c_mutations.txt re-run)
    - Round 6 found two more forms the staging guard missed (`__all__[:] = [...]` in
      a package, ten nested `exec` literals past the recursion limit, where the guard
      stopped looking silently: DC1). The declared stop rule fired; the maintainer
      had an external analysis written (pr8c_options_analysis_after_round6.md) and
      chose option C: stage only the row's tests and tests/ helpers, never a
      `lizyml/` file, and delete the guard.
    - With nothing from `lizyml/` staged, the 16 rows that run p2 in a before tree
      keep a failed node and collect cleanly; their outcome counts are the same as
      with staging. #277 cannot collect in 5ac725e: 2 collection errors, both from
      lizyml/calibration/_optimizer.py, which #296 added
      (test_calibration_param_contract.py imports it; test_calibration_params_codegen.py
      imports that test module). #264 and #288, which already used `red_mutation`,
      cannot collect there either.
    - #277 moved to `red_mutation`: `prepare_calibration_params` returning {} for
      platt and beta instead of the normalised params (#296 added the returned line).
      At 33a3f6e: unmutated 129 passed; mutated 6 failed, 123 passed, all six
      params-reach tests (test_calibration_params_reach.py, 4;
      test_calibration_params_codegen.py, 2).
    - Full trial (pr8c_trial5.txt, 215 s): COMPLETE 19, COMPLETE-RED-BY-MUTATION 3
      (#264, #277, #288), PARTIAL 1, NOT-PLANNED 1, INCOMPLETE 1 (#271), UNKNOWN 0,
      exit 1. Against trial 4, only #277's lines and the summary changed.
    - Criterion 10 re-run (pr8c_mutations.txt): 9 of 9 mutations caught.

--- Added after the acceptance review (round 7) ---

15. Test-path grammar and the pinned comment read by id
    (pr8c_trial6.txt = pr8c_completion_33a3f6e.txt; pr8c_mutations.txt re-run)
    - A row listing `lizyml/new.py` among its tests passed validation and was staged,
      bypassing option C (round 7, finding 1). Every `tests` entry must now match
      `tests/(dir/)*test_*.py`; validate_row and files_to_stage both refuse anything
      else. All 25 shipped rows match.
    - p6 read the pinned comment from `comments(first:100)`, so a comment past the
      hundredth would read as absent (round 7, finding 2; no shipped issue has that
      many). It is now fetched by id (`gh api repos/nbx-liz/LizyML/issues/comments/<id>`)
      and its issue_url must name the row's issue.
    - Full trial (pr8c_trial6.txt): identical to trial 5 apart from the run-exclusive wrapper lines;
      all 22 pinned comments are found by id. Criterion 10: 9 of 9 caught; (h) now
      reads "pinned comment 5924845155 is not on #258" from the comment's own issue.

--- Added after the acceptance review (round 8, the last of the budget) ---

16. Staging confined to real files under tests/
    (pr8c_trial7.txt = pr8c_completion_33a3f6e.txt; pr8c_mutations.txt re-run)
    - Round 8 staged an ordinary after-tree helper over a before-tree
      `tests/_alias.py -> ../lizyml/a.py`, overwriting package code and
      manufacturing RED. `staged` now checks every path in both trees before
      copying anything: under tests/, no symlink on the file or any parent, and
      resolving inside tests/; otherwise ManifestError and nothing is copied.
    - No symlink exists in the repository at HEAD, at 33a3f6e, or in the before
      commits 5fb8a80, 1d7c4e2, ccae32b, fc9d820 (`git ls-tree -r`, mode 120000).
    - Round 8 also corrected two statements: the review prompt said the instrument
      has 145 unit tests where it had 125 (145 was all of tests/test_docs; now 128
      and 148), and the test module claimed real-pytest runs for round 7's checks.
    - Full trial (pr8c_trial7.txt): identical to trial 6 apart from the run-exclusive
      wrapper lines. Criterion 10: 9 of 9 caught.
