# Pipeline Data Points Audit
Date: 2026-02-27
Config: ~/Duetto/repos/gitflow-all-engineers.yaml

## Config Requirements Checklist

| Requirement | Status | Location in Config |
|-------------|--------|--------------------|
| `github.organization` set | YES | `organization: "duettoresearch"` (line 11) |
| `github.fetch_pr_reviews: true` | YES | `fetch_pr_reviews: true` (line 12) |
| `github_repo` on all repo entries | YES | All 146 repos have `github_repo:` |
| `analysis.llm_classification.enabled: true` | YES | `enabled: true` (line 982) |
| `analysis.llm_classification.provider: bedrock` | YES | `provider: bedrock` (line 983) |
| `analysis.identity.strip_suffixes` | YES | `strip_suffixes: ["-duetto"]` (line 998) |
| `developer_aliases` for canonical names | YES | 90+ entries (lines 605-974) |

Note: `github_repo: "admin-panel"` without a slash is auto-expanded to
`"duettoresearch/admin-panel"` in `config/repository.py:122` when `organization` is set.

---

## Data Point Verification

### 1. Commit data — via `gfa collect`
**YES — works correctly**

Code path:
- `pipeline_collect.py:run_collect()` → `GitDataFetcher.fetch_repository_data()`
- Writes `CachedCommit` rows to `gitflow_cache.db`
- Stats: commits_count, tickets_count tracked

### 2. PR data (reviews, merge status, rejection metrics) — via `gfa collect`
**YES — works correctly, with caveat**

Code path:
- `pipeline_collect.py:158-190` — after commit fetch, checks `repo_config.github_repo`
- Calls `orchestrator.enrich_repository_data(repo_config, commits, start_date)`
- `IntegrationOrchestrator` → `GitHubIntegration.enrich_repository_with_prs()`
- `fetch_pr_reviews=True` (passed from config via `IntegrationOrchestrator.__init__:35`)
- Fetches review counts, approvals, change_requests, time_to_first_review
- Writes `PullRequestCache` rows with: `pr_state`, `is_merged`, `closed_at`,
  `approvals_count`, `change_requests_count`, `reviewers`, `revision_count`
- Stale open PR refresh runs automatically (cap: 50 per run)

Caveat: PRs for already-cached repos are re-fetched via `cached_with_github` block
(lines 222-262) so incremental runs still update PR state.

### 3. LLM classification (change_type via Bedrock) — via `gfa classify`
**YES — works correctly**

Code path:
- `pipeline_classify.py:run_classify()` → `BatchCommitClassifier.classify_date_range()`
- Reads `DailyCommitBatch` rows (status=pending/failed)
- Calls `LLMCommitClassifier` with Bedrock provider
- Writes `change_type`, `change_type_confidence`, `complexity` to `qualitative_commits` table

### 4. Complexity rating (1-5) — via `gfa classify`
**YES — stored in DB, BUT NOT SURFACED IN PIPELINE REPORTS (GAP)**

Code path:
- Bedrock prompt requests: `"category, confidence (0.0-1.0), complexity (1-5), reasoning"`
- `bedrock_client.py:513-528` — parses and clamps complexity to 1-5
- `batch_classifier_impl.py:551` — `qualitative.complexity = classification_result.get("complexity")`
- `QualitativeCommitData.complexity` column in `qualitative_commits` table

**GAP**: `pipeline_report.py` queries only `CachedCommit` rows and calls
`cache._commit_to_dict()` which does NOT join `qualitative_commits`. The
`change_type` and `complexity` fields are written by `gfa classify` but NOT
read back by `gfa report`. They are invisible to all report generators
called from `pipeline_report.py`.

The only report that would expose complexity from LLM classification is
`ClassificationCsvMixin.generate_detailed_csv_report()` (in
`reports/classification_csv.py:59,88`) — but this report is NOT generated
by `pipeline_report.py`. It is only generated by `gfa analyze --classification-report`
or the old qualitative pipeline.

### 5. Canonical developer names — resolved at report time
**YES — works correctly**

Code path:
- `pipeline_report.py:107-113` — `DeveloperIdentityResolver(identity_db_path, similarity_threshold, manual_mappings)`
- `cfg.analysis.manual_identity_mappings` — loaded from `developer_aliases` section
  (auto-converted by `config/loader.py:LoaderSectionsMixin`)
- `identity_resolver.update_commit_stats(all_commits)` — processes all commits
- `get_developer_stats()` returns canonical names

### 6. GitHub noreply email auto-detection — at report time
**YES — works correctly**

Code path:
- `DeveloperIdentityResolver._resolve_identity()` line 366-403
- Pattern: `{numeric_id}+{username}@users.noreply.github.com`
- Extracts username, matches against existing identities, registers under same canonical ID
- Works automatically during `identity_resolver.update_commit_stats(all_commits)`

Note: `strip_suffixes: ["-duetto"]` from config is passed to `LLMIdentityAnalyzer`
(used by `gfa identity analyze`), NOT to `DeveloperIdentityResolver`.
`DeveloperIdentityResolver.__init__` does not accept `strip_suffixes`.
The suffix stripping only applies when running `gfa identity analyze`,
not during `gfa report`. This is a separate, minor gap for the `-duetto`
username suffix pattern.

### 7. DORA metrics (lead time, MTTR, deployment frequency, change failure rate) — in `gfa report`
**YES — computed and written to CSV + used in narrative**

Code path:
- `pipeline_report.py:354-360` — `DORAMetricsCalculator().calculate_dora_metrics(all_commits, all_prs, start_date, end_date)`
- Returns: `deployment_frequency`, `lead_time_hours`, `change_failure_rate`, `mttr_hours`
- Also: `pipeline_report.py:343-351` — `generate_weekly_dora_report()` writes `weekly_dora_metrics_YYYYMMDD.csv`
- All four DORA metrics are computed from commit messages (deployment/failure patterns) + PR merge times

Note: DORA calculations use heuristic keyword matching:
- Deployments: "deploy", "release", "ship", "live", "production", "prod"
- Failures: "revert", "rollback", "hotfix", "emergency", "incident", "outage"

### 8. Complexity in CSV output — in classification CSV report
**PARTIAL — in classification_csv.py but NOT generated by pipeline_report.py**

Code path where it IS included:
- `reports/classification_csv.py:59,88` — `generate_detailed_csv_report()` has `"complexity"` column
- `commit.get("complexity", "")` — blank when None (rule-based), integer when LLM classified

Code path where it is NOT included:
- `pipeline_report.py` does not call any `ClassificationCsvMixin` method
- `generate_weekly_categorization_report()` (called by pipeline) uses rule-based
  `ticket_extractor.categorize_commit()` — no LLM complexity

**GAP**: Complexity is never written to any CSV in the `gfa collect → classify → report` pipeline.

### 9. PR rejection metrics in narrative report
**YES — works correctly**

Code path:
- `pipeline_report.py:363-395` — builds `all_prs` from cache, calls `narrative_gen.generate_narrative_report(..., all_prs, ...)`
- `reports/narrative_team.py:104-188` — computes rejection_rate from `pr_state == "closed" and not is_merged`
- Shows: rejection rate %, avg change_requests_per_pr, avg revisions on rejected PRs, assessment text
- Requires `fetch_pr_reviews: true` to get `change_requests_count` and `revision_count` fields

### 10. Incremental PR refresh (stale open PRs re-checked) — in collect/github_integration
**YES — works automatically**

Code path:
- `github_integration.py:162-175` — `_refresh_stale_open_prs()` called every `gfa collect` run
- Selects cached PRs with `pr_state == "open"` that were not already fetched in current run
- Re-fetches up to 50 PRs per run (configurable via `stale_open_pr_refresh_cap`)
- Updates state to merged/closed, writes back to cache

---

## Summary Table

| Data Point | Collected | Classified | In Report | Gap? |
|------------|-----------|------------|-----------|------|
| Commit data | YES | - | YES | None |
| PR data + reviews | YES | - | YES | None |
| LLM change_type | YES (stored) | YES (written to qualitative_commits) | NO | **CRITICAL GAP** |
| Complexity (1-5) | YES (stored) | YES (written to qualitative_commits) | NO | **CRITICAL GAP** |
| Canonical dev names | YES | - | YES | None |
| GitHub noreply email | YES | - | YES | None |
| strip_suffixes (-duetto) | - | - | Partial | Minor: only via `gfa identity analyze` |
| DORA metrics | YES | - | YES | None |
| Complexity in CSV | - | YES (qualitative_commits) | NO | Same as item 4 |
| PR rejection metrics | YES | - | YES | None |
| Incremental PR refresh | YES | - | YES | None |

---

## Critical Gap: change_type and complexity not surfaced in pipeline reports

**Root cause**: `pipeline_report.py` builds commit dicts exclusively from `CachedCommit`
via `cache._commit_to_dict()`. This method returns 12 fields (hash, author_name,
author_email, message, timestamp, branch, is_merge, files_changed, insertions,
deletions, complexity_delta, story_points) but does NOT join `qualitative_commits`.

`BatchCommitClassifier` writes `change_type`, `change_type_confidence`, and
`complexity` to `qualitative_commits` (keyed by `commit_id = CachedCommit.id`),
but there is no code path in `pipeline_report.py` or `cache._commit_to_dict()`
that reads these back.

**Fix needed**: In `pipeline_report.py`, after loading `cached_commits_rows`, do
a bulk query against `qualitative_commits` joined on `commit_id`, then merge
`change_type` and `complexity` into each commit dict before passing to report generators.

**What works only in `gfa analyze` (not 3-stage pipeline)**:
- Qualitative analysis output (`qualitative_result.results`) is produced by the
  old `QualitativeProcessor` (NLP-based, reads from `qualitative.db`, not `gitflow_cache.db`)
- This is a completely separate pathway from `BatchCommitClassifier` (Bedrock)
- `gfa analyze` passes `qualitative_result` directly to `generate_all_reports()`
  as `qualitative_data` parameter, enabling the JSON exporter and enhanced analysis
- The 3-stage pipeline does NOT do this — `pipeline_report.py` passes `qualitative_data=None`
  to `NarrativeReportGenerator` and `ComprehensiveJSONExporter`

---

## Key Questions Answered

1. **Does `gfa collect` automatically fetch PRs for repos with `github_repo` set?**
   YES. Lines 158-190 in `pipeline_collect.py`. For already-cached repos, lines 222-262
   also re-run PR enrichment every time.

2. **Does `gfa classify` automatically use Bedrock and include complexity?**
   YES. `BatchCommitClassifier` reads `provider: bedrock` from config, sends batches
   to Bedrock Claude Haiku, and stores `change_type + complexity` in `qualitative_commits`.
   The skipped/failed batch check (status=pending/failed) means re-runs are safe.

3. **Does `gfa report` load cached PRs and compute DORA metrics?**
   YES. Lines 166-174 load PRs via `get_cached_prs_for_report()`. Lines 354-360
   compute DORA. Lines 343-351 write `weekly_dora_metrics_*.csv`.

4. **Are there any manual steps still required?**
   The "batch status reset" workaround (setting failed batches back to pending)
   is no longer needed since `run_classify()` queries for status IN ("pending", "failed")
   — failed batches are retried automatically. However: if batches are stuck in
   "processing" state (e.g. from a crashed run), they would need manual reset.

5. **Is there anything that only works with `gfa analyze` but NOT the 3-stage pipeline?**
   YES — specifically:
   - `change_type` and `complexity` from Bedrock classification are written to the DB
     but NOT surfaced in any report generated by `gfa report`
   - The `qualitative_data` parameter in narrative/JSON reports is always `None`
     in the 3-stage pipeline
   - The old NLP-based qualitative analysis (`qualitative.db`) is only run in `gfa analyze`
