# PR Data Model, Collection Pipeline, and Status Tracking Research

**Date**: 2026-02-26
**Scope**: Understanding current PR data model to add "rejected" and "changes_requested" PR statuses plus incremental status refresh

---

## 1. Current PR Data Model

### Database Table: `pull_request_cache`

The canonical storage model is `PullRequestCache` in `/src/gitflow_analytics/models/database_metrics_models.py`.

**Current columns (at v3.0 schema):**

| Column | Type | Notes |
|--------|------|-------|
| `id` | Integer PK | |
| `repo_path` | String | e.g. `"owner/repo"` |
| `pr_number` | Integer | GitHub PR number |
| `title` | String | |
| `description` | String | PR body |
| `author` | String | GitHub login, lowercased |
| `created_at` | DateTime(tz) | |
| `merged_at` | DateTime(tz) | NULL if not merged |
| `story_points` | Integer | Extracted from title/body |
| `labels` | JSON | List of label strings |
| `commit_hashes` | JSON | List of SHA strings |
| `review_comments_count` | Integer | Inline code comments (v3.0) |
| `pr_comments_count` | Integer | Timeline/issue comments (v3.0) |
| `approvals_count` | Integer | APPROVED reviews count (v3.0) |
| `change_requests_count` | Integer | CHANGES_REQUESTED reviews count (v3.0) |
| `reviewers` | JSON | List of reviewer logins (v3.0) |
| `approved_by` | JSON | List of approving logins (v3.0) |
| `time_to_first_review_hours` | Float | Hours to first review event (v3.0) |
| `revision_count` | Integer | Commit pushes after open (v3.0) |
| `changed_files` | Integer | Files changed count (v3.0) |
| `additions` | Integer | Lines added (v3.0) |
| `deletions` | Integer | Lines removed (v3.0) |
| `cached_at` | DateTime(tz) | For TTL staleness checks |

**CRITICAL GAP: Missing columns for new requirements:**

- `closed_at` — the close/rejection timestamp (exists in `_extract_pr_data` dict output but is NOT persisted to the database table)
- `pr_state` — the lifecycle state string `"merged"` / `"closed"` / `"open"` (exists in dict output, NOT in the table)
- `is_merged` — boolean flag (computed in dict output, NOT in the table)

These fields exist in the in-memory PR data dicts produced by `_extract_pr_data()` and `_get_cached_prs_bulk()` but are never written to or read from `PullRequestCache`.

---

## 2. PR Collection Pipeline

### Entry Point: `gfa fetch`

**File**: `/src/gitflow_analytics/cli_fetch.py`

The `fetch` CLI command (`@click.command(name="fetch")`) does the following:

1. Loads config, initializes `GitAnalysisCache`
2. Calls `GitDataFetcher.fetch_repository_data()` for each repo — this handles **commits and tickets only**, not PRs
3. Initializes `IntegrationOrchestrator` but only uses it to get a `jira_integration` reference for ticket fetching

**PR data is NOT fetched during `gfa fetch`**. PRs are fetched lazily during report generation.

### Actual PR Fetch Path

PR data is fetched during analysis/reporting via:

```
IntegrationOrchestrator.enrich_repository()
  → GitHubIntegration.enrich_repository_with_prs()
      → _get_incremental_fetch_date()     # schema-version-based date gating
      → _get_cached_prs_bulk()            # read existing cache rows
      → _get_pull_requests()              # GitHub REST API call
      → _extract_pr_data()               # parse PyGitHub objects
      → _cache_prs_bulk()                # write new rows to pull_request_cache
      → calculate_pr_metrics()           # aggregate metrics for reporting
```

The call to `enrich_repository_with_prs` happens in:
**File**: `/src/gitflow_analytics/integrations/orchestrator.py` (line ~188)

---

## 3. GitHub API Usage

### Library

Uses **PyGitHub** (REST API wrapper), not GraphQL. Import: `from github import Github`.

### Authentication

Token passed directly: `Github(token)` — expects a GitHub personal access token (classic or fine-grained) in config `github.token`.

### Endpoints Used

| Operation | PyGitHub method | REST equivalent |
|-----------|----------------|-----------------|
| Get repo | `github.get_repo(repo_name)` | `GET /repos/{owner}/{repo}` |
| List PRs | `repo.get_pulls(state="all", sort="updated", direction="desc")` | `GET /repos/{owner}/{repo}/pulls` |
| PR commits | `pr.get_commits()` | `GET /repos/{owner}/{repo}/pulls/{pr}/commits` |
| PR reviews | `pr.get_reviews()` | `GET /repos/{owner}/{repo}/pulls/{pr}/reviews` |
| Issue comments | `pr.get_issue_comments()` | `GET /repos/{owner}/{repo}/issues/{pr}/comments` |

Reviews and issue comments are **only fetched** when `fetch_pr_reviews: true` is set in config (default: `false`).

### Rate Limiting

- Handled by retry loop with exponential backoff in `_get_pull_requests()`
- Config: `max_retries` (default 3), `backoff_factor` (default 2)
- On final failure, PR enrichment is silently skipped (returns `[]`)
- Each review fetch (`fetch_pr_reviews=true`) adds 2 extra API calls per PR

---

## 4. PR State Tracking: Current State vs. What's Needed

### Current State

`_get_pull_requests()` (line ~300, `github_integration.py`) already fetches **both merged and closed-without-merge PRs**:

```python
for pr in repo.get_pulls(state="all", sort="updated", direction="desc"):
    if pr.updated_at < since:
        break
    if pr.merged and pr.merged_at >= since:
        prs.append(pr)   # merged
    elif not pr.merged and pr.state == "closed" and pr.closed_at and pr.closed_at >= since:
        prs.append(pr)   # rejected (closed without merge)
```

`_extract_pr_data()` already computes `pr_state`:

```python
if pr.merged:
    pr_state = "merged"
elif pr.state == "closed":
    pr_state = "closed"   # closed without merge = rejected
else:
    pr_state = pr.state   # "open"
```

And includes `closed_at` and `is_merged` in the returned dict.

**But these fields are NOT written to `PullRequestCache`** and NOT read back from it. The `cache_pr()` and `_get_cached_prs_bulk()` methods reconstruct `pr_state` heuristically from `merged_at IS NOT NULL`.

### The `change_requests_count` Field

`change_requests_count` **is already stored** in `PullRequestCache` (v3.0). It counts the number of `CHANGES_REQUESTED` review events per PR. This is only populated when `fetch_pr_reviews: true`.

The distinction between "changes_requested" as a **review state** (a count field already stored) vs. "changes_requested" as a **PR lifecycle status** (not stored) is important:

- The PR itself can be in one of three lifecycle states: `open`, `closed` (rejected), or `merged`
- A merged PR can still have had `change_requests_count > 0` (reviewer asked for changes, author fixed them, PR was eventually merged)
- A rejected PR can have `change_requests_count = 0` (closed due to other reasons)

---

## 5. Incremental PR Status Update Pattern

### Existing Incremental Mechanism (Schema-Version-Based)

The current incremental system (`_get_incremental_fetch_date()` in `github_integration.py`) uses:
- `SchemaVersionManager` (in `core/schema_version.py`) stored in `schema_versions.db`
- Tracks the `last_processed_date` for the `"github"` component
- On each run, fetches PRs updated since `max(last_processed_date, requested_since)`
- This prevents re-fetching ALL PRs every time the report runs

**Key behavior**: The fetch query uses `sort="updated"`, so ANY PR that was updated (new commits, comments, reviews, status change) after the fetch date will be re-fetched. This means status changes ARE captured on subsequent runs — but only if the PR was updated after the last fetch date.

### Staleness / TTL Mechanism

`_is_pr_stale()` checks `cached_at` against the configured `ttl_hours` (default 168 hours = 1 week). Stale PRs are excluded from cache reads but only re-fetched if the API fetch query returns them (i.e., they were updated recently enough).

### The Staleness Problem for Status Refresh

**Open PRs that get closed or rejected after the initial fetch are NOT automatically refreshed** unless:
1. The PR was updated (committed to, commented on, reviewed) after the fetch date — in which case GitHub includes it in the `sort="updated"` listing
2. The cache entry is stale (TTL expired) — but it still won't be re-fetched unless it's also in the API results

A PR that was opened and closed between two fetch runs will be captured normally. A PR that was cached as `open` and subsequently rejected with no further activity after the last fetch date will remain stale in the cache with incorrect state.

---

## 6. How PRs Connect to Commits

### Link Mechanism: Commit Hashes (via GitHub API)

PRs are linked to commits via `pr.get_commits()` in `_extract_pr_data()`. The commit SHAs are stored in `PullRequestCache.commit_hashes` (JSON list).

The mapping is built at report time in `enrich_repository_with_prs()`:

```python
commit_to_pr = {}
for pr_data in all_pr_data:
    for commit_hash in pr_data.get("commit_hashes", []):
        commit_to_pr[commit_hash] = pr_data
```

Then commits receive `pr_number` and `pr_title` annotations, and story points flow from PR to commit if the commit has none.

**Linkage type: PR commit list via GitHub API** (not merge commit SHA, not branch name pattern matching, not PR number in commit message).

This means:
- A commit that appears in multiple PRs (cherry-picked) will be mapped to the LAST PR in `all_pr_data` that references it (last-write-wins)
- Commits on closed-without-merge PRs are still linked to the PR, but may not appear in the main branch's commit history

---

## 7. Report Integration

### PR Metrics in Narrative Report

The narrative report has a `## Pull Request Analysis` section (in `narrative_analysis.py:_write_pr_analysis()`). It shows:

**Always shown:**
- Total PRs merged
- Average PR size (lines)
- Average files per PR
- Average PR lifetime (created → merged)
- Story point coverage %

**Only shown when `fetch_pr_reviews: true`:**
- Approval rate
- Review coverage %
- Average approvals per PR
- Average change requests per PR
- Time-to-first-review (avg and median)
- Revision count

**NOT shown (missing):**
- Rejected/closed PR count
- Rejection rate (closed without merge / total)
- Breakdown by PR lifecycle state

The report section gate is: `pr_metrics.get("total_prs", 0) > 0` — where `total_prs` currently counts only PRs that matched a commit hash in the commit list (see `enrich_repository_with_prs`). Rejected PRs whose commits are not in the analyzed branch/period will be missed.

### CSV Reports

PR data surfaces in DORA metrics CSVs via `csv_reports_dora.py` (lead time calculations use `merged_at - created_at`). Rejected PRs are excluded from DORA lead time by design since they never shipped.

---

## 8. Specific Gaps and Proposed Additions

### Gap 1: `pr_state`, `closed_at`, `is_merged` Not Persisted

**Problem**: The in-memory dict from `_extract_pr_data()` includes these fields but `PullRequestCache` does not have columns for them. `cache_pr()` and `_get_cached_prs_bulk()` reconstruct state heuristically.

**Fix**: Add three columns to `PullRequestCache` via a v4.0 `ALTER TABLE` migration in `_apply_migrations()`:

```python
("pr_state", "TEXT DEFAULT 'merged'"),   # 'merged', 'closed', 'open'
("closed_at", "DATETIME"),
("is_merged", "INTEGER DEFAULT 1"),       # SQLite boolean
```

Update `cache_pr()` to write these, and `_get_cached_prs_bulk()` / `_pr_to_dict()` to read them.

### Gap 2: Rejection Rate Not Surfaced in Reports

**Problem**: `calculate_pr_metrics()` and `_write_pr_analysis()` do not track or display rejected PRs separately.

**Fix**: In `calculate_pr_metrics()`, partition the input `prs` list by `pr_state` and emit:
- `total_prs_merged`
- `total_prs_rejected`  (closed without merge)
- `rejection_rate` (rejected / (merged + rejected) * 100)

Update `_write_pr_analysis()` to display these in the Overview section.

### Gap 3: Incremental Status Refresh for Open PRs

**Problem**: PRs cached as `open` (or stale rejected) are not refreshed unless they had recent GitHub activity.

**Proposed approach — "stale open PR re-fetch"**:

In `enrich_repository_with_prs()`, after reading the cache, identify PRs where `pr_state == "open"` and `cached_at` is older than a configurable threshold (e.g., 24 hours). Re-fetch those specific PRs by number using `repo.get_pull(pr_number)` (single-PR REST call, no pagination). Update the cache with the refreshed state.

This avoids re-fetching the full PR list and targets only stale open PRs that may have been closed/rejected.

```python
# Pseudocode for incremental status refresh
open_stale_prs = [
    pr for pr in cached_prs_data
    if pr.get("pr_state") == "open"
    and self._is_pr_stale(pr["cached_at"])
]
for pr_data in open_stale_prs:
    try:
        fresh_pr = repo.get_pull(pr_data["number"])
        updated = self._extract_pr_data(fresh_pr)
        self.cache.cache_pr(repo_name, updated)
    except Exception:
        pass   # Non-fatal; stale data is acceptable
```

### Gap 4: `fetch_pr_reviews` Not Wired Through `IntegrationOrchestrator`

The `GitHubConfig.fetch_pr_reviews` config field exists and is loaded by the config loader, but `IntegrationOrchestrator.__init__()` does NOT pass it to `GitHubIntegration()`. The integration always defaults to `fetch_pr_reviews=False`.

**Fix**: In `orchestrator.py` line ~29:
```python
self.integrations["github"] = GitHubIntegration(
    config.github.token,
    cache,
    config.github.max_retries,
    config.github.backoff_factor,
    allowed_ticket_platforms=...,
    fetch_pr_reviews=config.github.fetch_pr_reviews,  # ADD THIS
)
```

---

## 9. Files That Need Modification

| File | Change Required |
|------|----------------|
| `/src/gitflow_analytics/models/database_metrics_models.py` | Add `pr_state`, `closed_at`, `is_merged` columns to `PullRequestCache` |
| `/src/gitflow_analytics/models/database.py` | Add `_migrate_pull_request_cache_v4()` migration method in `_apply_migrations()` |
| `/src/gitflow_analytics/core/cache_commits.py` | Update `cache_pr()` to write new fields; update `_pr_to_dict()` to return them |
| `/src/gitflow_analytics/integrations/github_integration.py` | Update `_get_cached_prs_bulk()` to read `pr_state`/`closed_at`/`is_merged` from DB; add stale open PR refresh logic; update `calculate_pr_metrics()` to include rejection stats |
| `/src/gitflow_analytics/integrations/orchestrator.py` | Pass `fetch_pr_reviews=config.github.fetch_pr_reviews` to `GitHubIntegration.__init__()` |
| `/src/gitflow_analytics/reports/narrative_analysis.py` | Add rejection rate / closed PR count to `_write_pr_analysis()` |
| `/src/gitflow_analytics/reports/data_models.py` | Add `rejected_count` or `rejection_rate` field to `ProjectMetrics` |
| `/src/gitflow_analytics/core/schema_version.py` | Update `CURRENT_SCHEMAS["github"]` version if schema change should force re-fetch |

---

## 10. Gotchas and Rate Limit Considerations

### Rate Limits

- GitHub REST API: 5,000 requests/hour for authenticated users
- `repo.get_pulls(state="all")` paginates at 30 PRs/page by default; large repos with thousands of PRs will consume many requests
- Each PR with `fetch_pr_reviews=True` costs 2 extra calls (`get_reviews()` + `get_issue_comments()`)
- The stale-open-PR refresh adds 1 call per stale open PR per run
- Recommendation: Add a `max_stale_pr_refresh_count` config limit (e.g., 50 per run)

### Pagination Behavior

`repo.get_pulls()` returns a `PaginatedList`. The current loop breaks early when `pr.updated_at < since`, which is safe because `sort="updated", direction="desc"`. PRs are processed newest-to-oldest; the loop exits when all remaining PRs predate the fetch window.

### Rejected PRs and Commit Linkage

Rejected PRs' commits may not appear in the main branch. The current commit enrichment flow (`enrich_repository_with_prs`) matches commits by hash against the analyzed commit list. Rejected-PR commits typically won't match (since they weren't merged), so `enriched_prs` returned from that function will often exclude rejected PRs entirely.

To make rejected PRs visible in reports, they must be tracked separately from the commit-centric enrichment flow — stored as `prs` in the cache and returned as a separate list, not gated on commit hash matching.

### `closed_at` vs `merged_at` in PyGitHub

- `pr.merged_at`: available on the PR object directly, `None` if not merged
- `pr.closed_at`: available as `getattr(pr, "closed_at", None)` — this is the timestamp of closure for BOTH merged and rejected PRs. For merged PRs, `closed_at` and `merged_at` are usually the same moment or very close.
- `pr.state`: returns `"open"` or `"closed"` (regardless of whether it was merged or not)
- `pr.merged`: boolean — the definitive check for whether a closed PR was merged

### SQLite JSON Columns

`reviewers` and `approved_by` are stored as JSON in SQLite (TEXT column type after ALTER TABLE). When reading from cache, always use `.get("field") or []` pattern since NULL is possible for pre-migration rows.

---

## Summary

The codebase has done significant groundwork for PR tracking. The core data pipeline is solid and already fetches both merged and rejected PRs from the GitHub API. The main gaps are:

1. **`pr_state`, `closed_at`, `is_merged` are computed but not persisted** — a straightforward v4.0 column migration fixes this
2. **Rejection rate is not surfaced in reports** — add to `calculate_pr_metrics()` and `_write_pr_analysis()`
3. **`fetch_pr_reviews` config is not wired to the orchestrator** — a one-line fix in `orchestrator.py`
4. **No mechanism to refresh stale open-PR status** — requires targeted single-PR re-fetch logic for cached open PRs
5. **Rejected PRs are excluded from `enriched_prs`** because the commit-hash matching gate filters them out — need a separate tracking path for rejected PRs in the report pipeline
