# Research: Classify Pipeline — Repo+Week Targeted Re-Classification

**Date:** 2026-02-26
**Scope:** `gfa classify` end-to-end flow, storage model, skip logic, and where to add
`--repo` / `--week` filtering plus a force-reclassify capability scoped to a specific
repo and week.

---

## 1. End-to-End Flow of `gfa classify`

```
gfa classify -c config.yaml [--weeks N] [--reclassify]
        |
        v
cli_pipeline_commands.py :: classify_command()
        |
        | calls
        v
pipeline.py  re-exports ->  pipeline_classify.py :: run_classify(cfg, weeks, reclassify)
        |
        | 1. Computes date range:
        |     current_week_start = get_week_start(now)
        |     last_complete_week_start = current_week_start - 1 week
        |     start_date = last_complete_week_start - (weeks-1) weeks
        |     end_date   = get_week_end(last_complete_week_start + 6 days)
        |
        | 2. Checks cache for commits and daily batches in that range
        |
        | 3. Builds llm_config dict from cfg.analysis.llm_classification.*
        |
        | 4. Instantiates BatchCommitClassifier(cache_dir, llm_config, batch_size=50)
        |
        | 5. Derives project_keys from cfg.repositories
        |       = [repo.project_key or repo.name for repo in cfg.repositories]
        |
        | 6. Calls batch_classifier.classify_date_range(
        |         start_date, end_date, project_keys, force_reclassify=reclassify)
        v
classification/batch_classifier.py :: classify_date_range()
        |
        | _get_batches_to_process()
        |   -> queries DailyCommitBatch WHERE date IN range AND project_key IN keys
        |      AND (if not force) classification_status IN ('pending', 'failed')
        |
        | _group_batches_by_repository()   -> dict[(project_key, repo_path)] -> [batches]
        | _group_batches_by_week()         -> dict[week_monday] -> [daily batches]
        |
        | for each repo -> for each week -> _classify_weekly_batches(week_batches)
        |                                         |
        |                                         v (batch_classifier_impl.py mixin)
        |       Re-fetches live ORM objects by ID (detached-session bug fix)
        |       Marks batches "processing"
        |       _get_commits_for_batch()  -> queries CachedCommit WHERE repo_path + day
        |       _get_ticket_context_for_commits()
        |       for each API batch (50 commits):
        |           _classify_commit_batch_with_llm()
        |               -> LLMCommitClassifier.classify_commits_batch()
        |               -> fallback_classify_commit() if LLM fails/low-confidence
        |       _store_commit_classification()
        |           -> upserts QualitativeCommitData (commit_id PK)
        |       Marks daily batches "completed", sets classified_at
        |       session.commit()
        |
        v
_store_daily_metrics()  (placeholder, not yet aggregating)
```

---

## 2. Storage and Cache Structure

### Primary Tables Involved

| Table | Purpose | Key columns |
|---|---|---|
| `cached_commits` | Raw commits from Stage 1 collect | `id`, `repo_path`, `commit_hash`, `timestamp`, `author_*`, `message`, `insertions`, `deletions` |
| `daily_commit_batches` | One row per (date, project_key, repo_path) | `id`, `date`, `project_key`, `repo_path`, `commit_count`, `classification_status` (`pending`/`processing`/`completed`/`failed`), `classified_at` |
| `qualitative_commits` | Classification results | `commit_id` (FK PK to `cached_commits.id`), `change_type`, `change_type_confidence`, `confidence_score`, `processing_method`, `analyzed_at` |
| `weekly_fetch_status` | Tracks which repo+week combos were collected | `repository_path`, `week_start`, `week_end` |

### How Commits Are Looked Up During Classification

`_get_commits_for_batch(session, batch)` filters `CachedCommit` by:
- `CachedCommit.repo_path == batch.repo_path`
- `CachedCommit.timestamp` within `[start_of_day, end_of_day)` UTC

### How Classifications Are Persisted

`_store_commit_classification()` does an **upsert** on `QualitativeCommitData`:
- Looks up `CachedCommit` by `commit_hash` (note: no `repo_path` filter — potential hash
  collision edge case across repos).
- If row exists in `qualitative_commits` for that `commit_id` → **UPDATE** in place.
- If not → **INSERT** new row.

The upsert makes re-classification for existing rows naturally idempotent.

---

## 3. How Repos and Weeks Are Identified

### Repo Identification

- In `cfg.repositories` (list of `RepositoryConfig`):
  - `name` — human name, e.g. `"my-service"`
  - `path` — filesystem path to local clone
  - `project_key` — explicit override; auto-derived as `name.upper().replace("-","_")` if omitted
- The classify pipeline uses `project_key` as the primary filter identifier.
- The batch table stores both `project_key` AND `repo_path`.

### Week Identification

Weeks are Monday-aligned. The system uses:
- `get_week_start(date)` → Monday 00:00:00 UTC
- `get_week_end(date)` → Sunday 23:59:59.999999 UTC

The `--weeks N` CLI arg is the only way to address a week range today. Week N=1 means
"the last complete Mon–Sun week". There is no way to address a specific ISO week number
directly without computing the date.

---

## 4. The "Skip Already Classified" Logic

**Location:** `batch_classifier.py :: _get_batches_to_process()` (lines 322-363)

```python
if not force_reclassify:
    query = query.filter(
        DailyCommitBatch.classification_status.in_(["pending", "failed"])
    )
```

- The gate is on `DailyCommitBatch.classification_status`.
- A batch starts as `"pending"` (set during collect / Stage 1).
- After successful classification it is set to `"completed"`.
- Batches with status `"completed"` are **skipped** unless `force_reclassify=True`.
- Failed batches (`"failed"`) are always retried.
- The existing `--reclassify` CLI flag already passes `force_reclassify=True`, which
  bypasses this filter globally for all repos and all weeks in the date window.

Note: `ClassifyResult.skipped_batches` is declared in `pipeline_types.py` but
`classify_date_range()` does **not currently return a `skipped_batches` key** in its
dict — it only returns `processed_batches` and `total_commits`. So
`result.skipped_batches` is always 0 at present.

---

## 5. Where to Add `--repo` and `--week` Filtering

### CLI Layer (`cli_pipeline_commands.py`)

Add two new options to `classify_command`:

```python
@click.option(
    "--repo",
    "-r",
    "repo_filter",
    multiple=True,
    help="Limit classification to this repo (by name or project_key). Repeatable.",
)
@click.option(
    "--week",
    "week_filter",
    type=str,
    default=None,
    help="Classify only the week containing this date (YYYY-MM-DD).",
)
```

Pass them to `run_classify()`:

```python
result = run_classify(
    cfg=cfg,
    weeks=weeks,
    reclassify=reclassify,
    repo_filter=list(repo_filter),   # new
    week_filter=week_filter,          # new
    progress_callback=...,
)
```

### Pipeline Layer (`pipeline_classify.py :: run_classify`)

Add parameters:

```python
def run_classify(
    cfg,
    weeks: int,
    reclassify: bool = False,
    repo_filter: list[str] | None = None,    # new
    week_filter: str | None = None,           # new
    progress_callback=None,
) -> ClassifyResult:
```

**Repo filtering** — narrow `project_keys`:

```python
all_project_keys = [repo.project_key or repo.name for repo in cfg.repositories]

if repo_filter:
    # Accept either name or project_key
    name_map = {r.name: (r.project_key or r.name) for r in cfg.repositories}
    pkey_set = {(r.project_key or r.name) for r in cfg.repositories}
    resolved = []
    for f in repo_filter:
        if f in pkey_set:
            resolved.append(f)
        elif f in name_map:
            resolved.append(name_map[f])
        else:
            _emit(f"Warning: repo '{f}' not found in config, skipping")
    project_keys = resolved or all_project_keys
else:
    project_keys = all_project_keys
```

**Week filtering** — override start_date / end_date:

```python
if week_filter:
    from .utils.date_utils import get_week_start, get_week_end
    anchor = datetime.strptime(week_filter, "%Y-%m-%d").replace(tzinfo=timezone.utc)
    start_date = get_week_start(anchor)
    end_date   = get_week_end(anchor)
    _emit(f"Week filter: {start_date.strftime('%Y-%m-%d')} to {end_date.strftime('%Y-%m-%d')}")
# else: existing weeks-based range calculation unchanged
```

### Batch Classifier Layer (`batch_classifier.py`)

No changes needed. `classify_date_range()` already accepts:
- `project_keys` — narrows the `DailyCommitBatch.project_key.in_(...)` filter
- `force_reclassify` — already bypasses the "skip completed" gate

---

## 6. Force Re-Classification for a Specific Repo+Week

The `--reclassify` flag already exists but acts globally. To make it targeted:

**Option A (Simplest): Use combined flags**

```bash
gfa classify -c config.yaml --repo my-service --week 2026-02-17 --reclassify
```

This combination would:
1. Narrow `project_keys` to `["MY_SERVICE"]`
2. Narrow the date window to the week of 2026-02-17
3. Pass `force_reclassify=True` so completed batches are included

**What gets reset?**

The `DailyCommitBatch.classification_status` for matching rows will be set back to
`"processing"` and then `"completed"` by `_classify_weekly_batches()`. The
`QualitativeCommitData` rows are updated in-place via the upsert in
`_store_commit_classification()`. Nothing is deleted — it is a clean overwrite.

**Option B (Explicit reset before re-running)**

If you want a clean slate that makes the status `"pending"` before running (e.g., to
also count correctly in the status output), add a helper:

```python
# pipeline_classify.py or a new utility
def reset_batch_status(cfg, repo_filter, week_filter):
    from .core.cache import GitAnalysisCache
    from .models.database import DailyCommitBatch
    cache = GitAnalysisCache(cfg.cache.directory)
    with cache.get_session() as session:
        q = session.query(DailyCommitBatch)
        if week_filter:
            anchor = ...
            q = q.filter(DailyCommitBatch.date >= start_date.date(),
                         DailyCommitBatch.date <= end_date.date())
        if repo_filter:
            q = q.filter(DailyCommitBatch.project_key.in_(project_keys))
        q.update({"classification_status": "pending"}, synchronize_session=False)
        session.commit()
```

Option A is sufficient for normal use. Option B is useful for debugging or when the
status display accuracy matters.

---

## 7. Gotchas and Architectural Constraints

### A. `skipped_batches` counter is broken today
`classify_date_range()` does not return a `"skipped_batches"` key. The
`ClassifyResult.skipped_batches` field is always 0. This should be fixed alongside the
new feature — compute it in `_get_batches_to_process()` and return it.

### B. `commit_hash` lookup is repo-agnostic in `_store_commit_classification()`
```python
cached_commit = session.query(CachedCommit).filter(
    CachedCommit.commit_hash == commit_hash
).first()
```
This does NOT filter by `repo_path`. If two repos happen to share a commit hash (e.g.,
forks), the classification could be written to the wrong commit. The fix is to also pass
and filter on `repo_path` in this query.

### C. Date type mismatch in `DailyCommitBatch.date`
`DailyCommitBatch.date` is `Column(DateTime)` (naive, no timezone) even though the
unique index uses it. `_get_batches_to_process()` compares it with `datetime.date()`.
This is fine for SQLite but worth noting when adding a `week_filter` that parses a date
string — use `.date()` when filtering, not the full aware datetime.

### D. `BatchCommitClassifier` opens and closes sessions independently
The session used in `_get_batches_to_process()` is closed before
`_classify_weekly_batches()` opens a new one. The detached-object re-fetch pattern in
`_classify_weekly_batches()` handles this correctly already.

### E. `project_key` vs `repo name` duality
Config lookup needs to handle both. `RepositoryConfig` auto-derives `project_key =
name.upper().replace("-","_")` when not set explicitly. A `--repo my-service` filter
needs to be resolved to `"MY_SERVICE"` before being passed to the classifier.

### F. Weeks-based range still runs even with `--week`
Today `run_classify` always recomputes a weeks-based range. If `--week` is added, the
weeks-based range computation must be **replaced** (not supplemented) for that run, or
the week-filter date must fall inside the computed range, which may not always be true.
The simplest fix: if `week_filter` is set, skip the weeks-based range computation
entirely and use only the anchor week.

---

## 8. Summary of Changes Required

| File | Change |
|---|---|
| `cli_pipeline_commands.py` | Add `--repo` and `--week` options to `classify_command` |
| `pipeline_classify.py` | Add `repo_filter` and `week_filter` params; conditional date range override; repo_keys narrowing |
| `batch_classifier.py` | Return `skipped_batches` count from `classify_date_range()` |
| `batch_classifier_impl.py` | Fix `_store_commit_classification()` to filter by `repo_path` (optional but correct) |
| `pipeline_types.py` | `ClassifyResult.skipped_batches` already exists — just needs to be populated |

The `BatchCommitClassifier.classify_date_range()` interface already supports both
filtering axes (`project_keys` + date range). No changes are needed to the classifier
itself — all changes land at the CLI and pipeline layers.
