# Cache Investigation: Why Only 94/146 Repos Have Data

**Date:** 2026-02-27
**Trigger:** User ran `gfa collect -w 16 -c gitflow-all-engineers.yaml` from `/Users/masa/Duetto/repos/`
**Config:** `/Users/masa/Duetto/repos/gitflow-all-engineers.yaml` (146 repos)
**Cache DB:** `/Users/masa/Duetto/repos/.gitflow-cache/gitflow_cache.db`

---

## TL;DR: Not a Bug

The cache is correct. All 146 repos were processed. 52 repos have zero commits in the 16-week collection window (Nov 3, 2025 - Feb 22, 2026) because they are inactive/archived. The `cached_commits` table only stores rows for repos that actually had commits.

The premise of "only 8 repos" was incorrect - the cache has **94 repos** with actual commit data.

---

## Actual Counts

| Category | Count |
|---|---|
| Repos in config | 146 |
| Repos processed (have `weekly_fetch_status` rows) | 146 |
| Repos with commits in `cached_commits` | 94 |
| Repos with zero commits in 16-week window | 52 |

---

## Original Query Error

The user's proposed diagnostic query would have failed:
```sql
SELECT repo_name, COUNT(*) as commit_count FROM cached_commits GROUP BY repo_name ORDER BY repo_name;
```

The `cached_commits` table does **not** have a `repo_name` column. The correct column is `repo_path`. This would have returned an error, not "8 repos".

Correct query:
```sql
SELECT replace(repo_path, '/Users/masa/Duetto/repos/', '') as repo_name,
       COUNT(*) as commit_count
FROM cached_commits
GROUP BY repo_path
ORDER BY repo_path;
```

---

## Root Cause: 52 Repos Are Legitimately Inactive

All 52 repos that have no data in `cached_commits` were fully processed by the collection pipeline. They show `status = 'completed'` and `commit_count = 0` in `repository_analysis_status`. They also have all 16 weeks logged in `weekly_fetch_status` with `commit_count = 0`.

**These repos have no commits in the Nov 3, 2025 - Feb 22, 2026 window on any branch.**

Verified against all branches, not just `main`.

### The 52 Inactive Repos

All exist on disk at `/Users/masa/Duetto/repos/`. All were fetched. All had 0 commits in 16 weeks.

| Repo | Last Commit Date | Notes |
|---|---|---|
| advance-chatbot | 2025-05-19 | Archived |
| advance-domain | 2025-01-29 | Archived |
| athena | 2020-01-10 | Abandoned |
| aws-cost-explorer | (no commits) | Empty repo |
| bb-whitbread-poc | (unknown) | PoC |
| beacon-cookie-proto | (unknown) | Proto |
| booking-engine-connect | 2022-06-17 | Archived |
| booking-engine-proto | 2022-? | Archived |
| cds-proto | (unknown) | Archived |
| code-deployment | (unknown) | Inactive |
| common-helm-charts | (unknown) | No recent activity |
| common-orchestrator | (unknown) | No recent activity |
| dari-play | (unknown) | Archived |
| docker-base-java-tools | (unknown) | Archived |
| duetto-common-java | (unknown) | Inactive |
| duetto-docs-site | (unknown) | Inactive |
| duetto-it | (unknown) | Inactive |
| duetto_modeling | 2024-12-26 | Inactive |
| gcp-import-prototype | 2023-11-02 | PoC, archived |
| generator-dr | (unknown) | Archived |
| group-forecast-service | 2022-12-12 | Archived |
| historical-stream-loader | (no commits) | Empty repo |
| insight | (unknown) | Archived |
| interviews | (unknown) | Internal |
| ipgeo | 2025-01-29 | Inactive |
| it-okta | (unknown) | Inactive |
| jenkins-configs | (unknown) | Inactive |
| kinesis-firehose-proto | (unknown) | Inactive |
| loyalty | 2019-04-16 | Abandoned |
| market-events-generator | (unknown) | Inactive |
| meltano | 2024-08-08 | Inactive |
| microfrontend-poc | 2025-10-17 | PoC (just outside window) |
| mxbus | 2022-04-12 | Archived |
| notifications-prototype | (unknown) | PoC |
| openspace-backend | (unknown) | Inactive |
| openspace-backend-kpi | (unknown) | Inactive |
| openspace-frontend | (unknown) | Inactive |
| openspace-generic-frontend | (unknown) | Inactive |
| openspace-rscript | (unknown) | Inactive |
| ops-gh-actions-test | (unknown) | Test repo |
| personalshopper | (unknown) | PoC |
| phabricator | (unknown) | Archived |
| pricing | 2019-09-12 | Abandoned (on `playground` branch) |
| pricing-domain | (unknown) | Inactive |
| pricing-orchestrator | (unknown) | Inactive |
| rl_pricing | (no commits) | Empty repo |
| sample-app | (unknown) | Inactive |
| science | 2024-03-09 | Inactive |
| system-integration-tests | (unknown) | Inactive |
| third-party-integ-svc | (unknown) | Inactive |
| thirdparty | 2023-10-20 | Archived |
| xsd-schemas | 2025-07-02 | Last commit July 2025 (before window) |

### Special Cases: Empty Repos

Three repos have no commits at all (not just inactive):
- `aws-cost-explorer`: `fatal: your current branch 'main' does not have any commits yet`
- `historical-stream-loader`: `fatal: your current branch 'main' does not have any commits yet`
- `rl_pricing`: ERROR (no commits)

These are initialized git repos that were never populated.

### Branch Divergence (Not the Cause)

30 of the 52 repos use non-`main` branches (master/develop/other), but this is **not** the cause of missing data. The collection code (`_get_branches_to_analyze`) analyzes ALL local branches regardless of the configured branch, and falls back to any accessible branch. These repos were confirmed to have 0 commits on ALL branches within the window.

---

## Code Path: How Zero-Commit Repos Are Handled

**File:** `/Users/masa/Projects/gitflow-analytics/src/gitflow_analytics/pipeline_collect.py`

1. `run_collect()` calls `data_fetcher.fetch_repository_data()` for each repo
2. `fetch_repository_data()` calls `_fetch_commits_by_day()` (in `data_fetcher_git.py`)
3. `_fetch_commits_by_day()` iterates all branches via `_get_branches_to_analyze()`, then iterates day-by-day across the window
4. For each day: `repo.iter_commits(branch, since=day_start, until=day_end)` returns 0 commits
5. `daily_commits` dict remains empty `{}`
6. Back in `fetch_repository_data()`: all weeks get marked in `weekly_fetch_status` with `commit_count=0` (line 385-401)
7. Back in `run_collect()`: `cache.mark_repository_analysis_complete()` is called with `commit_count=0`
8. `repository_analysis_status` gets `status='completed', commit_count=0`
9. **No row is written to `cached_commits`** because there are no commits to cache

This is correct, intentional behavior. The code at lines 385-401 of `data_fetcher.py` explicitly says:
```python
# Mark fetched weeks as cached regardless of commit count.
# Even weeks with 0 commits must be marked so we don't re-fetch
# them on the next run (the repo simply had no activity that week).
```

---

## Key Insight: The Week Cache Prevents Re-Fetching

On subsequent `gfa collect` runs, the 52 inactive repos will be **skipped entirely** because all 16 weeks already exist in `weekly_fetch_status`. The check in `data_fetcher.fetch_repository_data()`:

```python
weeks_to_fetch = self.cache.get_missing_weeks(repo_path_str, required_weeks)

if not weeks_to_fetch:
    # Every required week is already cached — return a lightweight summary
    cached_count = self._count_cached_commits(repo_path_str, start_date, end_date)
    return { "stats": { "total_commits": cached_count, ... } }
```

This is efficient: inactive repos cost almost no time on subsequent runs.

---

## Recommendations

1. **No action needed** - the cache is correct. All 146 repos were processed.

2. **Config cleanup (optional)**: The 52 inactive repos could be removed from `gitflow-all-engineers.yaml` to reduce noise. The ones that are clearly abandoned (pre-2023 last commit, empty repos) are safe candidates for removal.

3. **Branch configuration audit**: Repos using `pricing` (on branch `playground`) and `rl_pricing` (empty) should be removed from the config entirely.

4. **`microfrontend-poc` borderline case**: Last commit Oct 17, 2025 - just 2.5 weeks before the window start (Nov 3, 2025). If the collection window were expanded to 18+ weeks, this repo would appear.

5. **Diagnosing "8 repos"**: If the user saw only 8 repos in some output, that was likely from `gfa analyze` run over a subset of the data (the `daily_metrics.db` only has data for repos that went through the LLM classification step), not from the raw `cached_commits` table.
