# Data Fetching Architecture Research
**Date**: 2026-02-26
**Purpose**: Understand current data fetching system to inform incremental data collection design

---

## Executive Summary

The system already has a **complete incremental fetch infrastructure** implemented. The `GitDataFetcher` class in `core/data_fetcher.py` uses `RepositoryFetchStatus` records in SQLite to track per-repository fetch timestamps and skip already-cached commit ranges. The system is two-step: (1) fetch raw git commits into SQLite, then (2) classify them with LLM. The incremental mechanism is gated by the `incremental_mode` flag, which is exposed through the CLI as `--incremental/--no-incremental`.

---

## 1. Current Data Flow

```
CLI (analyze command)
  |
  +--> [Cache-first check]
  |      cache.get_repository_analysis_status(repo, start, end, config_hash)
  |      → If "completed" status AND commits physically present → skip fetch
  |
  +--> [Step 1: GitDataFetcher.fetch_repository_data()]
  |      |
  |      +--> cache.get_fetch_status(repo_path)
  |      |      → Returns last_successful_fetch timestamp from RepositoryFetchStatus
  |      |
  |      +--> [Incremental mode decision]
  |      |      If last_successful_fetch > start_date:
  |      |        effective_start_date = last_successful_fetch  (incremental)
  |      |      Else:
  |      |        effective_start_date = start_date            (full scan)
  |      |
  |      +--> _fetch_commits_by_day(repo_path, effective_start_date, end_date, ...)
  |      |      |
  |      |      +--> git Repo(repo_path)
  |      |      +--> _update_repository(repo)   ← git fetch + pull
  |      |      +--> _get_branches_to_analyze(repo, branch_patterns)
  |      |      |
  |      |      +--> [Day loop: effective_start_date → end_date]
  |      |             For each day:
  |      |               For each branch:
  |      |                 repo.iter_commits(branch, since=day_start, until=day_end)
  |      |                 _extract_commit_data(commit, ...)
  |      |               _store_day_commits_incremental(repo_path, day_str, commits, ...)
  |      |               → bulk-inserts to cached_commits via cache.bulk_store_commits()
  |      |
  |      +--> _extract_all_ticket_references(daily_commits)
  |      +--> _fetch_detailed_tickets(ticket_ids, jira_integration, ...)
  |      +--> _build_commit_ticket_correlations(daily_commits, repo_path)
  |      +--> _store_daily_batches(daily_commits, repo_path, project_key)
  |             → Creates/updates DailyCommitBatch records (metadata only)
  |
  +--> cache.update_fetch_status(repo_path, end_date, successful, commits_added, ...)
  |      → Updates RepositoryFetchStatus.last_successful_fetch = end_date (on success)
  |
  +--> cache.mark_repository_analysis_complete(repo_path, start, end, ...)
  |      → Creates RepositoryAnalysisStatus record with status="completed"
  |
  +--> [Step 2: Batch LLM classification]
         Reads from daily_commit_batches, classifies, writes to qualitative_commits
```

---

## 2. Key Method Signatures

### CLI Entry Point: `cli.py`

```python
def analyze(
    config: Path,
    weeks: int,
    output: Optional[Path],
    ...
    force_fetch: bool = False,       # Bypasses RepositoryAnalysisStatus cache check
    incremental: bool = True,        # Enables commit-level incremental fetch
    force_full_fetch: bool = False,  # Alias for --no-incremental (disables incremental)
    reset_incremental: bool = False, # Clears RepositoryFetchStatus watermarks
    ...
) -> None
```

**Effective incremental mode resolution** (line ~1594):
```python
effective_incremental = incremental and not force_full_fetch
# --force-fetch bypasses high-level cache skip but still allows commit-level incremental
# --no-incremental OR --force-full-fetch → full git history scan
```

**Cache-first decision** (line ~1509):
```python
if not force_fetch:
    status = cache.get_repository_analysis_status(repo_path, start_date, end_date, config_hash)
    if status:
        cached_repos.append(...)   # Skip this repo entirely
    else:
        repos_needing_analysis.append(...)
```

### GitDataFetcher: `core/data_fetcher.py`

```python
class GitDataFetcher:
    def __init__(
        self,
        cache: GitAnalysisCache,
        branch_mapping_rules: Optional[dict[str, list[str]]] = None,
        allowed_ticket_platforms: Optional[list[str]] = None,
        exclude_paths: Optional[list[str]] = None,
        skip_remote_fetch: bool = False,
        exclude_merge_commits: bool = False,
        incremental_mode: bool = True,   # KEY: controls per-commit incremental logic
    )

    def fetch_repository_data(
        self,
        repo_path: Path,
        project_key: str,
        weeks_back: int = 4,
        branch_patterns: Optional[list[str]] = None,
        jira_integration: Optional[JIRAIntegration] = None,
        progress_callback: Optional[callable] = None,
        start_date: Optional[datetime] = None,  # Explicit start overrides weeks_back
        end_date: Optional[datetime] = None,    # Explicit end overrides weeks_back
    ) -> dict[str, Any]:
        # Returns: {incremental_fetch: bool, incremental_since: datetime, stats: {...}}
```

**Incremental logic inside `fetch_repository_data`** (lines ~188–225):
```python
if self.incremental_mode:
    fetch_status = self.cache.get_fetch_status(repo_path_str)
    if fetch_status and fetch_status.get("last_successful_fetch"):
        last_success = fetch_status["last_successful_fetch"]
        if last_success > start_date:
            incremental_since = last_success
            incremental_fetch = True   # Only fetch commits AFTER last_success
        # else: last fetch predates requested window → full scan
    # else: no prior fetch record → full scan

effective_start_date = incremental_since if incremental_since else start_date
# effective_start_date is passed to _fetch_commits_by_day
```

### `_fetch_commits_by_day` (core/data_fetcher.py, line ~448)

```python
def _fetch_commits_by_day(
    self,
    repo_path: Path,
    project_key: str,
    start_date: datetime,     # = effective_start_date (already narrowed for incremental)
    end_date: datetime,
    branch_patterns: Optional[list[str]],
    progress_callback: Optional[callable] = None,
) -> dict[str, list[dict[str, Any]]]:
```

Git iteration per day, per branch (line ~687):
```python
repo.iter_commits(branch, since=day_start, until=day_end, reverse=False)
```

Commit count estimation (line ~634):
```python
repo.git.rev_list(branch, "--count", f"--after={start_date.isoformat()}", f"--before={end_date.isoformat()}")
```

### `_store_day_commits_incremental` (data_fetcher.py, line ~2057)

```python
def _store_day_commits_incremental(
    self, repo_path: Path, date_str: str, commits: list[dict[str, Any]], project_key: str
) -> None:
    # Persists full commit dicts to cached_commits via cache.bulk_store_commits()
    # Called immediately after each day's commits are fetched (streaming write)
    # Memory-efficient: full dicts are GC'd after this call
```

### GitAnalyzer (legacy path): `core/analyzer.py`

```python
class GitAnalyzer:
    def analyze_repository(
        self, repo_path: Path, since: datetime, branch: Optional[str] = None
    ) -> list[dict[str, Any]]:
        # Uses _get_commits_optimized(repo, since, branch)
        # which calls repo.iter_commits(ref, since=since)
        # Checks per-commit cache via cache.get_cached_commit(commit_hash)
```

**Note**: `GitAnalyzer` is the older, single-step path. The primary path for production use is `GitDataFetcher` (two-step).

---

## 3. Cache Infrastructure

### SQLite Database Models (`models/database.py`)

**Schema version**: `Database.CURRENT_SCHEMA_VERSION = "2.0"` (timezone-aware timestamps)

#### `RepositoryFetchStatus` — Core of Incremental Fetch
```python
class RepositoryFetchStatus(Base):
    __tablename__ = "repository_fetch_status"

    repository_path = Column(String, primary_key=True)  # One row per repo
    last_fetch_timestamp = Column(DateTime(timezone=True))    # Upper bound of last fetch window
    last_successful_fetch = Column(DateTime(timezone=True))   # KEY: drives incremental --since
    total_commits_cached = Column(Integer, default=0)
    last_commit_hash = Column(String)
    fetch_count = Column(Integer, default=0)
    incremental_fetch_count = Column(Integer, default=0)
    created_at = Column(DateTime(timezone=True))
    updated_at = Column(DateTime(timezone=True))
```

**Separation of `last_fetch_timestamp` vs `last_successful_fetch`**: On a failed fetch, only `last_fetch_timestamp` is updated. `last_successful_fetch` only advances on full success, preventing skipping partially-fetched windows.

#### `RepositoryAnalysisStatus` — High-level Cache-first Gate
```python
class RepositoryAnalysisStatus(Base):
    __tablename__ = "repository_analysis_status"

    repo_path = Column(String)
    analysis_start = Column(DateTime(timezone=True))
    analysis_end = Column(DateTime(timezone=True))
    status = Column(String, default="pending")  # pending|in_progress|completed|failed
    git_analysis_complete = Column(Boolean, default=False)
    commit_count = Column(Integer, default=0)
    config_hash = Column(String)  # Invalidates cache on config change
    analysis_version = Column(String, default="2.0")

    # Unique index: (repo_path, analysis_start, analysis_end)
```

#### `CachedCommit` — Individual Commit Storage
```python
class CachedCommit(Base):
    __tablename__ = "cached_commits"

    repo_path = Column(String)
    commit_hash = Column(String)
    author_name, author_email, message = Column(String) * 3
    timestamp = Column(DateTime(timezone=True))
    branch = Column(String)
    is_merge = Column(Boolean)
    files_changed, insertions, deletions = Column(Integer) * 3
    filtered_insertions, filtered_deletions = Column(Integer) * 2  # After path exclusions
    ticket_references = Column(JSON)
    cached_at = Column(DateTime(timezone=True))
    cache_version = Column(String, default="1.0")

    # Unique index: (repo_path, commit_hash)
    # Also indexed on: timestamp, cached_at
```

#### `DailyCommitBatch` — Two-Step Process Metadata
```python
class DailyCommitBatch(Base):
    __tablename__ = "daily_commit_batches"

    date = Column(DateTime)
    project_key = Column(String)
    repo_path = Column(String)
    commit_count = Column(Integer)
    classification_status = Column(String, default="pending")  # pending|processing|completed
    active_developers = Column(JSON)
    unique_tickets = Column(JSON)

    # Unique index: (date, project_key, repo_path)
```

### Cache Methods (`core/cache.py`)

```python
# Incremental fetch status
cache.get_fetch_status(repository_path: str) -> Optional[dict]
    # Returns: {last_fetch_timestamp, last_successful_fetch, total_commits_cached, ...}

cache.update_fetch_status(
    repository_path: str,
    fetch_timestamp: datetime,
    successful: bool,
    commits_added: int = 0,
    last_commit_hash: Optional[str] = None,
    incremental: bool = False,
) -> None

cache.reset_fetch_status(repository_path: Optional[str] = None) -> int
    # Clears watermarks for one or all repos (used by --reset-incremental)

cache.get_all_fetch_statuses() -> list[dict]

# High-level analysis completion
cache.get_repository_analysis_status(
    repo_path: str,
    analysis_start: datetime,
    analysis_end: datetime,
    config_hash: Optional[str] = None,
) -> Optional[dict]
    # Returns None if: no record, config changed, or no actual commits in DB

cache.mark_repository_analysis_complete(
    repo_path, repo_name, project_key,
    analysis_start, analysis_end, weeks_analyzed,
    commit_count=0, ...
) -> None

# Individual commit cache (used by legacy GitAnalyzer path)
cache.get_cached_commits_bulk(repo_path: str, commit_hashes: list[str]) -> dict[str, dict]
cache.get_cached_commit(commit_hash: str, repo_path: str) -> Optional[dict]
```

---

## 4. The Two-Step Process

```
STEP 1: git fetch  (GitDataFetcher)
  → writes to: cached_commits, daily_commit_batches, commit_ticket_correlations, detailed_tickets
  → updates:   repository_fetch_status, repository_analysis_status

STEP 2: LLM classify  (BatchClassifier / legacy GitAnalyzer)
  → reads from: daily_commit_batches WHERE classification_status='pending'
  → writes to:  qualitative_commits, classification_batches, daily_metrics
  → updates:    daily_commit_batches.classification_status = 'completed'
```

The CLI flag `--use-batch-classification/--use-legacy-classification` (default: batch) selects Step 2 mode. Step 1 is always `GitDataFetcher` in the batch path.

---

## 5. How `--force-fetch` Works

| Flag | Effect |
|------|--------|
| `--force-fetch` | Skips `RepositoryAnalysisStatus` check (re-fetches all repos). Still respects `RepositoryFetchStatus` for commit-level incremental if `--incremental` is set. |
| `--no-incremental` | Sets `effective_incremental=False` → `GitDataFetcher(incremental_mode=False)` → ignores `RepositoryFetchStatus`, full git scan. |
| `--force-full-fetch` | Equivalent to `--no-incremental` (sets `effective_incremental=False`). |
| `--reset-incremental` | Calls `cache.reset_fetch_status()` to delete all `RepositoryFetchStatus` records, then exits. Next run does full scan. Does NOT delete `cached_commits`. |
| `--clear-cache` | Deletes `cached_commits`, `pull_request_cache`, `issue_cache`, `repository_analysis_status`. Effectively wipes everything. |

---

## 6. How `since` Date Passes Through the Chain

```
CLI.analyze()
  weeks=4 → start_date = now() - timedelta(weeks=4)
  end_date = now()

  → GitDataFetcher.fetch_repository_data(start_date=start_date, end_date=end_date)
      → cache.get_fetch_status() → last_successful_fetch
      → effective_start_date = max(last_successful_fetch, start_date)
          [only if last_successful_fetch > start_date, else effective_start_date = start_date]

      → _fetch_commits_by_day(effective_start_date, end_date, ...)
          → days_to_process = [effective_start_date.date() .. end_date.date()]
          → per day: repo.iter_commits(branch, since=day_start, until=day_end)
                     ↑ GitPython --since/--until maps to git log --after/--before

Legacy path (GitAnalyzer):
  CLI passes start_date directly as `since` arg
  → analyze_repository(repo_path, since=start_date)
  → _get_commits_optimized(repo, since=start_date)
  → repo.iter_commits(ref, since=start_date)
```

---

## 7. Git Commands Used

All git operations go through **GitPython** (`git` package), not raw subprocess (except in `core/subprocess_git.py` for auth-safe fallback):

| Operation | GitPython API |
|-----------|---------------|
| Open repo | `Repo(repo_path)` |
| Remote fetch | `repo.remotes.origin.fetch()` |
| Remote pull | `repo.remotes.origin.pull()` |
| Iterate commits | `repo.iter_commits(branch, since=start, until=end)` |
| Count commits | `repo.git.rev_list(branch, "--count", "--after=...", "--before=...")` |
| List branches | `repo.refs`, `repo.remotes` |

**Timeout protection**: All `iter_commits` calls are wrapped in `GitTimeoutWrapper.run_with_timeout()` with `Timeouts.GIT_BRANCH_ITERATION` timeout to prevent hangs.

---

## 8. Integration Points for Incremental Fetching

The incremental fetching infrastructure is **already implemented**. The key integration points are:

### Point A: Commit-level incremental (already exists)
**File**: `core/data_fetcher.py`, lines 188–225
**Method**: `GitDataFetcher.fetch_repository_data()`
**Mechanism**: Reads `RepositoryFetchStatus.last_successful_fetch` and narrows `effective_start_date`

**To modify**: Change the logic that decides `incremental_since`. Currently it uses `last_successful_fetch` as-is. Could be enhanced to handle gaps, multiple windows, or per-branch timestamps.

### Point B: High-level analysis status (already exists)
**File**: `core/cache.py`, line 1538
**Method**: `cache.get_repository_analysis_status()`
**Mechanism**: Checks `RepositoryAnalysisStatus` for "completed" status AND verifies commits are physically present (TTL-checked)

**To modify**: Could add finer-grained period matching (e.g., week-level tracking instead of full period matching).

### Point C: Fetch status update (already exists)
**File**: `core/cache.py`, line 542
**Method**: `cache.update_fetch_status()`
**Mechanism**: Only advances `last_successful_fetch` on full success; partial fetches only update `last_fetch_timestamp`

### Point D: CLI flags (already exists)
**File**: `cli.py`, lines 460–488
**Flags**: `--incremental`, `--force-full-fetch`, `--reset-incremental`

---

## 9. Existing Incremental-Like Logic Already in Place

1. **`RepositoryFetchStatus`** table exists and is populated after every fetch
2. **`GitDataFetcher.incremental_mode`** parameter wires to per-commit range narrowing
3. **`effective_start_date`** is correctly computed from `last_successful_fetch`
4. **Deduplication**: `all_commit_hashes` set prevents double-counting cross-branch commits
5. **Streaming writes**: `_store_day_commits_incremental` writes each day's commits immediately (no all-in-memory accumulation)
6. **Conflict-safe inserts**: `bulk_store_commits` uses INSERT OR IGNORE (upsert) on the `(repo_path, commit_hash)` unique index, so duplicate commits from overlapping incremental windows are harmless
7. **TTL on CachedCommit**: `cached_at` column with TTL check in `get_repository_analysis_status` ensures stale entries force a re-fetch

---

## 10. Recommendations for Extending Incremental Fetching

Given what already exists, the incremental system is largely complete. Gaps to address:

1. **Per-branch tracking**: Currently `RepositoryFetchStatus` tracks one timestamp per repo. If a new branch is added mid-run, the timestamp won't cover it. Consider storing per-branch timestamps in a JSON field.

2. **Gap filling**: If `last_successful_fetch` is older than TTL hours but newer than `start_date`, the current code skips the gap window. The code at line 199 (`if last_success > start_date`) handles this correctly, but the overlap between `incremental_since` and cached commits relies on upsert deduplication—could be made explicit.

3. **`RepositoryFetchStatus` not cleared on `--clear-cache`**: `clear_cache()` deletes commits but not `RepositoryFetchStatus` records. This means after a cache clear, the next run will still try incremental mode (from an old timestamp), potentially missing commits. **Fix**: Call `cache.reset_fetch_status()` inside `clear_cache()` or at minimum when commit table is wiped.

4. **`DailyCommitBatch.classification_status` as incremental anchor**: The `classification_status` field (`pending`/`completed`) is a natural incremental checkpoint for Step 2 (classification). Any extension to incremental classification should read from `WHERE classification_status='pending'`.

5. **`analysis_start/analysis_end` exact match**: `get_repository_analysis_status` requires an exact match on `(repo_path, analysis_start, analysis_end)`. If the user changes `--weeks`, no cached status is found even if most commits are cached. An overlapping-period check would improve cache reuse.

---

## File Reference

| File | Role |
|------|------|
| `src/gitflow_analytics/cli.py` | CLI entry point, two-step orchestration, flag definitions |
| `src/gitflow_analytics/core/data_fetcher.py` | Step 1: raw git fetch, incremental logic, commit storage |
| `src/gitflow_analytics/core/analyzer.py` | Legacy single-step analyzer (still used by identity/training commands) |
| `src/gitflow_analytics/core/cache.py` | Cache reads/writes, fetch status methods, analysis status methods |
| `src/gitflow_analytics/models/database.py` | SQLAlchemy models: CachedCommit, RepositoryFetchStatus, RepositoryAnalysisStatus, DailyCommitBatch, etc. |
| `src/gitflow_analytics/core/subprocess_git.py` | Auth-safe fallback git operations via subprocess |
| `src/gitflow_analytics/core/git_timeout_wrapper.py` | Timeout protection for GitPython operations |
