# Memory Leak Investigation: gitflow-analytics

**Date:** 2026-02-26
**Analyst:** Research Agent
**Severity:** High — multiple confirmed leaks, one catastrophic

---

## Executive Summary

The investigation found **8 confirmed memory issues** ranging from catastrophic to moderate. The most severe is a quadratic-time, unbounded-memory algorithm in `analyzer.py::_get_commit_branch` that iterates every commit in every branch for every single commit analyzed. A large repo with 10K commits and 50 branches would cause this method to load 500K commit objects into memory per analysis run. Secondary issues include: all commit objects loaded into RAM before processing, unbounded in-memory performance tracking lists, redundant extractor instantiation on every PR, and full `.all()` result sets loaded for verification queries.

---

## Issue Inventory

### CRITICAL — BUG-1: Catastrophic O(N*B) Memory Usage in `_get_commit_branch`

**File:** `src/gitflow_analytics/core/analyzer.py`
**Lines:** 942-948

```python
def _get_commit_branch(self, repo: Repo, commit: git.Commit) -> str:
    """Get the branch name for a commit."""
    # This is a simplified approach - getting the first branch that contains the commit
    for branch in repo.branches:
        if commit in repo.iter_commits(branch):   # <-- CATASTROPHIC
            return branch.name
    return "unknown"
```

**Problem:** `commit in repo.iter_commits(branch)` materializes the ENTIRE commit history of `branch` into an in-memory iterable so Python can perform the `in` membership test. This is called once per commit being analyzed.

- If a repo has 10,000 commits and 50 branches:
  - Each call loads ~50 branches × full commit history = potentially millions of commit objects
  - This method is called once per commit from `_analyze_commit`
  - Total memory: `commits × branches × avg_branch_depth × ~2KB per GitPython commit object`
  - For 1,000 commits × 50 branches × 500-deep history: **~50GB of commit object allocations** (GC may reclaim many, but peak RSS will be enormous)
- This is also **O(N²)** in time complexity.
- The `_get_commit_branch` result is stored in `commit_data["branch"]`, but `branch` is also already directly available from the branch loop variable in `_get_all_branch_commits` / `_get_smart_branch_commits`. The method is used only in `_analyze_commit` which is called from `_process_batch`.

**Fix:** Use `git name-rev` or tag the commit with its source branch at the point it is collected from `iter_commits`, rather than doing a reverse lookup:

```python
# In _get_all_branch_commits / _get_smart_branch_commits, when collecting:
for commit in repo.iter_commits(ref, since=since):
    commit._source_branch = ref.name  # Attach branch at collection time
    ...

# In _analyze_commit:
def _get_commit_branch(self, repo: Repo, commit: git.Commit) -> str:
    # Use the branch attached at collection time (O(1), no extra queries)
    if hasattr(commit, '_source_branch'):
        return commit._source_branch
    # Lightweight fallback using git name-rev subprocess
    try:
        result = repo.git.name_rev(commit.hexsha, '--name-only', '--refs=refs/heads/*')
        return result.split('~')[0].split('^')[0]
    except Exception:
        return "unknown"
```

**Memory impact reduction:** From O(N*B*D) to O(1) per commit.

---

### HIGH — BUG-2: Full Commit List Loaded into RAM Before Processing (`analyzer.py`)

**File:** `src/gitflow_analytics/core/analyzer.py`
**Lines:** 372-437 (`_get_all_branch_commits`), 284-320 (`_get_commits_optimized`)

```python
# Line 415 — no limit, full history materialized
branch_commits = list(repo.iter_commits(ref, since=since))
# ...
commits.extend(branch_commits)     # accumulates ALL branches into one list
# ...
unique_commits = self._deduplicate_commits(commits)  # another full copy
# ...
return sorted(unique_commits, ...)  # yet another copy
```

Then in `analyze_repository`:
```python
commits = self._get_commits_optimized(repo, since, branch)  # all in RAM
total_commits = len(commits)                                 # need len, forces full list
analyzed_commits = []
for batch in self._batch_commits(commits, self.batch_size):  # batch slices the RAM list
```

**Problem:** The entire commit graph is loaded into RAM first as GitPython `Commit` objects (each ~2-5 KB), then deduplicated (second copy), then sorted (third copy), then sliced into batches. For a repo with 50K commits since the start date: `50,000 × 5KB × 3 copies = ~750MB` before any analysis begins.

**Memory impact:** For a 90-day window of an active 100-person team: 50K-100K commits × 5KB = 250-500MB just for commit objects, before analysis dicts are built.

**Fix:** Stream commits through a deduplication set and process in batches without materializing the full list:

```python
def analyze_repository_streaming(self, repo_path, since, branch=None):
    repo = Repo(repo_path)
    seen_hashes = set()
    batch = []

    for ref in repo.refs:
        for commit in repo.iter_commits(ref, since=since):
            if commit.hexsha in seen_hashes:
                continue
            seen_hashes.add(commit.hexsha)
            commit._source_branch = ref.name
            batch.append(commit)
            if len(batch) >= self.batch_size:
                yield from self._process_batch(repo, repo_path, batch, since)[0]
                batch.clear()

    if batch:
        yield from self._process_batch(repo, repo_path, batch, since)[0]
```

This reduces peak RAM from O(total_commits) to O(batch_size).

---

### HIGH — BUG-3: `daily_commits` Dict Held in RAM Alongside Database Storage

**File:** `src/gitflow_analytics/core/data_fetcher.py`
**Lines:** 512-655 (`_fetch_commits_by_day`)

```python
daily_commits = {}          # grows for entire analysis period
all_commit_hashes = set()   # grows unbounded

for day_date in days_to_process:
    day_commits = []
    for branch_name in branches_to_analyze:
        branch_commits = list(repo.iter_commits(...))  # full branch day materialized
        for commit in branch_commits:
            commit_data = self._extract_commit_data(...)  # fat dict per commit
            day_commits.append(commit_data)
            all_commit_hashes.add(commit.hexsha)

    daily_commits[day_str] = day_commits    # accumulated — never cleared
    self._store_day_commits_incremental(...)  # stored to DB, but dict not freed
```

The dict `daily_commits` is returned at line 655 and also stored in the result dict at line 310: `"daily_commits": daily_commits`. This means the entire in-memory representation of every commit for the entire analysis window is held simultaneously in RAM, even though the data was already written to SQLite incrementally.

**Memory impact:** For 90 days, 10 repos, 500 commits/day: `90 × 500 × ~3KB per commit dict = ~135MB` per repo. With 10 repos and 3 parallel workers: **~4GB peak**.

**Fix:** After `_store_day_commits_incremental` succeeds, drop the day's data from `daily_commits`:

```python
if day_commits:
    self._store_day_commits_incremental(repo_path, day_str, day_commits, project_key)
    # Drop from memory immediately — data is now in SQLite
    daily_commits[day_str] = len(day_commits)  # store count, not data
    del day_commits  # explicit release

# In fetch_repository_data, don't include daily_commits in return value:
# "daily_commits": daily_commits  <-- REMOVE THIS LINE (line 310)
```

---

### HIGH — BUG-4: `sample_commits` Loads Full Branch History for Progress Estimation

**File:** `src/gitflow_analytics/core/data_fetcher.py`
**Lines:** 517-523

```python
# Count total commits first for Rich display
try:
    for branch_name in branches_to_analyze[:1]:  # Sample from first branch
        sample_commits = list(               # <-- FULL BRANCH MATERIALIZED
            repo.iter_commits(branch_name, since=start_date, until=end_date)
        )
        # Estimate based on first branch
        len(sample_commits) * len(branches_to_analyze)
        break
```

The result of `len(sample_commits) * len(branches_to_analyze)` is computed but **never assigned to a variable** (it's a bare expression). The `sample_commits` list is loaded into RAM purely to get a commit count for a display estimate that is not even used.

**Memory impact:** Loading one full branch's commits for a 90-day window: up to 50MB wasted, immediately garbage-collected but causes a memory spike.

**Fix:** Use `--count` flag or `rev-list --count` to get count without materializing:

```python
try:
    count_str = repo.git.rev_list(
        branches_to_analyze[0],
        '--count',
        f'--after={start_date.isoformat()}',
        f'--before={end_date.isoformat()}',
    )
    estimated_count = int(count_str.strip()) * len(branches_to_analyze)
except Exception:
    estimated_count = len(days_to_process) * BatchSizes.COMMITS_PER_WEEK_ESTIMATE
```

---

### HIGH — BUG-5: `_extract_pr_data` Instantiates Fresh Extractors Per PR Call

**File:** `src/gitflow_analytics/integrations/github_integration.py`
**Lines:** 423-427

```python
def _extract_pr_data(self, pr, fetch_reviews: bool = False) -> dict[str, Any]:
    from ..extractors.story_points import StoryPointExtractor
    from ..extractors.tickets import TicketExtractor

    sp_extractor = StoryPointExtractor()           # new instance every PR
    ticket_extractor = TicketExtractor(...)        # new instance every PR
```

`StoryPointExtractor` and `TicketExtractor` both compile regex patterns at construction time. Creating a new instance for every PR call (which may happen for hundreds or thousands of PRs) causes:
1. Redundant regex compilation (CPU + memory per call)
2. Multiple live extractor objects if PRs are processed concurrently

**Memory impact:** If processing 2,000 PRs, 2,000 extractor pairs are created. Each compiled regex object can be tens of KB. Estimated: 2,000 × 2 × ~50KB = ~200MB in regex state.

**Fix:** Move to instance-level attributes initialized in `__init__`:

```python
class GitHubIntegration:
    def __init__(self, token, cache, ...):
        ...
        from ..extractors.story_points import StoryPointExtractor
        from ..extractors.tickets import TicketExtractor
        self._sp_extractor = StoryPointExtractor()
        self._ticket_extractor = TicketExtractor(allowed_platforms=self.allowed_ticket_platforms)

    def _extract_pr_data(self, pr, fetch_reviews=False):
        # Use self._sp_extractor and self._ticket_extractor
        pr_text = f"{pr.title} {pr.body or ''}"
        story_points = self._sp_extractor.extract_from_text(pr_text)
        tickets = self._ticket_extractor.extract_from_text(pr_text)
```

---

### MEDIUM — BUG-6: `_verify_storage` Loads All Committed Hashes into RAM

**File:** `src/gitflow_analytics/core/data_fetcher.py`
**Lines:** 1577-1586

```python
stored_commits = (
    session.query(CachedCommit)
    .filter(
        CachedCommit.commit_hash.in_(expected_hashes),
        CachedCommit.repo_path == str(repo_path),
    )
    .all()   # loads full ORM objects
)
stored_hashes = {commit.commit_hash for commit in stored_commits}
```

This loads full `CachedCommit` ORM objects (which include message text, all metrics, JSON fields) just to check if hashes exist. For 10K commits, each `CachedCommit` row deserialized into Python is ~5-10 KB: **50-100MB** for a verification step.

**Fix:** Query only the `commit_hash` column:

```python
stored_hashes = set(
    row[0]
    for row in session.query(CachedCommit.commit_hash)
    .filter(
        CachedCommit.commit_hash.in_(expected_hashes),
        CachedCommit.repo_path == str(repo_path),
    )
    .all()
)
```

This reduces memory by ~50-100x for this query.

---

### MEDIUM — BUG-7: `get_fetch_status` Loads All Batches as ORM Objects for Summation

**File:** `src/gitflow_analytics/core/data_fetcher.py`
**Lines:** 1626-1634

```python
batches = (
    session.query(DailyCommitBatch)
    .filter(...)
    .all()   # loads all ORM objects
)
# ...
total_commits = sum(batch.commit_count for batch in batches)    # only needs sum
classified_batches = sum(
    1 for batch in batches if batch.classification_status == "completed"
)
```

Also:
**File:** `src/gitflow_analytics/core/data_fetcher.py`
**Lines:** 1580-1583

The same `.all()` pattern appears in `get_fetch_status` for correlations query that only needs `.count()`.

**Fix:** Use database aggregations instead:

```python
from sqlalchemy import func, case

result = (
    session.query(
        func.count().label('total_batches'),
        func.sum(DailyCommitBatch.commit_count).label('total_commits'),
        func.sum(
            case((DailyCommitBatch.classification_status == 'completed', 1), else_=0)
        ).label('classified_batches'),
    )
    .filter(
        DailyCommitBatch.project_key == project_key,
        DailyCommitBatch.repo_path == str(repo_path),
    )
    .one()
)
```

---

### MEDIUM — BUG-8: `NLPEngine.processing_times` and `PerformanceMetrics.method_performance` Are Unbounded Lists

**File:** `src/gitflow_analytics/qualitative/core/nlp_engine.py`
**Line:** 78, 164

```python
self.processing_times = []       # never bounded
...
self.processing_times.append(processing_time)  # grows forever
```

**File:** `src/gitflow_analytics/qualitative/utils/metrics.py`
**Line:** 51, 95

```python
self.method_performance = defaultdict(list)    # inner lists never bounded
...
self.method_performance[method_used].append(processing_time_ms / items_processed)
```

`PerformanceMetrics.processing_metrics` correctly uses `deque(maxlen=10000)`, but `method_performance` values are plain `list` objects that grow without bound for the lifetime of the NLP engine instance. For a long analysis run with 100K commits processed as batches of 100, `method_performance["nlp"]` accumulates 1,000 floats — modest here, but `NLPEngine.processing_times` also grows to 1,000 entries.

**Fix:**

```python
# nlp_engine.py
from collections import deque
self.processing_times: deque[float] = deque(maxlen=1000)

# metrics.py
from collections import deque
self.method_performance: defaultdict[str, deque[float]] = defaultdict(lambda: deque(maxlen=5000))
```

---

### LOW — BUG-9: `branch_health.py` Loads Full Branch History for Divergence Check

**File:** `src/gitflow_analytics/metrics/branch_health.py`
**Lines:** 190, 202-205, 282

```python
# Line 190: loads ALL commits in main branch to check if one commit is in it
main_commit_hashes = set(commit.hexsha for commit in repo.iter_commits(main_branch))

# Lines 202-205: loads full ahead/behind history
ahead = list(repo.iter_commits(f"{main_branch.name}..{branch.name}"))
behind = list(repo.iter_commits(f"{branch.name}..{main_branch.name}"))

# Line 282: loads ALL commits in a branch to get first commit (just needs last element)
commits = list(repo.iter_commits(branch))
first_commit = commits[-1]
```

**Fix:**

```python
# For is_merged check — use git merge-base
merge_base = repo.merge_base(branch.commit, main_branch.commit)
is_merged = merge_base and merge_base[0].hexsha == branch.commit.hexsha

# For ahead/behind count only — use --count
ahead_count = int(repo.git.rev_list('--count', f'{main_branch.name}..{branch.name}'))
behind_count = int(repo.git.rev_list('--count', f'{branch.name}..{main_branch.name}'))

# For branch creation date — use max_count=1 with reverse
first_commit = next(repo.iter_commits(branch, reverse=True, max_count=1), None)
```

---

## Memory Impact Summary

| Issue | Severity | Estimated Peak Impact | Affected Code |
|-------|----------|----------------------|---------------|
| BUG-1: `_get_commit_branch` O(N*B) scan | CRITICAL | Gigabytes, unbounded | `analyzer.py:946` |
| BUG-2: Full commit list before processing | HIGH | 250-750 MB | `analyzer.py:415` |
| BUG-3: `daily_commits` held in RAM | HIGH | 135 MB × workers | `data_fetcher.py:617` |
| BUG-4: `sample_commits` wasted load | HIGH | 50 MB spike | `data_fetcher.py:519` |
| BUG-5: Extractor re-instantiation per PR | HIGH | 200 MB (2K PRs) | `github_integration.py:426` |
| BUG-6: Full ORM objects for hash check | MEDIUM | 50-100 MB | `data_fetcher.py:1577` |
| BUG-7: `.all()` for aggregatable queries | MEDIUM | 10-50 MB | `data_fetcher.py:1626` |
| BUG-8: Unbounded metrics lists | MEDIUM | Low (~1 MB but grows) | `nlp_engine.py:78` |
| BUG-9: Full history for divergence | LOW | 50-200 MB per branch | `branch_health.py:190` |

---

## Recommended Fix Priority

1. **Immediately**: BUG-1 — the `_get_commit_branch` O(N*B) scan is the most likely OOM cause for any non-trivial repo. A 5-line fix eliminates gigabytes.
2. **Short-term**: BUG-3 and BUG-4 — nullify the `daily_commits` retention and sample_commits waste.
3. **Short-term**: BUG-2 — restructure `analyze_repository` to stream commits rather than load all.
4. **Short-term**: BUG-5 — move extractor construction to `__init__`.
5. **Medium-term**: BUG-6, BUG-7 — replace full ORM loads with projection or aggregate queries.
6. **Housekeeping**: BUG-8, BUG-9 — cap metrics lists and use git primitives for divergence.

---

## Additional Observations

### Large JSON Column: `DetailedTicketData.platform_data`

`src/gitflow_analytics/core/data_fetcher.py` line 1245 stores the **full JIRA issue response** in a JSON column (`platform_data=issue_data`). When this is loaded by `.all()` queries (e.g., `data_fetcher.py:1146`), every byte of the raw JIRA API response is deserialized into Python dicts. For large projects with thousands of tickets and rich JIRA fields, each ticket can be 5-50 KB. 10K tickets × 20KB = 200MB just from ticket data.

**Fix:** Either store only the fields needed for analysis, or use lazy loading / column projection when reading ticket data.

### Session Management in `_fetch_detailed_tickets`

`src/gitflow_analytics/core/data_fetcher.py` lines 1136-1200 use `session = self.database.get_session()` (not a context manager) and only close in `finally`. This is correct, but the session accumulates all ticket ORM objects in its identity map throughout the entire ticket fetch loop. For large ticket sets, calling `session.expire_all()` or `session.expunge_all()` periodically after committing each batch would allow Python to garbage-collect the ORM objects.

### `all_commit_hashes` Set in `data_fetcher.py`

`data_fetcher.py:514`: `all_commit_hashes = set()` grows to hold one SHA per unique commit processed. For 100K commits: `100,000 × 40 bytes (hex SHA string) = ~4MB`. This is acceptable but worth noting as a lower bound.
