# CI Failure Patterns

Known failure patterns across Pysae repos, with diagnostic signals and typical fixes.

## Build failures

### TypeScript compile error
- **Signal**: `error TS`, `Type '...' is not assignable`, `Cannot find module`
- **Repos**: op, driver, info, editor, screen, gtfsrt-to-siri
- **Fix**: Fix the type error in source. Check `tsconfig.json` paths if module not found.

### Python import/syntax error
- **Signal**: `ModuleNotFoundError`, `ImportError`, `SyntaxError`
- **Repos**: api, prefect, nav-data-processor
- **Fix**: Check `requirements.txt` / `pyproject.toml` for missing dep. Check Python version.

### Orval codegen mismatch
- **Signal**: `orval`, `generated types`, `openapi`, type errors on API client
- **Repos**: op (primary consumer of generated API types)
- **Fix**: Regenerate Orval types (`yarn orval`). Check if API spec changed.

## Test failures

### Vitest / Jest
- **Signal**: `FAIL`, `AssertionError`, `expect(`, `toEqual`, `toMatchSnapshot`
- **Repos**: op, driver, info, gtfsrt-to-siri
- **Fix**: Read test + source. Check if snapshot needs update (`-u` flag).

### Pytest
- **Signal**: `FAILED`, `assert`, `E   `, `fixtures`, `conftest`
- **Repos**: api, prefect
- **Fix**: Read test + source. Check fixture setup. Check DB migrations if test uses DB.

### Snapshot outdated
- **Signal**: `Snapshot`, `toMatchSnapshot`, `obsolete snapshot`
- **Fix**: `yarn test -u` or `npx vitest run -u` to update snapshots.

## Lint / SAST

### ESLint
- **Signal**: `eslint`, `error`, `warning`, rule names like `no-unused-vars`
- **Fix**: Apply autofix `yarn lint --fix` or fix manually.

### Semgrep
- **Signal**: `semgrep`, `findings`, `severity`
- **Repos**: op, gtfs-validator, prefect, screen, editor
- **Fix**: Address finding or add `# nosemgrep: <rule>` with justification.

### Ruff / mypy
- **Signal**: `ruff`, `mypy`, `error:`, type annotations
- **Repos**: api, prefect
- **Fix**: `ruff check --fix .` or fix type annotations.

### Prettier / formatting
- **Signal**: `prettier`, `formatting`, `--check`
- **Fix**: `yarn prettier --write .`

## Dependency issues

### Yarn resolution conflict
- **Signal**: `YN0002`, `Resolution`, `can't be resolved`, `peer dependency`
- **Fix**: Check `yarn.lock` consistency. Run `yarn install` locally, commit lockfile.

### npm audit / yarn audit
- **Signal**: `audit`, `vulnerability`, `PYSAE_NPM_AUTH_TOKEN`
- **Repos**: op (needs `PYSAE_NPM_AUTH_TOKEN` for private registry)
- **Fix**: Token may need refresh. Check CI/CD variable `PYSAE_NPM_AUTH_TOKEN`.

### pip / poetry conflict
- **Signal**: `ResolutionImpossible`, `version conflict`, `pip install` failure
- **Fix**: Update `requirements.txt` or `pyproject.toml`. Check Python version compatibility.

## Docker issues

### Image build failure
- **Signal**: `docker build`, `COPY failed`, `RUN`, `layer`, `ENOENT`
- **Fix**: Check Dockerfile context (`.dockerignore`), missing files, base image availability.

### Registry auth
- **Signal**: `unauthorized`, `401`, `authentication required`, `registry.gitlab.com`
- **Fix**: Check `CI_REGISTRY_PASSWORD` / `CI_JOB_TOKEN`. May be a GitLab token rotation.

## Infrastructure

### Runner timeout / OOM
- **Signal**: `Job failed (system failure)`, `OOMKilled`, `terminated`, `timed out`
- **Fix**: Retry. If recurring: increase `timeout` in `.gitlab-ci.yml` or request more runner resources.

### Network issues
- **Signal**: `ETIMEDOUT`, `ECONNREFUSED`, `Could not resolve host`, `502 Bad Gateway`
- **Fix**: Usually transient. Retry the job. If recurring: check DNS/proxy config.

## Config issues

### Missing CI/CD variable
- **Signal**: variable name appears as empty string or literal `$VARNAME` in log
- **Fix**: Check Settings > CI/CD > Variables. Verify scope (environment, protected branches).

### Template include failure
- **Signal**: `include`, `could not fetch`, `gitlab-templates`, `404`
- **Repos**: all (shared templates from `pysae/infra/gitlab-templates`)
- **Fix**: Check template ref (`2.x` branch). Verify project access token.

### Wrong ref / branch
- **Signal**: `fatal: reference is not a tree`, `pathspec`, `branch not found`
- **Fix**: Check branch name in `.gitlab-ci.yml` includes. Default branch is `main` (not `develop`).

## Deploy issues

### Helm error
- **Signal**: `helm`, `Error:`, `UPGRADE FAILED`, `release`, `values`
- **Fix**: Check `helm/values.yaml`. Compare with working env (dev vs prod).

### ArgoCD sync failure
- **Signal**: `argocd`, `sync`, `OutOfSync`, `Degraded`, `ComparisonError`
- **Fix**: Check ArgoCD app config in `argo-apps/`. Verify image tag and values.

## Mobile-specific (driver / info)

### Android build (driver)
- **Signal**: `Gradle`, `BUILD FAILED`, `AAPT`, `flavor`, `signing`
- **Image**: `registry.gitlab.com/pysae/infra/docker-images/yarn-android:python-jod-platform-35`
- **Fix**: Check `build.gradle`, signing config. Note: `.v5` package name is legacy.

### iOS build (info)
- **Signal**: `xcodebuild`, `archive`, `signing`, `provisioning profile`
- **Fix**: Check signing certificates, provisioning profiles. Usually done on Mac runner.

## Flaky patterns

### Race condition in tests
- **Signal**: intermittent failure, different assertion each time, timing-related
- **Fix**: Add `retry: 1` in CI config as temporary fix. Fix the underlying race condition.

### External service dependency
- **Signal**: `timeout`, external API URL in logs, `503`, `429`
- **Fix**: Add retry/mock. Consider if test should be isolated from external services.

## Pysae transient quirks (retry once before debugging)

These recur across Pysae pipelines and are NOT code bugs. On any of them, retry the same job **once** before deeper diagnosis or escalation.

### ArgoCD first-deploy race
- **Signal**: `FailedPrecondition`, `another operation is already in progress`
- **When**: `deploy_*` / `redeploy`, typically the first deploy on a fresh pipeline
- **Fix**: Retry the same job. The concurrent ArgoCD operation clears within seconds.

### Runner eviction after success
- **Signal**: `EvictionByEvictionAPI`; a red `deploy_*` right after the log already showed `Synced` + `Healthy`
- **When**: spot/preemptible runner evicted after the deploy already applied
- **Fix**: The deploy already landed. Re-check with `/ci-deploy-status` before retrying; usually nothing to redo.

### test_api routes count flake
- **Signal**: `routes.test.ts` assertion such as `toHaveLength(151|61|13)` off by one
- **Repos**: shift/app (api)
- **Fix**: Known count flake. Retry the job once before investigating the code.

### glab auth crash (year-2026 token parse)
- **Signal**: `glab` exits parsing the token expiry / a time-parse error
- **Fix**: Pass `--hostname gitlab-pysae` to bypass the auth time-parse bug.
