Metadata-Version: 2.4
Name: coltrace
Version: 0.1.0.dev4
Summary: Static Pandas column lineage, impact analysis, and local dependency graphs
Author: Cristiano Diniz
License-Expression: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: networkx<4,>=3.2
Requires-Dist: typer<1,>=0.12
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: ruff>=0.9; extra == "dev"
Requires-Dist: mypy>=1.14; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: types-networkx>=3.2; extra == "dev"
Provides-Extra: viz
Requires-Dist: pyvis<0.4,>=0.3.2; extra == "viz"
Dynamic: license-file

# ColTrace

Static Pandas column lineage and potential impact analysis for Python projects.
ColTrace reads source files, identifies supported transformations, and connects
column versions across direct function calls. It never imports or executes the
project or reads the project's input data.

**Development preview: `0.1.0.dev4`.** The supported subset is deliberately small.
Unsupported operations produce diagnostics and may interrupt lineage. A graph
with no edge is not proof that a dependency does not exist.

## Install locally

Requires Python 3.11+. From this checkout:

```sh
python -m venv .venv
# Activate .venv using your shell, then:
python -m pip install -e ".[dev,viz]"
coltrace --help
```

On Windows without activation, use `.\.venv\Scripts\python.exe -m coltrace`.
The `viz` extra supplies PyVis for HTML export; basic analysis and JSON need only
NetworkX and Typer. Pandas is not a runtime dependency. Publication to PyPI is a
separate release step; these commands install the local project.

## Try the complete example

```sh
coltrace scan examples/pricing_pipeline
coltrace trace preco_referencia
coltrace impact preco_referencia
coltrace graph preco_referencia --open
coltrace export preco_referencia --format json --output .coltrace/pricing.json
```

The example connects extraction, pricing, and feature functions through explicit
calls in `pipeline.py`. No `prices.csv` file is needed to analyze it.

```text
valor_produto -> preco_bruto -> preco_referencia -> margem
                                      |              |
                                      +--------------+-> margem_percentual
custo ---------------------------------------------> margem
```

`impact preco_referencia` identifies `margem` and `margem_percentual` as direct
potential dependents. There is also a longer path through `margem`. A changed
input does not necessarily change every downstream value.

`scan` writes `.coltrace/scan.json` by default. `--out DIRECTORY` selects another
snapshot directory. Queries load that snapshot explicitly; they do not silently
rescan the project. Rescan after editing source, or use `--project DIRECTORY` to
analyze current files in memory. `--from FILE` selects an existing snapshot.

```sh
coltrace trace margem --project examples/pricing_pipeline --upstream --depth 2
coltrace impact preco_referencia --from .coltrace/scan.json --evidence confirmed
coltrace graph --from .coltrace/scan.json --output .coltrace/all.html
```

HTML files contain their visualization resources and work locally without a
server or CDN. A selected column opens its journey: direct reads, applications,
writes, derived columns and destinations. Frame passage is labeled separately
from a direct column read. Callback control and configuration appear in the
expandable creation history, rather than as direct uses of the selected column.
The graph uses generously spaced, numbered blocks in a vertical flow. The
scrollable sidebar contains the trace history, source code, and locations;
selecting a step highlights its matching graph block, and vice versa. Open Details
or Diagnostics without losing the current selection. The Outcomes tab shows
derived-column destinations and the DataFrames carrying the selected column.
Zoom in/out, use **Fit all**
for an overview, or **Reset view** to return to a readable scale.
`--view columns` shows the column dependency projection; `--operations` includes
operation nodes in that projection. A graph without a selected column shows the
whole project's column dependencies.

```sh
coltrace scan examples/text_pipeline
coltrace trace comment_text --function process_train
coltrace graph comment_text --function process_train --open
```

`trace` defaults to direct column actions, derived-column outcomes, and containing
DataFrames. `--full-history` includes creation ancestry and callback internals,
with control/configuration distinguished from value transformations.
`--diagnostics` also displays dependency lists and relevant diagnostics. Rescan after an
upgrade: schema 0.2 snapshots have no recorded history. See the
[column history guide](docs/column-history.md) and [current checkpoint](docs/history-checkpoint.md).
The [graph workspace checkpoint](docs/graph-workspace-checkpoint.md) describes the
current layout and [additional column checks](docs/other-column-checkpoint.md).
See the [column journey checkpoint](docs/column-journey-checkpoint.md) for the
usage-first view and its acceptance cases.

## Identity and ambiguity

Different DataFrames keep different identities, even when both contain a column
named `score`. Assignments create column versions. `frame["a"] = frame["a"] + 1`
connects the old and new versions; `frame["a"] = 0` cuts the old value's lineage.
History still records that overwrite without adding a false value dependency.

Selection by name uses the latest observed version within each DataFrame. If
several unrelated identities remain, the CLI lists candidates and exits with status 3.
Same-name copies connected by proven pass-through relations form one selectable
family, whose original observed column is selected. A rename, value derivation,
or shared label alone does not establish that equivalence.
Select one explicitly using `--column-id ID` or `--dataframe ID`. Use `--version`
for an older version, and `--module` or `--function` for exact source filters.
IDs are opaque, deterministic within equivalent scans, and may change when source
locations or call contexts change. They are not permanent database keys.

`--depth` counts column-to-column hops. `--relation derive|rename|pass` filters
relations, while `--evidence confirmed|all` filters their evidence. Filtering out
intermediate relations can disconnect a path. Direct and transitive lists use
the shortest eligible distance, so a node reached both ways appears as direct.

## Supported patterns

| Pattern | Support |
| --- | --- |
| Explicit Pandas imports and aliases | Recognized origins; names such as `df` alone are insufficient |
| `read_csv`, `read_parquet` | External schema is unknown; CSV chunk/iterator forms are excluded |
| Literal dictionary `DataFrame` construction | String column keys and literal values |
| `frame["column"]`, `frame.loc[:, "column"]` | Single literal string labels, all rows |
| `frame["c"] = frame["a"] + frame["b"]` | Arithmetic, scalar constants, temporary values, versions |
| `alias = frame` | Same abstract object and shared mutation |
| `rename(columns={"old": "new"})` | Literal, non-colliding map; new DataFrame, or shared mutation with `inplace=True` |
| `assign(b=lambda x: x["a"] * 2)` | Sequential simple lambdas; eager arguments use their original scope |
| `Series.apply(clean_text)` | Resolved scalar callbacks; bounded summaries and inferred evidence |
| `frame[["a", "b"]] = DataFrame(results.tolist(), index=frame.index)` | Fixed callback output width and matching source index |
| `dropna`, literal projection, boolean row filters | Preserve value lineage; predicate consumption tracked separately |
| `to_parquet` / `to_csv` | Located export consumption with literal options |
| Direct synchronous function calls | Explicit arguments/returns, bounded acyclic contexts and explicit imports |
| Annotated, uncalled functions | Separate inferred templates, not evidence of pipeline connections |

See [the supported subset and boundaries](docs/supported-patterns.md) for the exact
contract. Calls, branches, dynamic columns, and Pandas operations outside that
contract produce located diagnostics rather than invented connections.

The [text pipeline example](examples/text_pipeline/README.md) demonstrates
independent train/evaluation flows, multi-output text cleaning, row filters, and
exports. See the [NLP checkpoint](docs/nlp-checkpoint.md) for what was validated
against a real project and which downstream third-party operations remain opaque.

## Evidence and completeness

- `confirmed`: supported syntax and receiver identity establish the relation.
- `inferred`: the relation depends on an explicit assumption, such as an uncalled
  function's DataFrame annotation or an unknown external schema during rename.
- `unresolved`: insufficient evidence across an unsupported boundary.

`complete`, `partial`, and `unresolved` describe how much of an operation is
understood; evidence and completeness are separate. External column reads are
marked as external/unknown origins. They do not identify the real upstream data
source's column definition or guarantee its presence.

Use `--diagnostics` to inspect warnings and errors. Query output also includes
relevant gaps. An inconclusive trace prints `Not determined` for empty dependency
lists and shows the recognized DataFrame origin separately from column evidence.
See [troubleshooting incomplete traces](docs/troubleshooting.md).
The analyzer is not a Python type checker or an execution
validator. Its grammar is limited by the interpreter running ColTrace. Pandas
imports are assumed to refer to the real library; environment/module impersonation
is not resolved.

## Python API

```python
from coltrace import analyze_project
from coltrace.graph import LineageGraph

report = analyze_project("examples/pricing_pipeline")
graph = LineageGraph(report)
column = graph.select("preco_referencia")
impact = graph.impact(column.id)
print(report.to_json())
```

The earlier `scan_project` API remains available for raw occurrences and receiver
evidence. Its discovery report (`0.1`) is a different format from the lineage
snapshot (`0.3`). See [discovery JSON](docs/discovery-json.md) and
[lineage JSON](docs/lineage-json.md). Public facts are dataclasses; their serialized
form is independent of NetworkX and contains no AST or executable objects.

## Configuration

Only the scanned root's `pyproject.toml` is read:

```toml
[tool.coltrace]
exclude = ["generated", "legacy/*.py", "*_test.py"]
```

Glob patterns match case-sensitive, project-relative POSIX paths. Patterns with
no `/` also match individual path components. Built-in exclusions cover `.venv`,
`venv`, `.git`, `__pycache__`, `.coltrace`, pytest/mypy/Ruff caches, `build`, and
`dist`. Linked files/directories and Windows reparse points are skipped.
`ScanConfig(exclude=("generated",))` overrides TOML settings and extends built-ins.

## Development

```sh
python -m pytest -q
python -m ruff check src tests examples
python -m ruff format --check src tests examples
python -m mypy
python -m build
```

Tests include independent expected dependencies, forbidden edges, alias/version
semantics, cross-module calls, CLI, malformed snapshots, offline HTML, and a
sentinel proving that analyzed source is not executed. See the
[accuracy protocol](docs/accuracy.md), [current checkpoint](docs/nlp-checkpoint.md),
[decision log](docs/decisions.md), and [backlog](docs/backlog.md).

The GitHub Actions workflow configures Windows/Linux with Python 3.11–3.13. It
must run in the repository before those combinations can be claimed as verified.

Licensed under the [MIT License](LICENSE). Copyright 2026 Cristiano Diniz.
