Metadata-Version: 2.5
Name: vnpt_data_governance
Version: 0.1.2
Summary: VNPT data governance SDK (build LLM-assisted schema graphs over relational databases): introspect, classify, profile, describe, search and render.
Project-URL: Homepage, https://github.com/hainamnguyen192/vnpt_data_governance
Project-URL: Repository, https://github.com/hainamnguyen192/vnpt_data_governance
Project-URL: Issues, https://github.com/hainamnguyen192/vnpt_data_governance/issues
Author: minhdenthedev, hainamnguyen192
License: MIT
License-File: LICENSE
Keywords: data-governance,data-lineage,data-quality,database,mysql,pii,postgres,schema,sqlite,trino
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.11
Requires-Dist: rustworkx>=0.18.0
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: tqdm>=4.68.4
Provides-Extra: all
Requires-Dist: bm25s>=0.3.9; extra == 'all'
Requires-Dist: openai>=2.45.0; extra == 'all'
Requires-Dist: psycopg[binary]>=3.3.4; extra == 'all'
Requires-Dist: pymysql>=1.2.0; extra == 'all'
Requires-Dist: pystemmer>=3.1.0; extra == 'all'
Requires-Dist: trino>=0.338.0; extra == 'all'
Provides-Extra: mysql
Requires-Dist: pymysql>=1.2.0; extra == 'mysql'
Provides-Extra: openai
Requires-Dist: openai>=2.45.0; extra == 'openai'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.3.4; extra == 'postgres'
Provides-Extra: search
Requires-Dist: bm25s>=0.3.9; extra == 'search'
Requires-Dist: pystemmer>=3.1.0; extra == 'search'
Provides-Extra: trino
Requires-Dist: trino>=0.338.0; extra == 'trino'
Description-Content-Type: text/markdown

# vnpt_data_governance

**DBGraph** (VNPT Data Governance SDK) helps you explore and find relevant
data assets in a large, complex relational database. It introspects a schema,
builds a navigable graph of tables/columns, classifies PII, flags quality
issues, infers relationships, optionally enriches it with LLM-generated
descriptions, and lets you search/render/traverse that graph.

This package is a standalone SDK extracted from the internal `dbgraph`
project, packaged for reuse across teams/services (originally built by
minhdenthedev, packaged as an SDK by hainamnguyen192).

## Install

Core install (graph building/traversal/interactive HTML rendering; no network
DB drivers or LLM client):

```bash
pip install vnpt_data_governance
```

Pick the extras you actually need — each one only pulls in the dependency for
that piece:

```bash
pip install "vnpt_data_governance[postgres]"   # connect to postgres://... / postgresql://...
pip install "vnpt_data_governance[mysql]"      # connect to mysql://...
pip install "vnpt_data_governance[trino]"      # connect to trino://...
pip install "vnpt_data_governance[openai]"     # OAICompatibleLLM
pip install "vnpt_data_governance[search]"     # BM25SearchEngine
pip install "vnpt_data_governance[all]"        # everything above
```

SQLite is supported out of the box (Python's built-in `sqlite3`), no extra
needed.

## Quick start

The one-call entry point — introspects the database and returns a fully
enriched graph (schema, statistics, PII classification, quality issues,
normalization hints, inferred soft-FK/lineage relationships). Point it at a
sqlite file (or a `postgres://`/`mysql://`/`trino://` DSN) and it does the
rest, including writing a JSON dump, an interactive HTML diagram, and a
portable SQL export next to the source — each into its own subfolder, so the
3 output types don't pile up mixed together next to your source file(s):

```python
import vnpt_data_governance

graph = vnpt_data_governance.analyze("data/northwind.db")
# -> data/json/northwind.json, data/html/northwind.html, data/sql/northwind.sql
#    written automatically (subfolders created if missing)
```

Open `data/html/northwind.html` directly in a browser (no server needed) —
click any table/column/domain node or any relationship edge to see its full
detail (description, dtype, stats, PII classification, FK/soft-FK, lineage,
...), drag/zoom the diagram, and search assets by name. Pass
`open_browser=True` to have `analyze()` open it there for you as soon as it's
written (the CLI does this by default - see `--no-open` below).

`data/sql/northwind.sql` is a standalone script (`CREATE TABLE IF NOT EXISTS`
+ `INSERT INTO`, see `SQLGraphWriter`) that loads the same graph back into
the user's own database as two tables (`dbgraph_assets`, `dbgraph_links`) so
it can be queried/joined with SQL there — run it with `sqlite3 mydb.db <
data/sql/northwind.sql`, `psql`, or `mysql` as appropriate.

For a network database, pass its DSN instead of a file path — same one-liner.
There's no source *file* to write next to here, so output instead lands
under a `data/` folder relative to the current directory, named after the
database/catalog (`data/json/mydb.json`, `data/html/catalog-schema.html`, ...):

```python
graph = vnpt_data_governance.analyze("postgres://user:pass@host:5432/mydb")
graph = vnpt_data_governance.analyze("trino://user:pass@host:443/catalog/schema")
```

Pass an explicit `save_json=`/`render_html=`/`save_sql=` path to control
where those land (or `False` to skip one), and `db_schema=` when a
`trino://`/`postgres://` DSN doesn't carry a schema segment:

```python
graph = vnpt_data_governance.analyze(
    "trino://user:pass@host:443/catalog",  # no /schema in the DSN
    db_schema="app",
    save_json="out/graph.json",
    render_html=False,
    save_sql=False,
)
```

### CLI reference

`python -m vnpt_data_governance` runs the same one-call `analyze()` from the
terminal, no code needed — pass any source `analyze()` accepts and it does the
rest, printing a `[i/n] table (m columns)` / `  [j/m] table.column` progress
line to stderr as it goes (useful on a database profiled over the network,
where a single wide table can take minutes), then opens the rendered HTML
diagram in your default browser:

```bash
python -m vnpt_data_governance data/northwind.db
python -m vnpt_data_governance "postgres://user:pass@host:5432/mydb"
python -m vnpt_data_governance "trino://user:pass@host:443/catalog/schema"
```

Flags: `--db-schema SCHEMA` (for a `trino://`/`postgres://` DSN without one),
`--no-json` / `--no-html` / `--no-sql` (skip that output), `--no-open` (don't
open the HTML diagram in a browser), `--quiet` (don't print progress). On
success it prints `assets: N  links: M` and exits `0`. Any failure (DB
connection error, missing sqlite file, unrecognized DSN scheme, ...) is
caught and printed as a single `Error: ...` line on stderr with exit code `1`
— no Python traceback reaches the end user. Exit code `2` means a CLI usage
error (missing `<source>`), caught by `argparse` before `analyze()` ever
runs.

### Full control: pass a gateway instead of a path

For anything the DSN auto-detection can't cover (a custom SQLAlchemy engine
with connection-pool tuning, an engine built from a dialect this package
doesn't special-case, ...), pass a ready-made `SQLDataGateway` instead of a
string — `analyze()` then behaves exactly like before: no JSON/HTML/SQL
unless you ask for them. `SQLDataGateway` wraps any SQLAlchemy `Engine`
(built via `create_engine(...)` yourself, or via its own `from_url(...)`
convenience constructor), so it works with every dialect SQLAlchemy has a
driver for — sqlite/postgres/mysql/trino out of the box, anything else by
just installing that dialect's driver package:

```python
from pathlib import Path
from vnpt_data_governance import OAICompatibleLLM

graph = vnpt_data_governance.analyze(
    vnpt_data_governance.SQLDataGateway.from_url("sqlite:///data/northwind.db"),
    llm=OAICompatibleLLM(model=..., base_url=..., api_key=...),  # requires the `openai` extra
    save_json=Path("data/northwind-graph.json"),
    search_index_dir=Path("data/northwind-index"),  # requires the `search` extra
)

# load a previously saved graph back later
graph = vnpt_data_governance.JSONGraphLoader(json_path=Path("data/northwind-graph.json")).load()

# ...and re-render its interactive HTML straight from that JSON, without rebuilding the graph
vnpt_data_governance.GraphRenderer.render_json_file(
    "data/northwind-graph.json", "data/northwind-graph.html"
)

# search it (requires the `search` extra)
search_engine = vnpt_data_governance.BM25SearchEngine(Path("data/northwind-index"))
asset_ids = search_engine.search("restricted PII columns in the Sales domain")
```

### Advanced: composing the pipeline yourself

`analyze()` is a thin wrapper around individually usable pieces — reach for
these instead if you need custom prompts, want to skip/reorder steps, or want
finer control than `analyze()`'s options give you:

```python
from pathlib import Path

from vnpt_data_governance import (
    JSONGraphWriter,
    RGraphBuilder,
    SQLDataGateway,
    SoftForeignKeyDetector,
    LineageDetector,
)

gateway = SQLDataGateway.from_url("sqlite:///data/northwind.db")
graph_builder = RGraphBuilder(gateway)
graph = graph_builder.build_graph()
graph = SoftForeignKeyDetector(gateway).detect(graph)
graph = LineageDetector(gateway).detect(graph)

# optional: generate semantic descriptions for assets via an LLM, with your
# own prompts
# from vnpt_data_governance import GraphDescriptorV1, OAICompatibleLLM
#
# graph_descriptor = GraphDescriptorV1(
#     llm=OAICompatibleLLM(model=..., base_url=..., api_key=...),
#     system_prompt=..., formating_prompt=..., target_prompt=...,
# )
# graph = graph_descriptor.rfill_semantic_aspects(graph)

JSONGraphWriter(json_path=Path("data/northwind-graph.json"), indent=2).write(graph)
```

For visualization purposes, here is a graph saved as JSON:

```json
{
  "assets": [
    {
      "asset_id": "8ab5a624-0596-497e-a0ee-3996d95dbe63",
      "name": "Categories",
      "type": "table",
      "aspects": {
        "schema_properties": { "name": "Categories_table_schema", "pks": ["CategoryID"], "indices": {} },
        "statistical_properties": { "name": "Categories_table_stats", "num_columns": 4, "num_rows": 8 },
        "semantic_properties": {
          "name": "Categories_semantic",
          "description": "Stores product category definitions and metadata, serving as a lookup table for classifying products in the inventory system.",
          "keywords": ["categories", "product classification", "category definitions", "inventory groups", "product types"]
        }
      }
    }
  ],
  "links": [
    {
      "link_id": "db6bea93-a02c-4426-a2db-449e4a7bba8f",
      "name": "Categories_CategoryID",
      "type": "contain",
      "source_id": "8ab5a624-0596-497e-a0ee-3996d95dbe63",
      "destination_id": "04c20046-2808-4021-bbf1-99876e0eea6e",
      "aspects": {}
    }
  ]
}
```

## Use cases

![Use cases of DBGraph](https://raw.githubusercontent.com/hainamnguyen192/dbgraph-sdk/main/diagrams/usecase.png)

- **Manipulating database schema** — build the schema graph, store it, and use it to traverse the database, find join paths, get referenced tables, ...
- **Profiling database** — the `Aspect` concept represents different kinds of properties attached to a data asset (schema, statistics, semantics, ...).
- **Render graph** — output a schema graph as a self-contained interactive HTML diagram (`GraphRenderer`) or Markdown/text to feed as LLM context.
- **Export back into a database** — write the graph as a portable `.sql` script (`SQLGraphWriter`) the user can load straight into their own database.
- **LLM assistance** — use an LLM to generate data assets' descriptions/keywords, and as input for downstream SQL generation.
- **Search for data assets** — search assets via BM25 indexing/retrieval, over a document built from every enriched aspect (description/keywords, PII classification, quality issues, domain), not just the description.

## Architecture

![Class diagram of DBGraph](https://raw.githubusercontent.com/hainamnguyen192/dbgraph-sdk/main/diagrams/entity.png)

DBGraph is designed to be easy to extend:

1. **Core classes** (_entities_) hold the shared business logic of database graphs (traversal, neighborhoods, ...) and core operations
   (building graphs, profiling databases, ...). The prefix `R...` stands for "Relational" (the only paradigm currently supported);
   `D...`, `V...`, `G...` are reserved for Document/Vector/Graph paradigms.
2. **Interfaces** (_extensions_) mark the parts of the system meant to be pluggable:
   - `RGraphBuilder` works against any RDBMS via the `RDataGateway` abstraction — `SQLDataGateway` is the bundled implementation, built on `sqlalchemy.inspect()`/SQLAlchemy Core so it works with SQLite/PostgreSQL/MySQL/Trino out of the box and with any other dialect SQLAlchemy has a driver for, no new gateway class needed.
   - `LLM` abstracts the model provider — `OAICompatibleLLM` is the bundled implementation (`[openai]` extra); bring your own by
     implementing `LLM.generate`/`agenerate`.
   - `GraphWriter`/`GraphLoader` abstract graph persistence — `JSONGraphWriter`/`JSONGraphLoader` are the bundled implementation.
   - `GraphRenderer` renders any `DatabaseGraph` (including one just loaded via `JSONGraphLoader`) as a self-contained, clickable interactive HTML schema diagram; also runnable as a script: `python -m vnpt_data_governance.io.graph_renderer graph.json graph.html`.
   - `SQLGraphWriter` (a `GraphWriter`) writes the graph as a portable `.sql` script the user can load back into their own database.
   - `SearchEngine` abstracts indexing/retrieval — `BM25SearchEngine` is the bundled implementation (`[search]` extra).

## Development

```bash
uv sync --group dev --all-extras
uv run pytest
uv run pylint vnpt_data_governance
uv run mypy vnpt_data_governance
uv run flake8 vnpt_data_governance
```

Tests that talk to Postgres/MySQL/Trino/OpenAI need real credentials (see
`tests/`) and are skipped/fail without them; the SQLite, entity and
loader/writer tests run standalone.

## License

[MIT](LICENSE)
