DataSentry logo DataSentry
Features Demo Quick start Docs

DataSentry

Evidence-driven, local-first AI copilot for data quality. Detect · Explain · Validate · Repair — with statistical evidence, AI assistance, and human approval. Your data never leaves your machine.

version 0.53.0 python 3.12 license Apache-2.0 tests 1229 passed coverage 95% detectors 39 ADR 112 MCP tools 20
Get started See a live report

What it does

One scan produces a six-dimension quality score; every issue carries a statistical evidence chain — samples, ratios, confidence. AI proposes; humans approve.

39 evidence-driven detectors

Missingness, dates, encodings, cross-field rules, cross-table foreign keys, exact + fuzzy duplicates, Isolation Forest / LOF outliers.

Six-dimension scoring

Completeness, validity, uniqueness, consistency, integrity, timeliness — explainable weights, per-dimension contributions.

Human-in-the-loop repair

Propose → preview (rules re-run) → apply (fingerprinted copy) → rollback. AI suggests; you decide. Single or batch — proposals are read-only, applies never overwrite the source file, every apply is undoable via its snapshot.

Drift engine

Compare historical scans: schema, row-count, score and issue-distribution drift between versions.

CI quality gates

scan --fail-on blocks releases; reports as JSON / Markdown / HTML / JUnit / SARIF.

Cron scheduling

Persistent job queue in SQLite: cron expressions, manual triggers, run history, webhooks, quality gates and change-aware skip (no re-scan when content is unchanged).

Distributed execution

Any instance runs as a worker (datasentry worker); a pool with round-robin routing, failover, cooldown and health checks, plus parallel dispatch (DATASENTRY_MAX_WORKERS).

Safe LLM assistance

PII redacted before every prompt, encrypted vault with key rotation, calls audited, rules approved after preflight simulation. Runs on local Ollama.

Four interfaces

CLI, REST API, server-rendered Web UI with cross-scan trends, and an MCP stdio server (24 tools) for LLM agents.

Local-first & fast

DuckDB execution engine, 1e6 rows in ~10s. No cloud, no telemetry, no vendor lock-in.

Open & extensible

Detector plugin API (plugins/ auto-load with SHA-256 integrity locks), contract exports to Pandera / Great Expectations, 112 ADRs of design rationale.

Live demo report

200 rows of order data with 15 injected quality issues — scanned and reported in one command. This is the actual report export --as html output, no custom rendering.

Quick start

Install from PyPI and scan your first file in under a minute.

# install
$ pip install datasentry-ai

# scan → fuse → score → persist, one step
$ datasentry scan orders.csv
{
  "scan_run_id": "scan_3426008c0380",
  "status": "completed",
  "row_count": 200,
  "issues_count": { ... }
}

# issues, score, repair, drift
$ datasentry issues list
$ datasentry score scan_3426008c0380
$ datasentry repair propose <issue_id> --file orders.csv
$ datasentry drift latest orders

# batch repair loop (propose → apply copies → rollback; partial failures exit 4)
$ datasentry repair propose-batch <scan_run_id> --file orders.csv --all
$ datasentry repair apply-batch <scan_run_id> --file orders.csv --all
$ datasentry repair rollback-batch run_1,run_2
$ datasentry repair list --run <scan_run_id>   # audit
$ datasentry repair verify <repair_run_id>   # re-scan repaired copy; exit 1 on regression

# scheduled jobs + distributed workers
$ datasentry job create nightly --cron "0 9 * * *" orders.csv
$ datasentry worker --port 8001          # execution node
$ DATASENTRY_WORKERS="http://localhost:8001:secret" datasentry-server

# Web UI + REST API
$ datasentry-server
→ http://localhost:8000/ui/  (scans, issues, repair workbench, trends)
  • scan — 39 detectors, fusion, six-dimension score
  • issues — filter by severity / dimension
  • score — 0–100 with explainable weights
  • repair — propose → preview → apply → rollback
  • drift — compare historical scans
  • rules — natural language → rule candidates + approval
  • job — cron scheduling, triggers, run history
  • worker — run as a remote execution node
  • mcp — 24 tools for LLM agents over stdio
  • detectors — registry incl. plugins

MCP tools (24)

    scan_file — scan a dataset file (detect issues) list_issues — list issues for a scan run (filter by severity) quality_score — 0-100 explainable quality score for a run detectors_list — list registered detectors (incl. plugins) drift_latest — drift report for the latest version of a dataset drift_compare — compare two historical scans trends_list — quality trends across scans profiles_get — column profiles for a run comparison_build — build a regression report between two runs contract_validate — validate a data contract YAML repair_propose_batch — propose repairs for all issues of a run repair_apply_batch — apply repairs (writes repaired copies, never the source) repair_rollback_batch — roll back applied repairs by run id repair_verify — re-scan a repaired copy; report fixed/persistent/new pii_sessions — list PII vault sessions pii_restore — restore a PII-redacted file pii_delete_session — delete a PII session pii_rotate_key — rotate the PII vault key pii_purge_sessions — purge all PII sessions job_create — create a scheduled job job_update — update a scheduled job job_trigger — trigger a job now job_remove — remove a scheduled job jobs_list — list scheduled jobs and run history

Documentation

README

Full feature tour, architecture diagram, development guide.

Open on GitHub →

Development notes

Per-step implementation notes, conventions, testing practices.

docs/DEVELOPMENT.md →

ADR journal

111 architecture decision records explaining every design choice.

docs/00-ADR.md →