Metadata-Version: 2.4
Name: datalore-cli
Version: 0.4.1
Summary: Deterministic local data analysis CLI for coding agents.
Author: Mihajlo Micic
License-Expression: MIT
Project-URL: Repository, https://github.com/micic-mihajlo/datalore-cli
Project-URL: Issues, https://github.com/micic-mihajlo/datalore-cli/issues
Keywords: cli,data-analysis,agents,codex,claude-code,pandas
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas
Requires-Dist: matplotlib
Requires-Dist: seaborn
Requires-Dist: scikit-learn
Requires-Dist: numpy
Dynamic: license-file

# Datalore

Datalore is a deterministic local data-analysis CLI designed to amplify coding agents.

The core idea is simple: when Codex, Claude Code, or a human operator needs a reliable answer about a local dataset, they should call a stable binary with explicit arguments and parse structured output instead of improvising pandas, sklearn, and matplotlib every time.

## What It Does

- Inspect local CSV, Excel, and JSON datasets
- Profile columns, dtypes, missing values, and duplicates
- Generate structured descriptive statistics
- Generate structured correlation reports
- Run deterministic linear regression
- Apply deterministic cleaning and export operations
- Compare two datasets structurally and at the row level
- Generate reproducible plot artifacts
- Diagnose runtime/install issues with a doctor command
- Emit either human-readable text or machine-readable JSON
- Optionally persist the full JSON envelope to a report file
- Keep outputs under `artifacts/` by default

## Install

Current install flow, from any machine where you want to analyze local data:

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install 'git+https://github.com/micic-mihajlo/datalore-cli.git'
```

After that, the public interface is just:

```bash
datalore --help
```

The repo-local `./datalore` wrapper is only a development convenience for contributors working inside this repository.

If you are developing Datalore itself, install it in editable mode from the repo root:

```bash
pip install -e .
```

## Use In Your Own Project

Move into the folder that contains your data:

```bash
cd /path/to/my/data
datalore init
```

That creates:
- `AGENTS.md`
- `CLAUDE.md`
- `./datalore`
- `artifacts/`
- a `.gitignore` entry for `artifacts/`

After that, both you and your coding agent can work from that folder directly:

```bash
./datalore files . --format json
./datalore profile sales.csv --format json
./datalore clean sales.csv --rename sales=>revenue --output artifacts/datasets/sales_clean.csv --format json
```

Why the wrapper matters:
- humans can keep using `datalore`
- agents are usually safer using `./datalore` because it binds the project to the exact Python interpreter that installed Datalore

## CLI Usage

Inspect a dataset:

```bash
datalore inspect data.csv
```

Profile a dataset in JSON mode for agent consumption:

```bash
datalore profile data.csv --format json
```

Run a simple regression:

```bash
datalore regression data.csv --target last --format json
```

Constrain the predictors:

```bash
datalore regression data.csv --target revenue --predictors spend,visits,leads --format json
```

Create a plot with a stable output path:

```bash
datalore plot data.csv --kind histogram --x revenue --output artifacts/plots/revenue_hist.png --format json
```

Generate a correlation heatmap:

```bash
datalore plot data.csv --kind correlation-heatmap --format json
```

Compare two datasets:

```bash
datalore compare before.csv after.csv --format json
datalore compare before.csv after.csv --key-columns id --format json
```

Generate structured summary statistics:

```bash
./datalore summary data.csv --format json
```

Generate a structured correlation report:

```bash
./datalore correlate data.csv --target revenue --min-abs-correlation 0.3 --format json
```

Clean and export a dataset deterministically:

```bash
datalore clean data.csv --fill-numeric median --drop-duplicates --output artifacts/datasets/data_clean.csv --format json
```

Apply deterministic transforms without falling back to ad hoc pandas:

```bash
datalore clean data.csv \
  --rename sales=>revenue \
  --derive sales_per_customer=>revenue\|/\|customers \
  --filter 'customers>=24' \
  --limit 100 \
  --output artifacts/datasets/data_prepared.csv \
  --format json
```

Write the full structured result to disk for downstream tooling:

```bash
datalore profile data.csv --format json --report-file artifacts/reports/profile.json
```

Find local datasets, including gitignored files in the repo:

```bash
datalore files . --format json
```

Inspect the runtime/install state:

```bash
datalore doctor --format json
```

Initialize a non-repo project for Codex and Claude Code:

```bash
datalore init
datalore init --agent codex
```

## Command Surface

`datalore inspect`
- Quick shape, dtype, missing-value, duplicate, and preview summary.

`datalore profile`
- Adds per-column detail on top of `inspect`.

`datalore regression`
- Runs a deterministic linear regression using numeric columns only.

`datalore summary`
- Emits machine-readable descriptive statistics for numeric and categorical columns.

`datalore correlate`
- Emits a machine-readable correlation matrix, strongest pairs, and optional target-column ranking.

`datalore clean`
- Applies deterministic cleaning and export operations such as filter, derive, rename, select, fill, drop NA, dedupe, sort, limit, and standardize.
- The transform pipeline runs in a fixed order: `rename -> derive -> filter -> select -> fill -> drop NA -> dedupe -> standardize -> sort -> limit`.

`datalore compare`
- Compares two datasets for shape, schema, dtype, missing-value, duplicate, and row-level changes. With `--key-columns`, it can report changed rows keyed by business identifiers.

`datalore files`
- Discovers local dataset files by walking the workspace, including gitignored inputs that `rg --files` would skip.

`datalore plot`
- Generates histogram, scatter, line, bar, or correlation heatmap artifacts.

`datalore doctor`
- Reports interpreter, artifact-root, and dependency status for install/runtime debugging.

`datalore init`
- Scaffolds `AGENTS.md`, `CLAUDE.md`, a local `./datalore` wrapper, `artifacts/`, and an `artifacts/` ignore rule in the current project.

## Reliability

- JSON output uses a stable envelope with `schema_version`, `command`, `result`, and `error`
- every command supports `--report-file` for downstream automation
- `datalore` works from any folder after installation
- `./datalore` prefers the repo venv automatically when developing inside this repo
- `datalore doctor` gives a stable diagnostics entrypoint when the runtime is suspect
- startup no longer crashes on `--help` if scientific dependencies are missing
- CI runs editable install, distribution builds, and the CLI test suite on multiple Python versions via `.github/workflows/ci.yml`

## Agent-Friendly Conventions

In a normal user project, prefer:

```bash
./datalore <command> ... --format json
```

Why:
- JSON output is stable and easy to parse
- exit codes are predictable
- artifact paths are explicit
- the analysis method is fixed rather than improvised
- `--report-file` gives you a stable on-disk result for chained workflows
- `datalore files` avoids the common failure mode where the dataset is gitignored and invisible to `rg --files`
- `datalore init` gives Codex and Claude Code repo-local instructions plus a reliable local wrapper in the user's own data project

Inside the Datalore repository itself, contributors can still prefer:

```bash
./datalore <command> ... --format json
```

Repo-native instructions are included for both ecosystems:
- `AGENTS.md` for Codex
- `CLAUDE.md` for Claude Code
- `.agents/skills/` for Codex skills
- `.claude/skills/` for Claude Code skills

These instructions tell agents when to use Datalore instead of writing ad hoc analysis code.

## Artifacts

Generated outputs go under `artifacts/` by default:

- `artifacts/plots/`
- `artifacts/datasets/`
- `artifacts/reports/`
- `artifacts/scripts/`

Override the root with:

```bash
export DATALORE_ARTIFACT_DIR=/path/to/artifacts
```

## Publishing

The repository is set up for PyPI Trusted Publishing through GitHub Actions.

Release flow:

```bash
git tag v0.4.1
git push origin v0.4.1
gh release create v0.4.1 --generate-notes
```

Before the first release, create or register the `datalore-cli` project on PyPI and add a Trusted Publisher that matches:

- owner: `micic-mihajlo`
- repository: `datalore-cli`
- workflow: `.github/workflows/publish.yml`
- environment: `pypi`

If the project does not exist yet on PyPI, use PyPI's pending publisher flow for the first release. After that, publishing happens from GitHub Actions without local API tokens.

## Testing

Run the CLI test suite:

```bash
.venv/bin/python -m unittest discover -s tests -v
```

Smoke test the fixture dataset manually:

```bash
./datalore files tests --format json
./datalore inspect tests/fixtures/mini_dataset.csv --format json
./datalore summary tests/fixtures/mini_dataset.csv --format json
./datalore correlate tests/fixtures/mini_dataset.csv --target sales --format json
./datalore regression tests/fixtures/mini_dataset.csv --target sales --predictors marketing,customers --format json
./datalore clean tests/fixtures/mini_dataset.csv --rename sales=>revenue --derive sales_per_customer=>revenue\|/\|customers --filter 'customers>=24' --limit 3 --format json
./datalore clean tests/fixtures/mini_dataset_changed.csv --fill-numeric median --output artifacts/datasets/mini_clean.csv --format json
./datalore compare tests/fixtures/mini_dataset.csv tests/fixtures/mini_dataset_changed.csv --key-columns month --format json
./datalore doctor --format json
```

Smoke test the installed package from outside the repo:

```bash
mkdir -p /tmp/datalore-demo
cp tests/fixtures/mini_dataset.csv /tmp/datalore-demo/
cd /tmp/datalore-demo
datalore init
./datalore files . --format json
./datalore profile mini_dataset.csv --format json
```
