Metadata-Version: 2.4
Name: datamine-lib
Version: 0.5.0
Summary: Shared data-mining library for the causal research suite — profile, associate, KPI, morphemes, insights, auto feature engineering
Author: Ephraim Hallford
License: MIT
Project-URL: Homepage, https://github.com/ehallford11714/datamine-lib
Project-URL: Documentation, https://github.com/ehallford11714/datamine-lib/blob/v0.5.0/docs/ARCHITECTURE.md
Keywords: data-mining,kpi,eda,causal,morpheme,tabular,vision,feature-engineering
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Provides-Extra: vision
Requires-Dist: opencv-python-headless>=4.8; extra == "vision"
Provides-Extra: sql
Requires-Dist: sqlalchemy>=2.0; extra == "sql"
Provides-Extra: sklearn
Requires-Dist: scikit-learn>=1.3; extra == "sklearn"
Provides-Extra: scipy
Requires-Dist: scipy>=1.11; extra == "scipy"
Requires-Dist: statsmodels>=0.14; extra == "scipy"
Provides-Extra: nltk
Requires-Dist: nltk>=3.8; extra == "nltk"
Provides-Extra: gensim
Requires-Dist: gensim>=4.3; extra == "gensim"
Provides-Extra: spacy
Requires-Dist: spacy>=3.7; extra == "spacy"
Provides-Extra: transformers
Requires-Dist: transformers>=4.36; extra == "transformers"
Provides-Extra: fe
Requires-Dist: datamine-lib[gensim,nltk,scipy,sklearn,spacy,transformers]; extra == "fe"
Provides-Extra: memory
Requires-Dist: mem0>=0.1; extra == "memory"
Provides-Extra: vector
Requires-Dist: chromadb>=0.4; extra == "vector"
Requires-Dist: faiss-cpu>=1.7; extra == "vector"
Provides-Extra: autocausal
Provides-Extra: causaliv
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Provides-Extra: all
Requires-Dist: datamine-lib[dev,fe,memory,sql,vector,vision]; extra == "all"
Dynamic: license-file

# DataMine Lib

**Shared data-mining library** for the causal research suite — one facade that profiles columns, mines associations, extracts KPIs (tabular + optional vision), exports morpheme catalogs, runs auto feature engineering (propose → build → test), and emits markdown/JSON reports.

Package: `datamine` · PyPI-style name: **`datamine-lib`** · version **0.5.0**

```python
from datamine import DataMiner, MineReport, AutoFeatureEngine, auto_features, library_help

miner = DataMiner.from_csv("x.csv", prefer_autocausal=False)  # also from_df, from_sqlalchemy, from_public_join
report = miner.profile().associate().kpi().morphemes().insights().report()
# or: report = miner.run()

print(report.to_markdown())
ac = miner.to_autocausal()   # optional, if AutoCausalLib installed
civ = miner.to_causaliv()    # CausalIV-shaped KPI catalog dict

# Auto feature engineering (miners → SLM guide → coding agent → test)
miner.auto_features(
    target="conversion",
    research_problem="What drives conversion?",
    use_slm=False,
    use_causal_adapter=True,  # optional causal-heuristic seeds (≠ identification)
    coding_llm="builders",    # offline; or "auto" (cloud → local → builders)
    coding_harness="sys",     # offline; or "auto" (openclaw → hermes → sys)
)
print(miner.report().auto_features["kept"])

# or convenience → AutoFeatureReport
from datamine import auto_features
fe = auto_features(
    df, target="conversion", research_problem="...", use_slm=False,
    use_causal_adapter=True, coding_llm="builders", coding_harness="sys",
    prefer_autocausal=False,
)
print(fe.to_markdown())

print(library_help(module="features"))
```

> Exploratory FE / association testing ≠ causal identification. Significance vs a target is associational evidence only.
> Causal heuristics (when enabled) **seed** proposals — they are not causal identification.

See [docs/auto_features.md](docs/auto_features.md) for the full pipeline and call path.

## Install

```bash
cd research/DataMineLib
pip install -e ".[dev]"

# optional extras
pip install -e ".[vision]"   # opencv
pip install -e ".[sql]"      # sqlalchemy
pip install -e ".[fe]"       # sklearn, scipy, statsmodels, nltk, gensim, spaCy, transformers
pip install -e ../AutoCausalLib   # preferred tabular backend
pip install -e ../CausalIVSuite   # preferred KPI backend
```

## Pipeline overview

```
Classic mine
  profile → associate → kpi → morphemes → insights → MineReport

Auto features
  miners (+ optional causal heuristics / morpheme+vision seeds)
       → optional SLM guide (default off)
       → coding harness (openclaw|hermes|sys) + coding LLM
       → significance / keep-reject (+ optional refine-on-failure)
       → adaptive multi-round on augmented work frame → AutoFeatureReport
```

## CLI

```bash
python -m datamine --help
python -m datamine help --all
python -m datamine help --module features
python -m datamine help --api
python -m datamine help --cli

python -m datamine mine --csv path/to/data.csv
python -m datamine mine --csv path/to/data.csv --json -o report.json
python -m datamine mine --csv data.csv --with-features --target conversion
python -m datamine guides
python -m datamine adapters

# Auto features: propose → build → significance-test (offline-safe flags)
python -m datamine features --csv path/to/data.csv --target conversion \
  --problem "What drives conversion?" --causal-heuristics \
  --coding-llm builders --coding-harness sys --show-backends \
  --feature-frame-out feats.csv --max-refine 1
python -m datamine features --parquet data.parquet --target y
python -m datamine pipeline --csv data.csv --target y --coding-llm builders
python -m datamine features --csv data.csv --target y --json -o fe.json
```

## Examples

Runnable offline demos under [`examples/`](examples/README.md):

| Script | Pipeline |
|---|---|
| `examples/01_classic_mine.py` | Fluent classic mine |
| `examples/02_auto_features.py` | Full auto-features (builders + sys) |
| `examples/03_causal_heuristics.py` | + `use_causal_adapter=True` |
| `examples/04_coding_harness.py` | `coding_llm="auto"` / `coding_harness="auto"` |
| `examples/run_cli_examples.ps1` | CLI walkthrough (Windows) |

```bash
python examples/01_classic_mine.py
python examples/02_auto_features.py
python examples/04_coding_harness.py --force-builders
```

## Auto feature engineering

Pipeline: mine associations / association rules / KPIs → **advanced statistical miners** (+ optional **causal heuristics** adapter) seed proposals → optional SLM re-rank → **coding agent LLM** (OpenAI / Claude / Grok / local / builders) → whitelist builders → significance + distribution tests → iterate.

Coding LLM (`coding_llm="auto"`): cloud if API key present → best on-device → deterministic builders-only. Keys via env (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `XAI_API_KEY` / `GROK_API_KEY`, `DATAMINE_LLM_*`) — never hardcode.

Coding harness (`coding_harness="auto"`): OpenClaw → Hermes → built-in `sys`. Soft-degrades when externals are missing. See [docs/auto_features.md](docs/auto_features.md) and [docs/coding_harness.md](docs/coding_harness.md).

**Advanced miners** (soft deps; sources on `FeatureProposal`):

| source | what it does |
|---|---|
| `mi` | Mutual information / dependence ranks (sklearn) |
| `anova` | ANOVA / Kruskal for categorical↔numeric |
| `cramers_v` | Chi-square / Cramér's V for categorical–categorical |
| `partial_corr` | Partial / residualized associations |
| `dist_transform` | Skew/kurtosis → log / rank / bin |
| `outlier` | Extreme-value indicator flags |
| `stratified_lift` | Level lift vs base rate |
| `vif` | Multicollinearity notes → ratio proposals |
| `causal_heuristic` / `autocausal_adapter` | Optional causal-role heuristics (treatment/confounder/…) — seeds only, ≠ identification |

Also: correlation pairs, association rules (`equals_value` for target-linked rules), research-problem term overlap, NLP term features. Enable causal seeds with `use_causal_adapter=True` or `--causal-heuristics`.

## Environment variables

| Variable | Role |
|---|---|
| `OPENAI_API_KEY` | OpenAI coding LLM |
| `ANTHROPIC_API_KEY` | Claude coding LLM |
| `XAI_API_KEY` / `GROK_API_KEY` | Grok coding LLM |
| `DATAMINE_LLM_API_KEY` + `DATAMINE_LLM_BASE_URL` | OpenAI-compatible endpoint |
| `DATAMINE_LLM_MODEL` | Model id for compatible endpoint |
| `DATAMINE_CODING_LLM` | Default provider (`auto` / `builders` / …) |
| `DATAMINE_CODING_HARNESS` | Default harness (`auto` / `sys` / …) |
| `DATAMINE_OPENCLAW_BIN` / `DATAMINE_HERMES_BIN` | External harness CLI paths |
| `DATAMINE_OPENCLAW_CMD` / `DATAMINE_HERMES_CMD` | Optional command templates |
| `DATAMINE_LOCAL_LLM_MODEL` | Local on-device model id |

Full table: [docs/coding_harness.md](docs/coding_harness.md#env-vars) · [docs/auto_features.md](docs/auto_features.md#environment-variables).

## Guides

`datamine.guides` detects installed suite packages and recommends which miner/backend to use (AutoCausal, CausalIV AutoKPI, VisionKPI, NextFrameSeq image_mine, MorphemeStudio, EquityIV, FactorIV, CausalSearch).

## Product adapters

Thin optional adapters live in each product as `<pkg>.datamine_adapter`. Soft-import `datamine` — products keep working without it. See [docs/ADAPTERS.md](docs/ADAPTERS.md).

## Docs

- [auto_features.md](docs/auto_features.md) — propose → SLM guide → coding harness/LLM → test + causal heuristics + env vars
- [coding_harness.md](docs/coding_harness.md) — OpenClaw / Hermes / sys harness
- [ARCHITECTURE.md](docs/ARCHITECTURE.md)
- [ADAPTERS.md](docs/ADAPTERS.md)
- [examples/README.md](examples/README.md) — examples index

## Help

```bash
python -m datamine --help
python -m datamine help --all|--module features|--api|--cli
```

```python
from datamine import library_help
print(library_help(all=True))
print(library_help(api=True))
```

Catalog covers: `DataMiner`, `auto_features`, `AutoFeatureEngine`, `CodingAgent`, harness, causal heuristics.

## Tests

```bash
pip install -e ".[dev]"
pytest -q
```

## CausalBridge

Registered as `datamine` in the CausalBridge catalog. Workflow:

```bash
python -m causalbridge workflow datamine_all --dry-run
```

## GitHub

https://github.com/ehallford11714/datamine-lib
