Metadata-Version: 2.4
Name: dataforge-sba
Version: 0.1.0
Summary: A local-first data science learning toolkit and notebook generator
Author: Selim Ben Ali
License-Expression: MIT
Project-URL: Homepage, https://github.com/SelimBenAli/dataforge
Project-URL: Repository, https://github.com/SelimBenAli/dataforge
Project-URL: Issues, https://github.com/SelimBenAli/dataforge/issues
Keywords: data-science,machine-learning,education,notebooks
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Framework :: Flask
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Education
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=2.0
Requires-Dist: numpy>=1.24
Requires-Dist: scikit-learn>=1.3
Requires-Dist: nbformat>=5.9
Provides-Extra: dashboard
Requires-Dist: Flask>=3.0; extra == "dashboard"
Requires-Dist: openpyxl>=3.1; extra == "dashboard"
Provides-Extra: visualization
Requires-Dist: matplotlib>=3.7; extra == "visualization"
Provides-Extra: excel
Requires-Dist: openpyxl>=3.1; extra == "excel"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Dynamic: license-file

# DataForge

DataForge is a local-first Python toolkit for inspecting datasets, learning core machine learning concepts, and generating editable Jupyter notebooks. Its Python library is the product; Flask is an optional local dashboard that calls the same library services.

## V1 features

- Load and inspect CSV and `.xlsx` datasets without changing the source file.
- Review schema, missing values, duplicates, descriptive statistics, constant columns, category frequencies, and numeric correlations.
- In the dashboard, preview and apply missing-value imputation or removal, duplicate removal, type conversion, categorical encoding, and IQR outlier detection. Transformations affect the session's in-memory dataset; the uploaded source file is not overwritten.
- Create selectable histograms, boxplots, scatterplots, correlation heatmaps, and category bar/frequency charts.
- Run Linear Regression, Logistic Regression, K-means, Decision Tree, and KNN using shared train/test and preprocessing pipelines.
- Generate structured, editable `.ipynb` notebooks from Python or the dashboard, including selected cleaning steps, EDA, features, targets, model parameters, and optionally the current Playground experiment.
- Explore actual K-means, regression, and classification experiments in the Playground, using reproducible built-in samples or a session's uploaded dataset. K-means includes cluster assignments, centroids, sizes, an elbow curve, and a silhouette score when valid. Linear Regression includes a fitted line, held-out actual-versus-predicted points, and evaluation metrics.
- Launch a local dashboard on an explicitly selected port.

The dashboard is local and has a 25 MB upload limit. Upload CSV or `.xlsx` files; use the **Remove uploaded data** button to delete the current session's uploaded file. A new upload replaces and removes that session's previous file. If the browser session is lost before removal, stop DataForge and delete files from `.dataforge/uploads` manually. Downloaded notebook copies are temporary and are removed after transfer.

Cleaning can be staged, previewed, applied, and reset to the uploaded original. Outlier detection only flags potential anomalies; it never removes them. Ordinal encoding requires the user to explicitly provide the real category order. Supervised-model preprocessing (including one-hot encoding and scaling) is fitted inside the training pipeline after the train/test split. The Notebook Builder tests the selected workflow before enabling download, and the generated notebook remains editable and must be run by the user in Jupyter.

The K-means Playground offers an elbow curve so learners can compare K values, plus silhouette where mathematically valid. Step-by-step centroid iteration animation is not included in V1. Playground chart features are limited to one numeric predictor for Linear Regression and two numeric features for the other algorithms; the visualized features are the features used for the Playground model.

## Install

Python 3.10 or newer is required. Install the core Python library and command-line tools with:

```bash
python -m pip install dataforge-sba
```

To use the optional local dashboard with Excel workbook support and notebook charts, install:

```bash
python -m pip install "dataforge-sba[dashboard,visualization]"
```

The distribution is named `dataforge-sba`; import it in Python as `dataforge_sba`. Both `dataforge-sba` and `dataforge` are installed as command names. Python 3.10–3.13 are declared supported. The core package requires pandas, NumPy, scikit-learn, and nbformat. Flask/openpyxl are in the `dashboard` extra; Matplotlib is in `visualization`.

For library and dashboard development:

```bash
python -m pip install -e ".[dev,dashboard,visualization]"
python -m pytest
```

The library itself does not start Flask or open a browser during import. Jupyter can be installed separately with `python -m pip install jupyter` to execute downloaded notebooks.

## Launch the dashboard

```bash
dataforge-sba dashboard
dataforge-sba dashboard --port 8080
dataforge dashboard --port 8080
dataforge-sba --help
dataforge-sba --version
dataforge-sba --creator
```

`--creator` prints `Selim Ben Ali | https://github.com/SelimBenAli/dataforge`. The default bind address is `127.0.0.1`; debug mode is off. If a port is already in use, DataForge reports it and exits; choose another with `--port`. Ports must be from 1 through 65535. Stop the dashboard with Ctrl+C. The dashboard requires the `dashboard` extra; if Flask is missing, the command prints the install instruction. The upload limit is 25 MB. Uploaded data stays in `.dataforge/uploads` under the directory where you launch the command, and each browser session refers to its own upload.

### Using the dashboard

1. Upload a CSV or `.xlsx` file and review the column types, missing values, duplicates, and summaries.
2. Use **Clean and prepare** to stage changes, preview them, then apply or reset them. The uploaded file itself is not changed.
3. Use **Explore with a chart** to inspect selected columns.
4. In **Notebook Builder**, choose chart types, algorithms, targets, optional features, and model settings. Click **Test selections** to check the choices before downloading.
5. In the **ML Playground**, choose sample data or the uploaded dataset, run an experiment, and optionally include its configuration and code in the notebook.
6. In Jupyter, open the downloaded `.ipynb`, update `DATA_PATH` if necessary, and run cells from top to bottom. The notebook is an editable starting point; it runs on your computer.

## Python API

```python
from dataforge_sba import Dataset
from dataforge_sba.cleaning import impute_missing, missing_summary

dataset = Dataset("data.csv")
print(dataset.shape)
print(dataset.schema())
report = dataset.explore()

cleaned = impute_missing(dataset.frame, {"age": "median", "city": "mode"})
print(missing_summary(cleaned))
```

To fit a model, pass a DataFrame with the target column named explicitly:

```python
from dataforge_sba.algorithms import run_algorithm

result = run_algorithm("linear_regression", dataset.frame, target="sales")
print(result["metrics"])
```

The returned model is a fitted scikit-learn pipeline. Preprocessing is fitted on training data only. Classification metrics are weighted averages; inspect the task and dataset before interpreting scores.

## Generate a notebook

```python
from dataforge_sba import Dataset
from dataforge_sba.notebooks import NotebookGenerator

dataset = Dataset("data.csv")
generator = NotebookGenerator(dataset)
generator.add_cleaning(drop_duplicates=True)
generator.add_visualizations()
generator.add_algorithm("kmeans", n_clusters=4)
generator.save("analysis.ipynb")
```

The generated notebook uses a portable dataset filename and a clear `DATA_PATH` variable. Move the original dataset beside the notebook or update `DATA_PATH` in its first code cell. Supervised experiments include a target and optional selected features; K-means does not ask for a target. The dashboard also offers algorithm-specific parameters, split fraction, seed, scaler choice, selected EDA charts, cleaning steps, and optional Playground export. Dashboard downloads are preflighted against the uploaded data and structurally validated; the notebook is not fully executed until you run its cells in Jupyter.

## K-means Playground from Python

```python
from dataforge_sba.playground import run_kmeans

experiment = run_kmeans(shape="moons", n_clusters=2, random_state=7)
print(experiment["silhouette"], experiment["cluster_sizes"])
```

Cluster labels are arbitrary identifiers, not rankings. Scaling affects distances, and silhouette scores are unavailable when the mathematical requirements are not met.

## Algorithm catalog

| Algorithm ID | Task | Current implementation |
|---|---|---|
| `linear_regression` | Numeric prediction | Train/test split; MAE, MSE, RMSE, R² |
| `logistic_regression` | Classification | Binary/multiclass; accuracy, weighted precision/recall/F1 |
| `kmeans` | Clustering | Numeric features, optional scaling, cluster sizes, silhouette score |
| `decision_tree` | Classification | Configurable depth and reproducible split |
| `knn` | Classification | Configurable neighbor count and scaled features |

Use `dataforge_sba.algorithms.list_algorithms()` to retrieve registry metadata. Extend the catalog through the central algorithm registry; UI labels are populated from its metadata endpoint.

## Privacy

DataForge runs locally and sends no dataset data to external services. It does not execute uploaded Python code or overwrite the source dataset. The browser dashboard uses first-party local requests only. Keep `.dataforge/` private if your data is sensitive, and remove temporary uploads when finished.

## Roadmap

Known V1 limitation: K-means provides an elbow curve and silhouette score but does not animate centroid updates step by step. Playground visualizations use one numeric feature for Linear Regression and two for the other algorithms. The dashboard does not persist experiment history. Further algorithms, PCA, cross-validation, and experiment persistence are possible future work.

## Troubleshooting

- **`dataforge-sba` is not recognized:** close and reopen the terminal after installation, check that Python's Scripts directory is on `PATH`, or run `python -m pip show dataforge-sba` to confirm which Python environment received the package.
- **Dashboard reports that Flask is missing:** install `python -m pip install "dataforge-sba[dashboard]"` in the same environment used to launch the command.
- **Excel workbook cannot be read:** install the dashboard extra (which includes `openpyxl`) or `python -m pip install openpyxl`.
- **Windows reports a NumPy DLL or application-control error:** use a fresh virtual environment and reinstall the package dependencies. If Windows security blocks a dependency DLL, resolve that Windows policy block before retrying; DataForge cannot bypass it.
- **Notebook cannot find the dataset:** put the original CSV/XLSX beside the notebook or edit `DATA_PATH` in the first code cell to point to the file.
- **Port is already in use:** select another port, such as `dataforge-sba dashboard --port 8080`.

## Audit and release checks

Build locally with `python -m build`. Run tests with `python -m pytest`. Release verification must also install the wheel in a fresh environment and check both CLI names, dashboard assets, and a generated notebook. DataForge is not published to PyPI by this repository workflow; package publication requires a separate release decision.

## Contributing and license

See [CONTRIBUTING.md](CONTRIBUTING.md). DataForge is released under the MIT License; see [LICENSE](LICENSE).
"# dataforge" 
