Metadata-Version: 2.4
Name: intelliant
Version: 0.1.0a2
Summary: Semantic core extraction on graphs using Ant Colony Optimization
Project-URL: Homepage, https://github.com/yourdisenchantment/intelliant
Project-URL: Repository, https://github.com/yourdisenchantment/intelliant
Project-URL: Issues, https://github.com/yourdisenchantment/intelliant/issues
Author-email: Pavel <p.idisenchantment@gmail.com>
License: MIT
License-File: LICENSE
Keywords: ant-colony-optimization,clustering,embeddings,graph-clustering,unsupervised-learning
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.14
Requires-Dist: numba
Requires-Dist: numpy
Requires-Dist: pynndescent
Requires-Dist: scikit-learn
Requires-Dist: scipy
Requires-Dist: tqdm
Description-Content-Type: text/markdown

# Intelliant: Graph-Based Algorithm for Semantic Core Extraction

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.14](https://img.shields.io/badge/python-3.14-blue.svg)](https://www.python.org/downloads/release/python-3140/)
[![intelliant-core (old)](https://img.shields.io/badge/intelliant--core-old%20%2F%20deprecated-lightgrey.svg)](https://pypi.org/project/intelliant-core/)
[![intelliant](https://img.shields.io/badge/intelliant-0.1.0a2%20alpha-orange.svg)](https://pypi.org/project/intelliant/)

**Intelliant** is a specialized clustering algorithm based on the Ant Colony
Optimization (ACO) metaheuristic.

Unlike classical methods, the algorithm does not draw mathematical cluster
boundaries in hyperspace. It transforms high-dimensional data (e.g. LLM
embeddings) into a k-nearest-neighbor graph (KNN) and uses swarm intelligence
to find the **densest semantic centers (cores)**.

The algorithm is optimized with `Numba` and sparse matrices (CSR), which
allows processing hundreds of thousands of objects in seconds on a regular
CPU, avoiding the memory footprint of industrial standards like HDBSCAN.

## Status and packages

> **Alpha.** The package is under active development (research phase). The
> public API still changes between versions. This is a personal research
> tool, not a production-ready solution. See [ROADMAP.md](ROADMAP.md) for
> what is done and what is ahead; long-term research ideas are in
> [RESEARCH_NOTES.md](RESEARCH_NOTES.md).

- **Current package:** [`intelliant`](https://pypi.org/project/intelliant/)
  (three-class architecture).
- **Old package (deprecated):**
  [`intelliant-core`](https://pypi.org/project/intelliant-core/) - single-class
  `IntelliantCoreExtractor` architecture. The link is kept for those who
  arrived via the old name; do not install it, development moved to
  `intelliant`.

## Features

- **Dimension-independent.** Works on a similarity graph (cosine, euclidean),
  not on raw coordinates. Suitable for both 2D/3D and 384D+ embeddings.
- **Min-Max Ant System (MMAS).** Pheromone stagnation protection prevents the
  graph from collapsing into a single hub "black hole".
- **Elite ants.** Accelerated core formation via greedy behavior of a
  designated agent group, with a configurable start iteration (separating
  exploration / exploitation phases).
- **Node density heuristic.** Optional: ants evaluate the local density of
  the target node, accelerating convergence.
- **Two-stage noise absorption.** Label propagation through pheromone waves,
  then centroid fallback for isolated points. Available both as a single call
  and as separate stages (for frame-by-frame visualization and intermediate
  state caching).
- **Giant detection.** Signals a suspected cluster merge/inflation by a size
  gap (without modifying the data).
- **Transparent state.** All intermediate artifacts (graph, pheromone field,
  raw cores, labels) are accessible via attributes for visualization,
  debugging, and caching between sessions.

## Architecture

Clustering is split into three conceptually independent classes, each in its
own module:

| Class | Module | Input | Output |
| --- | --- | --- | --- |
| `GraphBuilder` | `intelliant.graph_builder` | embeddings `X` | similarity graph (CSR) |
| `PheromoneExtractor` | `intelliant.pheromone_extractor` | graph | pheromone graph |
| `CoreClusterer` | `intelliant.core_clusterer` | pheromone graph + threshold | cluster labels |

There is no "all-in-one" call: the three parts solve different problems
(data processing, pheromone production, pheromone interpretation into
clusters), and the separation is deliberate so that each stage can be tuned
and inspected independently. The pheromone cutoff threshold and embedding
extraction are currently prepared by the user outside the library.

## Requirements

- **Python:** `>= 3.14`
- **Package manager:** `uv`
- **Dependencies** (installed automatically): numpy, scipy, numba,
  scikit-learn, pynndescent, tqdm.
- **Hardware acceleration** (for generating embeddings with a separate
  model): CUDA / MPS via PyTorch - in the `embeddings` dependency group.

## Installation

From PyPI (alpha):

```bash
pip install intelliant
# or explicitly the alpha version:
pip install intelliant==0.1.0a2
```

From source (for development):

```bash
git clone https://github.com/yourdisenchantment/intelliant.git
cd intelliant
uv sync                           # library only
uv sync --all-groups --all-extras # + notebooks, embeddings, dev tools
```

Dependency groups: `notebooks` (jupyter, visualization, polars, umap),
`embeddings` (torch, sentence-transformers, datasets),
`dev` (pre-commit, commitizen, ruff, pyright, scipy-stubs, bandit, deptry,
vulture).

To load test datasets, create a `.env` in the project root with a Hugging
Face token:

```text
HF_API_TOKEN=hf_your_token_string
```

## Usage example

The pipeline consists of three steps: build the graph, run the ant colony,
extract clusters. The pheromone cutoff threshold is set by the user (below -
a simple example via percentile).

```python
import numpy as np
from intelliant import GraphBuilder, PheromoneExtractor, CoreClusterer

# X - embedding matrix (N, D), prepared by the user

# 1. Similarity graph from embeddings
graph = GraphBuilder(
    n_neighbors=15,
    metric="cosine",
    mutual=True,         # mutual KNN (AND-symmetrization)
    min_connections=5,   # connectivity top-up for isolated points
    knn_method="auto",   # exact for small datasets, approx for large
    random_state=42,     # for reproducible approx search
).build(X)

# 2. Ant colony run (graph -> pheromone graph)
aco = PheromoneExtractor(
    n_ants=len(X),              # explicit ant count (critical parameter)
    n_iterations=20,
    use_elite_ants=True,
    elite_start_iteration=10,   # elite kicks in from the middle of the run
    random_state=42,            # colony seed (independent of graph seed)
)
aco.fit(graph)
pheromones = aco.pheromone_matrix_

# 3. Cutoff threshold (user computes it; here - percentile)
threshold = np.percentile(pheromones.data, 90)

# 4. Core extraction and noise absorption
clusterer = CoreClusterer(min_cluster_size=50, batch_size=200_000)
cores = clusterer.extract_cores(pheromones, threshold)   # cores 0..k-1, noise -1
labels = clusterer.absorb(pheromones, X)                 # noise fill-in

# Intermediate state is available for diagnostics and visualization:
#   clusterer.cores_             - raw cores before absorption
#   clusterer.labels_pheromone_  - after pheromone waves (stage 1)
#   clusterer.labels_            - final labels
```

Absorption stages can be called separately (`absorb_pheromone`, then
`absorb_centroid`) - this gives three state snapshots (raw cores -> after
waves -> final) and allows saving intermediate results between sessions.

Each class stores its result in an attribute with a trailing underscore
(`graph_`, `pheromone_matrix_`, `cores_`, `labels_pheromone_`, `labels_`)
per sklearn convention and also returns it from the method.

## Project structure

```
src/intelliant/     # public API: three classes + threshold module
tests/              # pytest suite (224 tests, 20 files; 223 default + 1 slow)
notebooks/          # calibration and dataset notebooks
old_notebooks/      # first-iteration experiments (see below)
utils/              # notebook support scripts (metrics, tee)
data/, results/     # data and results (gitignored, not in repo)
```

### `old_notebooks/`

First-iteration notebooks from the early experimental phase of intelliant.
Two subdirectories:

- `old_notebooks/1/` - initial single-class prototypes (2D/3D synthetic tests,
  AG News, HDBSCAN comparison).
- `old_notebooks/2/` - second round of experiments before the three-class
  refactor (synthetic 2D/3D, AG News, benchmarks).

These are kept for historical reference. The current architecture
(`src/intelliant/`) supersedes them.

## Roadmap

- **What is ahead** (current and near-term work) - [ROADMAP.md](ROADMAP.md).
- **Long-term research ideas** (multilevel hierarchical clustering, etc.) -
  [RESEARCH_NOTES.md](RESEARCH_NOTES.md). These are notes for the future,
  not commitments of the current project.

## License

MIT.
