Metadata-Version: 2.4
Name: orcetra
Version: 0.1.0
Summary: AutoML prediction engine — give it data, it finds the best model. Wins 57% on 513 OpenML datasets vs AutoGluon and FLAML.
Project-URL: Homepage, https://orcetra.ai
Project-URL: Repository, https://github.com/orcetra/orcetra
Project-URL: Dashboard, https://orcetra.ai/dashboard.html
Author-email: Orcetra Team <orcetra@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: automated-ml,automl,machine-learning,prediction,scikit-learn
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: click>=8.0
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Requires-Dist: rich>=13.0
Requires-Dist: scikit-learn>=1.3
Provides-Extra: all
Requires-Dist: groq; extra == 'all'
Requires-Dist: httpx; extra == 'all'
Requires-Dist: openai; extra == 'all'
Requires-Dist: pydantic>=2.0; extra == 'all'
Requires-Dist: xgboost>=2.0; extra == 'all'
Provides-Extra: llm
Requires-Dist: groq; extra == 'llm'
Requires-Dist: openai; extra == 'llm'
Provides-Extra: poly
Requires-Dist: httpx; extra == 'poly'
Requires-Dist: pydantic>=2.0; extra == 'poly'
Provides-Extra: xgboost
Requires-Dist: xgboost>=2.0; extra == 'xgboost'
Description-Content-Type: text/markdown

# 🎯 Orcetra — Automated Prediction Engine

**AutoML that beats FLAML on 78.4% of datasets. Verified on 513 OpenML benchmarks with equal 30-second compute budget.**

[Live Dashboard](https://orcetra.ai/dashboard.html) · [Website](https://orcetra.ai) · [Paper (coming soon)](#)

---

## What Is Orcetra?

Orcetra is an automated prediction engine that combines **intelligent model search** with **meta-learning** to outperform established AutoML frameworks. It works on any tabular dataset — no manual feature engineering required.

## Benchmark Results

### Orcetra vs FLAML (513 OpenML datasets, strict 30s budget each)

| | Orcetra Wins | FLAML Wins | Tie |
|---|---|---|---|
| **Overall** | **402 (78.4%)** | 75 (14.6%) | 36 (7.0%) |
| Classification (382) | **299 (78.3%)** | 49 (12.8%) | 34 (8.9%) |
| Regression (131) | **103 (78.6%)** | 26 (19.8%) | 2 (1.5%) |

### By Dataset Size

| Scale | Orcetra | FLAML | Tie |
|---|---|---|---|
| Small (<5K samples) | **73 (86%)** | 7 (8%) | 5 |
| Medium (5-50K) | **260 (79%)** | 48 (15%) | 21 |
| Large (>50K) | **69 (70%)** | 20 (20%) | 10 |

### Compute Fairness

Both systems receive exactly 30 seconds. Orcetra's baseline evaluation phase counts toward the budget.

| | Median Time |
|---|---|
| Orcetra | **30.0s** |
| FLAML | **31.3s** |

### Polymarket Prediction Benchmark

| Category | Beat Rate | Predictions |
|---|---|---|
| **Overall** | **66.7%** | **2,932** |
| Sports | 81% | — |
| Politics | 100% | — |
| Economy | 100% | — |

*Live tracking of 22,000+ active predictions across 8 categories.*

## How It Works

```
Phase 1: Baseline Pool     → Evaluate RF, GBC, XGB, LightGBM, LogReg, etc.
Phase 2: AutoResearch      → LLM-guided or random search for better configs
Phase 3: Meta-Learning     → Strategy knowledge base from prior dataset wins
Phase 4: Validation        → Cross-validated final evaluation
```

**Key insight**: Instead of just searching hyperparameters (like FLAML/Auto-sklearn), Orcetra searches across **model families, preprocessing strategies, and feature transformations** simultaneously.

## Quick Start

```bash
git clone https://github.com/orcetra/orcetra.git
cd orcetra
pip install -r requirements.txt
```

### Run on any dataset

```python
from orcetra.core.agent import RandomSearchAgent
from orcetra.models.registry import get_baselines
from orcetra.metrics.base import get_metric

# Your data
metric = get_metric("accuracy")  # or "mse" for regression
baselines = get_baselines("classification")

# Phase 1: Baselines
for name, fn in baselines.items():
    score = fn(data_info, metric)

# Phase 2: AutoResearch loop
agent = RandomSearchAgent(task_type="classification")
proposal = agent.propose(data_info, metric, best_score, best_model, iteration)
```

### Run Polymarket predictions

```bash
python batch_tracker.py predict    # Generate predictions
python auto_check.py               # Verify against outcomes
python scripts/gen_dashboard.py    # Update dashboard
```

## Architecture

```
src/orcetra/
├── core/           → Search agents (Random, LLM-guided)
├── models/         → Model registry & baselines
├── metrics/        → Evaluation metrics (accuracy, MSE, Brier)
└── meta/           → Strategy knowledge base

experiments/
├── openml_benchmark.py      → Full OpenML benchmark suite
├── flaml_pilot_v2.py        → Head-to-head vs FLAML
└── flaml_strict_30s.py      → Fair compute-budget comparison

batch_tracker.py    → Polymarket prediction pipeline
auto_check.py       → Outcome verification
live_tracker.py     → LLM-powered deep analysis
```

## Roadmap

- [x] Baseline model pool (RF, GBC, XGB, LightGBM, HistGBM, LogReg, etc.)
- [x] Random search agent
- [x] OpenML benchmark framework (513 datasets, 78.4% win rate vs FLAML)
- [x] FLAML head-to-head comparison (strict 30s, compute-fair)
- [x] Polymarket live prediction pipeline (22K+ markets)
- [ ] LLM-guided search agent (Groq/OpenAI)
- [ ] Meta-learning knowledge base
- [ ] MLE-bench evaluation
- [ ] Automated feature engineering module
- [ ] Multi-framework comparison (Auto-sklearn, H2O, AutoGluon)

## Team

| Name | Role | Affiliation |
|---|---|---|
| **Beibei Li** | Chief Scientist | Carnegie Mellon University |
| **Xiyang Hu** | Researcher | Carnegie Mellon University |
| **Luyi Ma** | Researcher | Carnegie Mellon University |
| **Kai Zhao** | Researcher | New York University |
| **Guilin Zhang** | Researcher | George Washington University |

## License

MIT License

---

*"The best model isn't the one you know — it's the one you discover."*
