Metadata-Version: 2.1
Name: engram-generator
Version: 0.3.0
Summary: Procedural synthetic dataset generator for training reasoning AI — 2,022 generators across 100+ scientific domains
Author: deepnet.one
License: MIT
Project-URL: Homepage, https://www.engram.one
Project-URL: Repository, https://github.com/alexge233/engram_generator
Keywords: machine-learning,curriculum,reasoning,logic,synthetic-data,infinite-dataset
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: MIT License
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: sympy >=1.12
Requires-Dist: numpy >=1.24
Requires-Dist: rich >=13.0
Requires-Dist: pylatexenc >=2.10
Requires-Dist: unicodeit >=0.7
Provides-Extra: atoms
Requires-Dist: requests >=2.28 ; extra == 'atoms'
Requires-Dist: beautifulsoup4 >=4.12 ; extra == 'atoms'
Provides-Extra: dev
Requires-Dist: pytest >=8.0 ; extra == 'dev'
Requires-Dist: pytest-cov ; extra == 'dev'
Requires-Dist: ruff ; extra == 'dev'

[www.engram.one](https://www.engram.one) | [Samples](https://www.engram.one/samples.html) | [Skill Tree](https://www.engram.one/tree.html) | [3D Graph](https://www.engram.one/graph.html) | [PyPI](https://pypi.org/project/engram-generator/)

Teaching AI Models how to reason and think is tricky, very tricky. There are a lot of Datasets out there which attempt to do that, most by using NLP and text.

*This is not what the Engram Generator does!*

This a Real-Science focused Synthetic Generator, same to how a Teacher or Professor behaves:
- It uses real Theorems, Formulas, Examples and problems
- The ML model is then presented with a problem, just like you did at school
- While training, the generator decomposes the solution into steps, and then presents the final answer

For evaluation and testing, a similar approach is taken:
- The generator produces a new problem, passing the needed context to solve it
- The ML model is expected to compose a solution and an answer

Wether you want to reward the final answer or the entire solution (a.k.a the reasoning or logic chain) is entirely up to you. A final answer reward or loss can be used, or you can use a `ReasoningChain` which will evaluate the model's solution as well.
Me and my mate `Opus 4.6[1m]` built this Dataset generator to address 3 key issues:
1. I wanted a Dataset which is **infinite** and thus cannot be memorised. By adjusting the values in formulaisms and problems, no model can actually memorise the solution
2. I believe that learning the **underlying** logic of how to solve a problem is far more important that memorising a pattern, or a statistical solution.
3. The ability to reason must be **traceable** and **observable** and thus part of the Dataset.

And what better way to do this, other than using the largest part of the Human Scientific Knowledge? By doing so, English and Natural Language aren't the primary focus, they are simply the enabler. Symbolism at the heart of Sciences and the relationships within are the focus.
However, representing and using such a vast corpus is an enormous undertaking for a single Developer doing it as a side project, which is where Opus 4.6 comes into play.


## What we have created

This is a Python-based procedural dataset which contains:

**2,022 generators. 100+ scientific domains. 10^81 unique problems.**

It uses:
- Mathematics
- Physics
- Chemistry
- Biology
- Computer science
- Engineering
- Quantum theory
- Earth sciences
- Economics
- Logic
and more, as step-by-step reasoning problems.
The goal is not to build a benchmark. The goal is to teach machines how humans reason, discover, and invent.

Every problem is generated on-the-fly, there is no file to download.

## Installing
```
pip install engram-generator
```

Current version is 0.3.0 and it contains currently **unverified** samples which you should not use.

## Anti-Memorising Approach

Models trained on static datasets learn to pattern-match, not to reason. 
Train a model on 10,000 addition problems and it learns a lookup table, not addition. 
Change the digit count and it breaks. 
That's memorisation pretending to be intelligence.

Human reasoning didn't develop by memorising answers. 
It developed by solving problems across domains; by recognising that the same recursive structure appears in Fibonacci sequences, merge sort, and mathematical induction. 
That a conservation law works the same way in thermodynamics, circuit analysis, and chemical equilibria. 
That a proof by contradiction in logic uses the same mental move as a reducibility argument in computability theory.

So the approach here is exactly the same:
- Force the model to learn the underlying structure, by repeating the problems using different variables and digits.
- If the model learns easier Tiers (lower levels such as 0 ~ 3), the Dataset Generator has the ability to increase the difficulty and complexity

What this effectively creates, is a levelling up system


## Levelling System

You don't get to attempt RSA encryption until you can do modular arithmetic.
You don't get to critique a proof until you can write one.
The skill tree enforces this:

| Tier | Tasks | What You Unlock | Examples |
|---|---|---|---|
| 0 | 20 | **Fundamentals** | Addition, subtraction, sorting, boolean logic |
| 1 | 36 | **Building blocks** | Multiplication, Fibonacci, Caesar cipher |
| 2 | 47 | **Algebra & graphs** | Derivatives, quadratics, graph reachability |
| 3 | 95 | **Real maths** | Integrals, determinants, boolean algebra |
| 4 | 313 | **Applied science** | Physics, probability, dynamic programming |
| 5 | 730 | **Expert territory** | PDEs, cryptography, quantum mechanics |
| 6 | 521 | **Graduate level** | Topology, general relativity, information theory |
| 7 | 176 | **Meta-reasoning** | Proof strategy, error detection, generalisation |
| 8 | 31 | **Creative** | Conjecture, isomorphism detection |
| 9 | 29 | **Research** | Algorithm design, impossibility proofs |
| 10 | 24 | **Self-architecture** | Scaling laws, architecture search, loss design |


## Reasoning Balance

Without balancing, formula-substitution problems (55% of generators) would dominate training. The model would learn to plug numbers into equations and call it a day.

Instead, training is balanced across **26 reasoning strategies**. Each gets equal exposure regardless of how many generators belong to it:

| Pattern | Generators | What it teaches |
|---|---|---|
| Formula substitution | 1,188 | Plug values into known equations |
| Meta-reasoning | 112 | Proof strategy, error analysis, architecture design |
| Probabilistic reasoning | 87 | Bayes, distributions, expected values, stochastic processes |
| Differential equations | 76 | ODEs, PDEs, boundary value problems, numerical methods |
| Graph traversal | 73 | BFS, DFS, Dijkstra, flow networks, connectivity |
| Simulation trace | 60 | State machines, data structures, protocol execution |
| Symbolic manipulation | 49 | Differentiation, integration, algebraic simplification |
| Construction & verification | 39 | Group axioms, homomorphisms, topological invariants |
| Geometric computation | 34 | Areas, volumes, intersections, convex hulls |
| Conservation & balance | 33 | Thermodynamic laws, Kirchhoff, chemical equilibria |
| Linear algebra | 31 | Matrix decomposition, eigenvalues, null spaces |
| Counting & enumeration | 28 | Permutations, Catalan numbers, inclusion-exclusion |
| Statistical inference | 26 | Hypothesis testing, confidence intervals, regression |
| Logical deduction | 26 | Natural deduction, resolution, sequent calculus |
| Modular arithmetic | 22 | CRT, Euler's totient, discrete logarithms |
| Transform methods | 20 | Fourier, Laplace, Z-transform, wavelets |
| Dynamic programming | 19 | Optimal substructure, memoisation, alignment |
| Optimization | 19 | Gradient descent, KKT conditions, convex methods |
| Series & convergence | 19 | Ratio test, power series, uniform convergence |
| Encoding & decoding | 15 | RSA, Huffman, Reed-Solomon, stream ciphers |
| Approximation & numerical | 14 | Newton-Raphson, quadrature, interpolation |
| Recursive decomposition | 11 | Divide-and-conquer, Tower of Hanoi, merge sort |
| Comparison & ordering | 9 | Periodic trends, mineral identification, ranking |
| Dimensional analysis | 8 | Unit conversion, significant figures, calibration |
| Greedy selection | 3 | Interval scheduling, bin packing, set cover |
| Search & backtracking | 1 | A*, constraint satisfaction |

Formula substitution has 1,188 generators. Search & backtracking has 1. But during training, both patterns get **3.8% of samples**. No free rides.

## Why Memorisation is Impossible

The entire curriculum is **1.85 MB of algorithms**. It produces **terabytes of unique instances**. That's a compression ratio of 1,250,000:1.
Meaning that if you use the full Engram Generator without tool or function calling, no current model can memorise the dataset.
Which brings us to the second issue: **pattern matching** in modern ML.

The models can still learning pattern matching, by learning the Heurstics, and that, to a certain degree is useful.
However, what you can do is to evaluate or test the model in Out Of Sample (OOS) Difficulty tiers.
For example, a Model trained on simple Arithmetic, if capable of high quality pattern matching and heuristics, would be able to translate what it learned in unseen more difficult Problems.
Furthermore, Certain problems require other types of knowledge to be properly acquired, for example:

```
Addition (T0) -> Multiplication (T1) -> Matrix Multiply (T4) -> Eigenvalues (T4) -> PCA, Quantum Systems, Stability Analysis (T5-T7)
```
  
Meaning, a model that can't do addition can't learn multiplication.
Without multiplication, it can't do matrix multiplication.
Without matrix multiplication, it can't compute eigenvalues.
And without eigenvalues, it can't do PCA, solve quantum two-level systems, analyse ODE stability, compute stress tensors, or do spectral decomposition.

## Tokenizer

All mathematical notation is written in **LaTeX**.
The model learns to read and write LaTeX as a native language: fractions, integrals, matrices, Greek letters (spelled out), superscripts, subscripts, and nested expressions.
This means a model trained on Engram Generator doesn't just learn to solve maths -- it learns the standard notation that humans use to communicate it.

```
\frac{d}{dx}(-x^2-2x-2) <step> -1*2x=-2x <step> -2*1=-2 <step> 0 <step> -2x-2

\begin{pmatrix} -5 & 3 \\ 2 & 2 \end{pmatrix} \times \begin{pmatrix} -1 & -2 \\ -3 & 8 \end{pmatrix}

\oint_{|z|=3} \frac{1}{z^{2}+z-6} dz <step> poles: z=-3, 2 <step> Res(f,2)=0.2 <step> 1.2566i
```

Engram Generator uses a **character-level tokenizer**: every character maps to exactly one token. No subword merging. No BPE. No SentencePiece.

**Why?** Subword tokenizers destroy the structure that reasoning depends on:

- **Digit atomicity**: BPE merges `"123"` into a single token. The model can't see that the `3` is in the ones place and the `1` is in the hundreds place. Arithmetic becomes impossible. Character-level tokenization keeps every digit separate, so carry operations and place-value reasoning work naturally.
- **LaTeX preservation**: LaTeX uses nested braces, superscripts, and subscripts (`\frac{d}{dx}`, `x^{2}`). Subword tokenizers split these unpredictably -- `\frac` might become `\fr` + `ac`, breaking the command boundary. Character-level tokenization preserves brace matching, command names, and operator structure exactly as written.
- **Deterministic alignment**: Every character is exactly one token. No ambiguity about tokenization boundaries. The model's attention patterns can align precisely with the mathematical structure of the problem.

**The character set** (132 characters + 3 special tokens = 135 vocab):

| Category | Characters |
|---|---|
| Digits (10) | `0 1 2 3 4 5 6 7 8 9` |
| Lowercase (26) | `a b c ... z` |
| Uppercase (26) | `A B C ... Z` |
| Greek (12) | `α β γ δ ε θ λ μ π σ φ ω` |
| Arithmetic (5) | `+ - * / ^` |
| Relations (4) | `≤ ≥ ≠ ≈` |
| Grouping (6) | `( ) [ ] { }` |
| Calculus & analysis (4) | `∂ ∫ √ ∞` |
| Set theory (5) | `∈ ⊂ ∅ ∩ ∪` |
| Logic (9) | `∀ ∃ ¬ ∧ ∨ ⊢ ⊨ ↔ ⊥` |
| Punctuation (9) | `= : ; ? . , ! ' "` |
| LaTeX & structure (7) | `\ _ \| ~ < > %` |
| Other (9) | `# @ $ & ° × — → (space)` |
| **Special tokens** (3) | `<pad> <eos> <step>` |

The `<step>` token separates solution steps in the target sequence. All generator output is constrained to use only characters in this set -- any generator that produces a character outside it is a bug and is caught by the test suite.

## Samples

```
Input:  add two 5 digit numbers
Target: 13278 + 46048 <step> 8+8=16 <step> 7+4+1=12 <step> 2+0+1=3 <step> 3+6=9 <step> 1+4=5 <step> 59326
```

- **Input**: natural language task description
- **Target**: problem, solution steps, and answer separated by `<step>` tokens
- Both capped at **512 characters**

## Usage

### Generate samples

```python
from engram_generator.curriculum.registry import get_generator

gen = get_generator("addition", min_difficulty=3, max_difficulty=5)
samples = gen.generate(100)

for sample in samples[:3]:
    print(f"Input:  {sample.input_text}")
    print(f"Target: {sample.target_text}")
    print(f"Answer: {sample.answer}")
```

### Use the skill tree

```python
from engram_generator.curriculum.registry import get_all_generators
from engram_generator.curriculum.skill_tree import SkillTree

generators = get_all_generators()
tree = SkillTree(generators, retention_ratio=0.1)

# See what's unlocked
print(tree.get_unlocked_tasks())

# Level up by proving mastery
events = tree.update({"addition": 0.97, "subtraction": 0.85})
```

### Balanced training

```python
from engram_generator.curriculum.reasoning_patterns import (
    get_pattern_weights, get_pattern_summary,
)
from engram_generator.curriculum.registry import get_all_generators

gens = get_all_generators()
weights = get_pattern_weights(gens)

# Each of the 26 reasoning patterns gets equal training exposure
summary = get_pattern_summary(gens)
for pattern, count in sorted(summary.items(), key=lambda x: -x[1])[:5]:
    print(f"{pattern}: {count} generators -> 3.8% of training")
```

## A Word of Warning:
- I am actively developing this at the moment, and changes will happen
- I can not review manually the entire Universe of Human Knowledge and Discovery, which is why the code for the largest part is Claude-generated
- There's been a lot of tokens burnt to verify the input, solution and answers, this is done by using 3rd party libraries to verify answers where possible.
- Where **not possible** if a Wikipedia example exists, a double-blind verification is done, where Claude creates a generator with the same arguments, and the Generator's answer must match what the Wiki example states. This is still fallible to bad parsing
- Where there are no examples, no libraries, and only formulae or therems, they are currently marked as **unverified**. Those will be verified using `Fable` and eventually audited manually by a Human. This is too verify that the Engram Generator doesn't produce incorrect Samples or Garbage.

## License

MIT

## Organisation

[www.engram.one](https://www.engram.one) · [www.deepnet.one](https://www.deepnet.one)
