GitLab Repo

amachine

What is A-Machine?

A-Machine is a work in progress library for constructing epsilon-machines1 and other stochastic models for generating structured symbol sequences with ground truth causal structure and information-theoretic complexity. The intended use case is the study of neural network learning dynamics and internal representations.

Quickstart

Installation

# CPU only
pip install a-machine

# With GPU support (requires CUDA 13)
pip install "a-machine[cuda]" --extra-index-url https://pypi.nvidia.com

Example

More examples are available at the gitlab repo.

import amachine as am

am.srand_global( 42 )

m = am.random_machine( 
    n_states=23, 
    symbols=[ '0', '1' ],
    connectedness=0.35,
    randomness=0.25,
    # ensure_strongly_connected=True,
    # ensure_minimal=True 
)

# Collapse to the largest subgraph
m.collapse_to_largest_strongly_connected_subgraph()

# In case it is not minimal
m.minimize()

# Entropy rate, statistical complexity
print( f"h_mu : {m.h_mu()}" )
print( f"C_mu : {m.C_mu()}" )

# Display the epislon machine
m.draw_graph()

AM Graph

Note that ensure_strongly_connected will guarantee the result is strongly connected and has n_states before minimization, but ensure_minimal will minimize the result and potentially merge states. They each default to False.

HMMs and Epsilon Machines

An $\epsilon$-machine is the minimal, optimally predictive, unifilar, hidden Markov model of a stationary stochastic process. A large body of theory linking them to complex dynamical systems, information theory, and physics, has been developed by James P. Crutchfield and collaborators under the name computational mechanics 12.

A-Machine's HMM implementation is designed primarily for constructing $\epsilon$-machines to generate training data with interesting ground truth theoretical properties. This should be useful for making informed hypotheses about, designing experiments for, and interpreting the learning dynamics and internal representations of neural networks.

More expressive finite generative models, such as non-unifilar HMMs or probabilistic context-free grammars, can represent a broader class of languages than a finite $\epsilon$-machine, but may lack the theoretical tractability that makes $\epsilon$-machines particularly analytically useful. Thus, utilizing $\epsilon$-machines instead of alternatives, may involve a tradeoff between expressiveness and interpretability.

Complexity Measures

Complexity measures implemented within the HMM class include:

Citations, definitions, and interpretations are in their individual documentation sections linked above.

Isomorphic Construction

A few ways to construct $\epsilon$-machines are impemented so far:

  • random_machine, which generates a random HMM that can be reduced to an $\epsilon$-machine,

  • star_join, which joins a list of machines to a central hub,

  • isomorphic_to, which creates a copy of an existing machine with new labels,

  • cyclical_isomorphic_rotation, which creates and star joins $n$ isomorphic machines, where $n$ is the number of symbols, and the $i^{th}$ machine's state $j$ emits the $k^{th}$ symbol in the alphabet, then the ${i+1}^{th}$ machine's state $j$ will emit symbol $k+1 \bmod n$, and

  • unique_isomorphic, which creates and star joins n_machines isomorphic machines, where n_machines is a parameter, and each has its own alphabet.

  • random_structured, which is similar to random except that it accepts a [StructuredSymbolSet](amachine/am_structured_symbol_set.html#StructuredSymbolSet), that contains "cohesion" rules and symbol categories which promote structure in the placement of edge symbols, and provide a categorical basis for isomorphic rotation,

  • isomorphic_to_with_category_rotation, which creates a copy of an existing machine with symbols rotated by a shift within select categories given a symbol categorization,

  • full_structured_isomorphic_rotation, which produces the full rotation on a random_structured HMM using isomorphic_to_with_category_rotation and optionally star_join's them, otherwise returns the list of generated HMM's,

  • structured_composition, which accepts groups of HMM's, with variable multiplicity, optionally end_of_component symbols, and various structural and cohesion synthesis parameters, and joins according to the generated cohesion rules.

  • random_structured_composition, which is generates components and composes them using structured_composition.

A small example is shown below, that generates 2 random full_structured_isomorphic_rotation's and then composes them. Since there are 2 symbols per-category, the 2 full rotations produce 2 isomorphic groups of size 2, thus 2 unique subgraphs, and with instances_per_component=2, there will be 8 total subgraph instances. 

import amachine as am

am.srand_global(42)

n_symbol_categories=2
symbols_per_category=2

symbol_set = am.StructuredSymbolSet.generate(
    n_categories=n_symbol_categories,
    symbols_per_category=[symbols_per_category]*n_symbol_categories,
    rigidity=0.75,
    within_category_variability=0.1,
    category_repetition_penalty=0.75,
    symbol_repetition_penalty=0.75
)

instances_per_component = 2
component_target_sizes = [ 3, 5 ]
all_components : dict[str,list[am.HMM] ] = {}
component_instance_counts : dict[str,list[int]] = {}

for iso_idx, target_size in enumerate( component_target_sizes ) :

    iso_id = f"{iso_idx}" 

    all_components[ iso_id ] = am.full_structured_isomorphic_rotation(
        isoclass_name=f"{iso_idx}",
        n_states=target_size,
        symbol_set=symbol_set,
        connectedness=0.68,
        randomness=0.25,
        star_joined=False,
        max_search_time=60
    )

    component_instance_counts[ iso_id ] = [
        instances_per_component for _ in all_components[ f"{iso_idx}" ] 
    ]

m = am.structured_composition(
    composition_id="0",
    core_alphabet=symbol_set.symbols,
    composition_rigidity=0.85,
    component_groups=all_components,
    instances_per_component=component_instance_counts,
    component_residency_factor=0.75,
    component_repetition_penalty=0.75,
    composition_connectivity_factor=0.75,
    instance_cohesion_randomness=0.0,
    end_of_component_symbols={ "!", "?" },
    symbol_cohesion=symbol_set.symbol_cohesion,
    max_search_time=60
)

m.draw_graph( 
    output_dir=".",
    engine='dot',
    color_nodes_by="cm",
    color_borders_by="isoclass",
    color_edges_by="category",
    symbol_categories=symbol_set.symbol_categories,
    no_labels=True,
    show=True,
    format="svg"
)

The result is shown below. Nodes with the same fill color are from idential sub-graphs that appear more than once because of the instances_per_component parameter. The node border color represents the isoclass and in this case euqivalently the group the state belongs to (red and blue), which includes all sub-graphs that are isomorphic with eachother (i.e., same based HMM with category rotations). The edge color represents the category of the emitted symbols, with black being the end_of_component symbols.

AM Graph

A few things to note:

StructuredSymbolSet.generate automatically selects symbols from among digits 0-9, acii lowercase and uppercase letters, and uppercase and lowecase unicode Greek symbols. What remains that is part of the vocabulary of build in toy transformer models, and can be used for end_of_component_symbols, is ascii punctuation characters, brackets, math, and special characters, as well as the unicode geometry block. See Vocabulary for the full set of characters, which are one-to-one with the vocabulary of the toy models.

There will always only be one entry node and one exit node per component, and since the compositon must be unifilar, there can be at most len(end_of_component_symbols) outgoing edges from each component. Thus for more connectedness between components, in addition to using composition_connectivity_factor, you must choose enough end of component symbols. If end_of_component_symbols is omitted None, then components will be connected with symbols from the standard alphabet.

The effect of using end of component symbols will be that a neural network training on the data will have hints that help it better synchronize. Other not-yet-implemented options include unique entry symbols per-isogroup, per-component, or per-component instance, which would allow the neural network to instantly partially or fully syncronize upon each component to component transition. This could be interesting to control for the sake of encouraging or helping the model be able to utilize redundencies and generalize over isomorphic groups.

A large case with 2 full_structured_isomorphic_rotation's, 5 symbols per-category, and instances per-component is shown below. Note that the total number of symbols asides from the end of component symbols, is n_categories*symbols_per_category, and since the connectedness parameter in full_structured_isomorphic_rotation ranges from 0 to 1, and operates on the number of possible edges per-state, you can expect around connectedness*n_categories*symbols_per_category edges per-state.

import amachine as am

am.srand_global(42)

n_symbol_categories=4
symbols_per_category=5

symbol_set = am.StructuredSymbolSet.generate(
    n_categories=n_symbol_categories,
    symbols_per_category=[ symbols_per_category ]*n_symbol_categories,
    rigidity=0.75,
    within_category_variability=0.1,
    category_repetition_penalty=0.75,
    symbol_repetition_penalty=0.75
)

instances_per_component = 2
component_target_sizes = [ 9, 11 ]
all_components : dict[str,list[am.HMM] ] = {}
component_instance_counts : dict[str,list[int]] = {}

for iso_idx, target_size in enumerate( component_target_sizes ) :

    iso_id = f"{iso_idx}" 

    all_components[ iso_id ] = am.full_structured_isomorphic_rotation(
        isoclass_name=f"{iso_idx}",
        n_states=target_size,
        symbol_set=symbol_set,
        connectedness=0.15,
        randomness=0.25,
        star_joined=False,
        max_search_time=60
    )

    component_instance_counts[ iso_id ] = [
        instances_per_component for _ in all_components[ f"{iso_idx}" ] 
    ]

m = am.structured_composition(
    composition_id="0",
    core_alphabet=symbol_set.symbols,
    composition_rigidity=0.85,
    component_groups=all_components,
    instances_per_component=component_instance_counts,
    component_residency_factor=0.75,
    component_repetition_penalty=0.75,
    composition_connectivity_factor=0.75,
    instance_cohesion_randomness=0.0,
    end_of_component_symbols={ ".", "!", "?" },
    symbol_cohesion=symbol_set.symbol_cohesion,
    max_search_time=60
)

m.draw_graph( 
    output_dir=".",
    engine='dot',
    color_nodes_by="cm",
    color_borders_by="isoclass",
    color_edges_by="category",
    symbol_categories=symbol_set.symbol_categories,
    no_labels=True,
    show=False,
    format="svg"
)

The result is shown below.

AM Graph

Mixed State Presentation

The mixed states presentation construction 34 tracks beliefs about which intrinsic state the known $\epsilon$-machicine is in as an observer attempts to syncronize with the $\epsilon$ machine. MSP construction can be useful for analyzing learning dynamics, because this is something like what a neural network must learn to do to utilize context during inference. Experiments correlating HMM belief states and transformer representations have been done by Shai et al. 5.

The MSP can explode in terms of the number of states that are produced, or even be inifite for non-synchronizable processes. The belief states transitions also may produce fractal-like patterns in the probability simplex.

A-Machine implements MSP construction from a given $\epsilon$ machine. Since the MSP may have too many states to compute, a cap on the max number of states is imposed, and when the cap is reached, the MSP is closed by mapping the unresolved frontier of belief states back to the nearest existing belief states. When this happens, the final MSP is only an approximation.

The MSP also yields closed form expressions for a range of complexity measures 6, such as $\mathbf{E}, \mathbf{S}$, and $\mathbf{T}$, which can be solved for using spectral decompositon. A-machine implements this approach to computing these complexity measures, in addition to the alternative, block entropy convergence. When the MSP construction excedes the max states cap, comparing the measures computed from its approximation with those approximated by block entropy convergence is useful for corroboration and gaining additional insight into the synronization dynamics.

Block Entropy Convergence

As mentioned, block entropy convergence supports estimation of a range of complexity measures, and allows you to observe the convergence in uncertainty as a function of $L$ (after seeing all blocks of length $L$). A-machine implements this as a C++ extension.

You also plot block entropy curves and block measures, by calling amachine.HMM.draw_block_measure_curves, and amachine.HMM.draw_block_entropy_curve.

block measures plots

$\mathcal{H}(L)$ represents the average state uncertainty after setting all subsequences in the language of the machine of length $L$. And thus, you could utilize this measure to estimate how much training data and context a neural network may require at minimum to synchronize with the HMM. It often falls off very quickly, but also often has a very long tail depending on the complexity of the model. For example, with the larger structured composition in the previous example:

$L$ $\mathcal{H}(L)$
1 3.64153
100 0.202524
200 0.067278
300 0.022828
400 0.007783
500 0.002656
600 0.000905
700 0.000308
800 0.000104
900 0.000035
1000 0.000012 ,

while for the first random machine example,

$L$ $\mathcal{H}(L)$
1 2.17641
5 0.609825
10 0.126823
15 0.026362
20 0.005412
25 0.001112
30 0.000221
35 0.000045
40 0.000009

Data Generation

Data generation generates stochastic sequences from a given machine, optionally along with the sequence of hidden states that were traversed, and also optionally with an additional symbol sequence with what we are calling isomorphic_shift applied. This changes the symbols that were emited by a state, for symbols that would have been emitted by alternative isomorphic states (should they exist).

import amachine as am

am.srand_global(42)

m = am.random_machine( 
    n_states=11, 
    symbols=[ '0', '1', '2' ],
    connectedness=0.65,
    randomness=0.37 )

# Collapse to the largest recurrent subgraph
m.collapse_to_largest_strongly_connected_subgraph()

# Minimize the machine -> epsilon-machine.
m.minimize()

# Create an isomorphic machine with new labels
m_iso = am.isomorphic_to( m, alphabet=[ '3', '4', '5' ] )

m_iso.isoclass = 0
m.isoclass     = 0

for j, state in enumerate( m.states ) : 
    m.states[ j ].add_isomorph( m_iso.states[ j ].name )
    m_iso.states[ j ].add_isomorph( m.states[ j ].name )

# Join them together
m_star = am.star_join(
    exit_symbol='x', 
    enter_symbols=['a','b'],
    machines=[ m, m_iso ],
    mode_residency_factor=0.75 
)

m_star.draw_graph()

m_star.generate_data(
    file_prefix="./data/iso_012-345", # saves as iso_012-345.parqet
    n_gen=2_000_000_000, # 2 billion symbols    
    include_states=True,
    isomorphic_shifts={1}
)

iso machine

The two sequences will be aligned, and with identical local and global statistics. Thus, you can train on one of them, and then analyze activation patterns and residuals on both of them in parallel during inference, to study if and how the neural network learned polysemantic representations that exploit the structural redundancies.

Revisiting the large structured composition example, you can apply multiple shifts. In this case, since we have isomorphic groups over category rotations, with symbols_per_category symbols in each category, generate aligned rotated sequences for shifts in { 1, 2, ..., symbols_per_category-1 }.

from pathlib import Path
import amachine as am

am.srand_global(42)

n_symbol_categories=4
symbols_per_category=5
instances_per_component = 2
component_target_sizes = [ 9, 11 ]

symbol_set = am.StructuredSymbolSet.generate(
    n_categories=n_symbol_categories,
    symbols_per_category=[ symbols_per_category ]*n_symbol_categories,
    rigidity=0.75,
    within_category_variability=0.1,
    category_repetition_penalty=0.75,
    symbol_repetition_penalty=0.75
)

all_components : dict[str,list[am.HMM] ] = {}
component_instance_counts : dict[str,list[int]] = {}

for iso_idx, target_size in enumerate( component_target_sizes ) :

    iso_id = f"{iso_idx}" 

    all_components[ iso_id ] = am.full_structured_isomorphic_rotation(
        isoclass_name=f"{iso_idx}",
        n_states=target_size,
        symbol_set=symbol_set,
        connectedness=0.15,
        randomness=0.25,
        star_joined=False,
        max_search_time=60,
    )

    component_instance_counts[ iso_id ] = [
        instances_per_component for _ in all_components[ iso_id ] 
    ]

m = am.structured_composition(
    composition_id="0",
    core_alphabet=symbol_set.symbols,
    composition_rigidity=0.85,
    component_groups=all_components,
    instances_per_component=component_instance_counts,
    component_residency_factor=0.75,
    component_repetition_penalty=0.75,
    composition_connectivity_factor=0.75,
    instance_cohesion_randomness=0.2,
    end_of_component_symbols={ ".", "!", "?" },
    symbol_cohesion=symbol_set.symbol_cohesion,
    max_search_time=60
)

m.draw_graph( 
    output_dir=".",
    engine='dot',
    color_nodes_by="cm",         # component instance
    color_borders_by="isoclass", # equivalent to "g" for group (in this case)
    color_edges_by="category",   # other option is "symbol"
    symbol_categories=symbol_set.symbol_categories,
    no_labels=False,
    symbols_only=True,
    show=False,
    format="svg"
)

data_dir = Path( "data" ) / "structured_composition"
data_dir.mkdir( exist_ok=True )

m.generate_data(
    file_prefix=data_dir/"train",
    n_gen=4_000_000_000
)

m.generate_data(
    file_prefix=data_dir/"test",
    n_gen=40_000_000, 
    include_states=True,
    isomorphic_shifts=set( range( 1, symbols_per_category ) )
)

Toy Transformers

A-Machine has utilities for generating and training toy transformer models with a matching vocabulary and compatable training configurations.

The model class that is used is GraniteMoeHybridForCausalLM from transformers, which is the basis for IBM Granite 4.0 models like granite-4.0-h-350m and granite-4.0-350m. The rational for using this model class, is that it optionally supports mixture-of-experts (MoE) and Mamba layers, which means we can extend our analysis pipeline to experiment with Mamba and MoE eventually support subsequent, while we can also begin with a pure attention configuration which is simpler and more compatable with existing analys tools. Currently, only attention variants have been tested.

And example, starting from generating an HMM and training data then generating and training an ensemble of toy models:

from pathlib import Path
import json
import subprocess
import tempfile
from pathlib import Path
import amachine as am
import sys

experiment_id = "unique_iso_17"
data_dir = Path( "data" ) / experiment_id
experiment_dir = Path( "experiments" ) / experiment_id

data_dir.mkdir( exist_ok=True )
experiment_dir.mkdir( exist_ok=True )

am.srand_global(42)

m = am.unique_isomorphic(
    isoclass_name="i0",
    n_machines=4,
    n_states=17, 
    n_base_symbols=3,
    connectedness=0.7,
    randomness=0.35,
    mode_residency_factor=0.75
)

print( "Drawing a-machine graph, and saving the a-machine configuration with complexity." )

m.draw_graph( output_dir=experiment_dir, show=False )

m.save_config( 
    output_dir=experiment_dir,
    with_complexity=True, 
    with_non_trivial_complexity=True 
)

print( "Generating training data." )

m.generate_data(
    file_prefix=data_dir/"train",
    n_gen=2_000_000_000, 
    include_states=False
)

print( "Generating test data with isomorphic shift." )

m.generate_data(
    file_prefix=data_dir/"test",
    n_gen=200_000_000, 
    include_states=True,
    isomorphic_shifts={1}
)

ensemble = [
    {"width": 768, "depth": 16},
    {"width": 768, "depth":  8},
    {"width": 768, "depth":  2}
    {"width": 384, "depth": 16},
    {"width": 384, "depth":  8},
    {"width": 384, "depth":  2}
    {"width": 128, "depth": 16},
    {"width": 128, "depth":  8},
    {"width": 128, "depth":  2}
    {"width":  64, "depth": 16},
    {"width":  64, "depth":  8},
    {"width":  64, "depth":  2}
]

print( "Generating ensemble of toy models." )

manifest = am.transformers.generate_ensemble(
    output_dir=experiment_dir/"models/",
    core_params=ensemble,
    width_multiplier=1.0,
    depth_multiplier=1.0,
    head_dim=64,
    seq_len=1024
)

print( "Training the models." )

for model in manifest["models"] :

    model_dir = experiment_dir / "models" / model["name"]
    checkpoint_directory = model_dir / "checkpoints"
    checkpoint_directory.mkdir(parents=True, exist_ok=True)

    config_dict = {
        "data"           : str(data_dir / "train.parquet"),
        "eval_data"      : str(data_dir / "test.parquet"),
        "metadata"       : str(data_dir / "train.json"),
        "model_dir"      : str(model_dir),
        "output_dir"     : str(checkpoint_directory),
        "no_grad_ckpt"   : True,
        "seq_len"        : 1024,
        "compile"        : True,
        "batch_size"     : 32,
        "steps"          : 2800,
        "log_every"      : 10,
        "eval_every"     : 400,
        "eval_steps"     : 10,
        "save_every"     : 200,
        "aim_repo"       : ".",
        "experiment"     : experiment_id 
    }

    with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as temp_file:
        json.dump(config_dict, temp_file, indent=2)
        temp_config_path = temp_file.name

    try:
        print( f"Training model: {model['name']} ..." )
        subprocess.run( 
            [ sys.executable, "-m", "amachine.am_transformers.am_train", temp_config_path ],
            check=True
        )
        print( f"Done training {model['name']}.
" )

    except subprocess.CalledProcessError as e:
        print( f"Training failed with exit code {e.returncode}" )
        break
    finally:
        Path( temp_config_path ).unlink (missing_ok=True )

print( "Finally done!" )

  1. Crutchfield, "The calculi of emergence: computation, dynamics and induction.", 1994. https://csc.ucdavis.edu/~cmg/papers/CalcEmerg.pdf ↩

  2. Shalizi, Rohilla, and Crutchfield. "Computational mechanics: Pattern and prediction, structure and simplicity.", 2001. https://arxiv.org/abs/cond-mat/9907176 ↩

  3. Blackwell, David Harold. "The entropy of functions of finite-state Markov chains.", 1959. ↩

  4. Jurgens & Crutchfield "Shannon Entropy Rate of Hidden Markov Processes", 2021. https://link.springer.com/article/10.1007/s10955-021-02769-3 ↩

  5. Shai et al., "Transformers Represent Belief State Geometry in their Residual Stream", 2024. https://arxiv.org/abs/2405.15943 ↩

  6. Crutchfield, "Exact complexity: The spectral decomposition of intrinsic computation", 2016. https://csc.ucdavis.edu/~cmg/papers/ec.pdf ↩

  1"""
  2# What is A-Machine?
  3
  4A-Machine is a work in progress library for constructing epsilon-machines[^1] and other stochastic models for generating structured symbol sequences with ground truth causal structure and information-theoretic complexity. The intended use case is the study of neural network learning dynamics and internal representations.
  5
  6# Quickstart
  7
  8## Installation
  9
 10```bash
 11# CPU only
 12pip install a-machine
 13
 14# With GPU support (requires CUDA 13)
 15pip install "a-machine[cuda]" --extra-index-url https://pypi.nvidia.com
 16```
 17
 18##Example
 19
 20[More examples](https://gitlab.com/tneuroth/a-machine/-/tree/main/examples) are available at the gitlab repo.
 21
 22```python
 23import amachine as am
 24
 25am.srand_global( 42 )
 26
 27m = am.random_machine( 
 28    n_states=23, 
 29    symbols=[ '0', '1' ],
 30    connectedness=0.35,
 31    randomness=0.25,
 32    # ensure_strongly_connected=True,
 33    # ensure_minimal=True 
 34)
 35
 36# Collapse to the largest subgraph
 37m.collapse_to_largest_strongly_connected_subgraph()
 38
 39# In case it is not minimal
 40m.minimize()
 41
 42# Entropy rate, statistical complexity
 43print( f"h_mu : {m.h_mu()}" )
 44print( f"C_mu : {m.C_mu()}" )
 45
 46# Display the epislon machine
 47m.draw_graph()
 48```
 49
 50<img src="resources/am_graph.png" alt="AM Graph" style="width: 60%; margin-left: 20%;">
 51
 52Note that `ensure_strongly_connected` will guarantee the result is strongly connected and has `n_states` before minimization, but `ensure_minimal` will minimize the result and potentially merge states. They each default to `False`.
 53
 54# HMMs and Epsilon Machines
 55
 56An $\\epsilon$-machine is the minimal, optimally predictive, unifilar, hidden Markov model of a stationary stochastic process. A large body of theory linking them to complex dynamical systems, information theory, and physics, has been developed by James P. Crutchfield and collaborators under the name computational mechanics [^1][^2].
 57
 58A-Machine's [HMM implementation](amachine/am_hmm.html#HMM) is designed primarily for constructing $\\epsilon$-machines to generate training data with interesting ground truth theoretical properties. This should be useful for making informed hypotheses about, designing experiments for, and interpreting the learning dynamics and internal representations of neural networks.
 59
 60More expressive finite generative models, such as non-unifilar HMMs or probabilistic context-free grammars, can represent a broader class of languages than a finite $\\epsilon$-machine, but may lack the theoretical tractability that makes $\\epsilon$-machines particularly analytically useful. Thus, utilizing $\\epsilon$-machines instead of alternatives, may involve a tradeoff between expressiveness and interpretability.
 61
 62## Complexity Measures
 63
 64Complexity measures implemented within the HMM class include:
 65
 66* [statistical complexity $C_\\mu$](amachine/am_hmm.html#HMM.C_mu),
 67* [entropy rate $h_{\\mu}$](amachine/am_hmm.html#HMM.h_mu),
 68* [single symbol entropy H(1)](amachine/am_hmm.html#HMM.H_1), 
 69* [anticipated information $\\rho_{\\mu}$](amachine/am_hmm.html#HMM.rho_mu),
 70* [Excess entropy $\\mathbf{E}$](amachine/am_hmm.html#HMM.E),
 71* [Syncronization information $\\mathbf{S}$](amachine/am_hmm.html#HMM.S),
 72* [Transient information $\\mathbf{T}$](amachine/am_hmm.html#HMM.S),
 73* [Crypticity $\\chi$](amachine/am_hmm.html#HMM.chi), and
 74* [Block convergence measures $\\mathbf{S}(L), \\mathbf{T}(L), \\mathcal{H}(L),$ and $h_{\\mu}(L)$](amachine/am_hmm.html#HMM.block_convergence).
 75
 76Citations, definitions, and interpretations are in their individual documentation sections linked above.
 77
 78## Isomorphic Construction
 79
 80A few ways to construct $\\epsilon$-machines are impemented so far:
 81
 82* [random_machine](amachine/am_create/am_random_machine.html), which generates a random `HMM` that can be reduced to an $\\epsilon$-machine,
 83
 84* [star_join](amachine/am_create/am_star_join.html), which joins a list of machines to a central hub,
 85
 86* [isomorphic_to](amachine/am_create/am_isomorphic_to.html), which creates a copy of an existing machine with new labels,
 87
 88* [cyclical_isomorphic_rotation](amachine/am_create/am_cyclical_isomorphic_rotation.html), which creates and star joins $n$ isomorphic machines, where $n$ is the number of symbols, and the $i^{th}$ machine's state $j$ emits the $k^{th}$ symbol in the alphabet, then the ${i+1}^{th}$ machine's state $j$ will emit symbol $k+1 \\bmod n$, and
 89
 90* [unique_isomorphic](amachine/am_create/am_unique_isomorphic.html), which creates and star joins `n_machines` isomorphic machines, where `n_machines` is a parameter, and each has its own alphabet.
 91
 92* [random_structured](amachine/am_create/am_random_structured.html), which is similar to `random` except that it accepts a `[StructuredSymbolSet](amachine/am_structured_symbol_set.html#StructuredSymbolSet)`, that contains "cohesion" rules and symbol categories which promote structure in the placement of edge symbols, and provide a categorical basis for isomorphic rotation, 
 93
 94* [isomorphic_to_with_category_rotation](amachine/am_create/am_isomorphic_to_with_category_rotation.html), which creates a copy of an existing machine with symbols rotated by a shift within select categories given a symbol categorization,
 95
 96* [full_structured_isomorphic_rotation](amachine/am_create/am_full_structured_isomorphic_rotation.html), which produces the full rotation on a `random_structured` `HMM` using `isomorphic_to_with_category_rotation` and optionally `star_join`'s them, otherwise returns the list of generated `HMM`'s,
 97
 98* [structured_composition](amachine/am_create/am_structured_composition.html), which accepts groups of `HMM`'s, with variable multiplicity, optionally `end_of_component` symbols, and various structural and cohesion synthesis parameters, and joins according to the generated cohesion rules.
 99
100* [random_structured_composition](amachine/am_create/am_random_structured_composition.html), which is generates components and composes them using `structured_composition`.
101
102A small example is shown below, that generates 2 random `full_structured_isomorphic_rotation`'s and then composes them. Since there are 2 symbols per-category, the 2 full rotations produce 2 isomorphic groups of size 2, thus 2 unique subgraphs, and with `instances_per_component=2`, there will be 8 total subgraph instances. 
103
104```python
105import amachine as am
106
107am.srand_global(42)
108
109n_symbol_categories=2
110symbols_per_category=2
111
112symbol_set = am.StructuredSymbolSet.generate(
113    n_categories=n_symbol_categories,
114    symbols_per_category=[symbols_per_category]*n_symbol_categories,
115    rigidity=0.75,
116    within_category_variability=0.1,
117    category_repetition_penalty=0.75,
118    symbol_repetition_penalty=0.75
119)
120
121instances_per_component = 2
122component_target_sizes = [ 3, 5 ]
123all_components : dict[str,list[am.HMM] ] = {}
124component_instance_counts : dict[str,list[int]] = {}
125
126for iso_idx, target_size in enumerate( component_target_sizes ) :
127
128    iso_id = f"{iso_idx}" 
129
130    all_components[ iso_id ] = am.full_structured_isomorphic_rotation(
131        isoclass_name=f"{iso_idx}",
132        n_states=target_size,
133        symbol_set=symbol_set,
134        connectedness=0.68,
135        randomness=0.25,
136        star_joined=False,
137        max_search_time=60
138    )
139
140    component_instance_counts[ iso_id ] = [
141        instances_per_component for _ in all_components[ f"{iso_idx}" ] 
142    ]
143
144m = am.structured_composition(
145    composition_id="0",
146    core_alphabet=symbol_set.symbols,
147    composition_rigidity=0.85,
148    component_groups=all_components,
149    instances_per_component=component_instance_counts,
150    component_residency_factor=0.75,
151    component_repetition_penalty=0.75,
152    composition_connectivity_factor=0.75,
153    instance_cohesion_randomness=0.0,
154    end_of_component_symbols={ "!", "?" },
155    symbol_cohesion=symbol_set.symbol_cohesion,
156    max_search_time=60
157)
158
159m.draw_graph( 
160    output_dir=".",
161    engine='dot',
162    color_nodes_by="cm",
163    color_borders_by="isoclass",
164    color_edges_by="category",
165    symbol_categories=symbol_set.symbol_categories,
166    no_labels=True,
167    show=True,
168    format="svg"
169)
170```
171
172The result is shown below. Nodes with the same fill color are from idential sub-graphs that appear more than once because of the `instances_per_component` parameter. The node border color represents the `isoclass` and in this case euqivalently the `group` the state belongs to (red and blue), which includes all sub-graphs that are isomorphic with eachother (i.e., same based `HMM` with category rotations). The edge color represents the category of the emitted symbols, with black being the `end_of_component` symbols.
173
174<img src="resources/structured_composition.png" alt="AM Graph" style="width: 100%; margin-left: 0%;">
175
176A few things to note: 
177
178[StructuredSymbolSet.generate](amachine/am_structured_symbol_set.html) automatically selects symbols from among digits 0-9, acii lowercase and uppercase letters, and uppercase and lowecase unicode Greek symbols. What remains that is part of the vocabulary of build in toy transformer models, and can be used for `end_of_component_symbols`, is ascii punctuation characters, brackets, math, and special characters, as well as the unicode geometry block. See [Vocabulary](amachine/am_vocabulary.html) for the full set of characters, which are one-to-one with the vocabulary of the toy models.
179
180There will always only be one entry node and one exit node per component, and since the compositon must be unifilar, there can be at most `len(end_of_component_symbols)` outgoing edges from each component. Thus for more connectedness between components, in addition to using `composition_connectivity_factor`, you must choose enough end of component symbols. If `end_of_component_symbols` is omitted None, then components will be connected with symbols from the standard alphabet.
181
182The effect of using end of component symbols will be that a neural network training on the data will have hints that help it better synchronize. Other not-yet-implemented options include unique entry symbols per-isogroup, per-component, or per-component instance, which would allow the neural network to instantly partially or fully syncronize upon each component to component transition. This could be interesting to control for the sake of encouraging or helping the model be able to utilize redundencies and generalize over isomorphic groups. 
183
184A large case with 2 `full_structured_isomorphic_rotation`'s, 5 symbols per-category, and instances per-component is shown below. Note that the total number of symbols asides from the end of component symbols, is `n_categories*symbols_per_category`, and since the `connectedness` parameter in `full_structured_isomorphic_rotation` ranges from 0 to 1, and operates on the number of possible edges per-state, you can expect `around connectedness*n_categories*symbols_per_category` edges per-state.
185
186```python
187import amachine as am
188
189am.srand_global(42)
190
191n_symbol_categories=4
192symbols_per_category=5
193
194symbol_set = am.StructuredSymbolSet.generate(
195    n_categories=n_symbol_categories,
196    symbols_per_category=[ symbols_per_category ]*n_symbol_categories,
197    rigidity=0.75,
198    within_category_variability=0.1,
199    category_repetition_penalty=0.75,
200    symbol_repetition_penalty=0.75
201)
202
203instances_per_component = 2
204component_target_sizes = [ 9, 11 ]
205all_components : dict[str,list[am.HMM] ] = {}
206component_instance_counts : dict[str,list[int]] = {}
207
208for iso_idx, target_size in enumerate( component_target_sizes ) :
209
210    iso_id = f"{iso_idx}" 
211
212    all_components[ iso_id ] = am.full_structured_isomorphic_rotation(
213        isoclass_name=f"{iso_idx}",
214        n_states=target_size,
215        symbol_set=symbol_set,
216        connectedness=0.15,
217        randomness=0.25,
218        star_joined=False,
219        max_search_time=60
220    )
221
222    component_instance_counts[ iso_id ] = [
223        instances_per_component for _ in all_components[ f"{iso_idx}" ] 
224    ]
225
226m = am.structured_composition(
227    composition_id="0",
228    core_alphabet=symbol_set.symbols,
229    composition_rigidity=0.85,
230    component_groups=all_components,
231    instances_per_component=component_instance_counts,
232    component_residency_factor=0.75,
233    component_repetition_penalty=0.75,
234    composition_connectivity_factor=0.75,
235    instance_cohesion_randomness=0.0,
236    end_of_component_symbols={ ".", "!", "?" },
237    symbol_cohesion=symbol_set.symbol_cohesion,
238    max_search_time=60
239)
240
241m.draw_graph( 
242    output_dir=".",
243    engine='dot',
244    color_nodes_by="cm",
245    color_borders_by="isoclass",
246    color_edges_by="category",
247    symbol_categories=symbol_set.symbol_categories,
248    no_labels=True,
249    show=False,
250    format="svg"
251)
252```
253
254The result is shown below.
255
256<img src="resources/structured_composition_large.png" alt="AM Graph" style="width: 100%; margin-left: 0%;">
257
258## Mixed State Presentation
259
260The mixed states presentation construction [^3][^4] tracks beliefs about which intrinsic state the known $\\epsilon$-machicine is in as an observer attempts to syncronize with the $\\epsilon$ machine. MSP construction can be useful for analyzing learning dynamics, because this is something like what a neural network must learn to do to utilize context during inference. Experiments correlating HMM belief states and transformer representations have been done by Shai et al. [^5].
261
262The MSP can explode in terms of the number of states that are produced, or even be inifite for non-synchronizable processes. The belief states transitions also may produce fractal-like patterns in the probability simplex.
263
264A-Machine implements [MSP construction](amachine/am_machine.html#get_msp) from a given $\\epsilon$ machine. Since the MSP may have too many states to compute, a cap on the max number of states is imposed, and when the cap is reached, the MSP is closed by mapping the unresolved frontier of belief states back to the nearest existing belief states. When this happens, the final MSP is only an approximation.
265
266The MSP also yields closed form expressions for a range of complexity measures [^6], such as $\\mathbf{E}, \\mathbf{S}$, and $\\mathbf{T}$, which can be solved for using spectral decompositon. A-machine implements this approach to computing these complexity measures, in addition to the alternative, block entropy convergence. When the MSP construction excedes the max states cap, comparing the measures computed from its approximation with those approximated by block entropy convergence is useful for corroboration and gaining additional insight into the synronization dynamics.
267
268## Block Entropy Convergence
269
270As mentioned, [block entropy convergence](amachine/am_hmm.html#HMM.block_convergence) supports estimation of a range of complexity measures, and allows you to observe the convergence in uncertainty as a function of $L$ (after seeing all blocks of length $L$). A-machine implements this as a [C++ extension](https://gitlab.com/tneuroth/a-machine/-/blob/main/src/amachine/am_fast/am_fast.cpp).
271
272You also plot block entropy curves and block measures, by calling [`amachine.HMM.draw_block_measure_curves`](amachine/am_hmmhtml#HMM.draw_block_measure_curves), and [`amachine.HMM.draw_block_entropy_curve`](amachine/am_hmm.html#HMMdraw_block_entropy_curve). 
273
274<img src="resources/curves.png" alt="block measures plots" style="width: 100%; margin-left: 0%;">
275
276$\\mathcal{H}(L)$ represents the average state uncertainty after setting all subsequences in the language of the machine of length $L$. And thus, you could utilize this measure to estimate how much training data and context a neural network may require at minimum to synchronize with the HMM. It often falls off very quickly, but also often has a very long tail depending on the complexity of the model. For example, with the larger structured composition in the previous example:
277
278| $L$ | $\\mathcal{H}(L)$ |
279| :--- | :--- |
280|   1 | 3.64153  |
281| 100 | 0.202524 |
282| 200 | 0.067278 |
283| 300 | 0.022828 |
284| 400 | 0.007783 |
285| 500 | 0.002656 |
286| 600 | 0.000905 |
287| 700 | 0.000308 |
288| 800 | 0.000104 |
289| 900 | 0.000035 |
290| 1000 | 0.000012 |,
291
292while for the first random machine example,
293
294| $L$ | $\\mathcal{H}(L)$ |
295| :--- | :--- |
296|   1 | 2.17641  |
297|   5 | 0.609825 |
298|  10 | 0.126823 |
299|  15 | 0.026362 |
300|  20 | 0.005412 |
301|  25 | 0.001112 |
302|  30 | 0.000221 |
303|  35 | 0.000045 |
304|  40 | 0.000009 |
305
306## Data Generation
307
308[Data generation](amachine/am_hmm.html#HMM.generate_data) generates stochastic sequences from a given machine, optionally along with the sequence of hidden states that were traversed, and also optionally with an additional symbol sequence with what we are calling [isomorphic_shift](amachine/am_hmm.html#HMM.isomorphic_shift) applied. This changes the symbols that were emited by a state, for symbols that would have been emitted by alternative isomorphic states (should they exist).
309
310```python
311import amachine as am
312
313am.srand_global(42)
314
315m = am.random_machine( 
316    n_states=11, 
317    symbols=[ '0', '1', '2' ],
318    connectedness=0.65,
319    randomness=0.37 )
320
321# Collapse to the largest recurrent subgraph
322m.collapse_to_largest_strongly_connected_subgraph()
323
324# Minimize the machine -> epsilon-machine.
325m.minimize()
326
327# Create an isomorphic machine with new labels
328m_iso = am.isomorphic_to( m, alphabet=[ '3', '4', '5' ] )
329
330m_iso.isoclass = 0
331m.isoclass     = 0
332
333for j, state in enumerate( m.states ) : 
334    m.states[ j ].add_isomorph( m_iso.states[ j ].name )
335    m_iso.states[ j ].add_isomorph( m.states[ j ].name )
336    
337# Join them together
338m_star = am.star_join(
339    exit_symbol='x', 
340    enter_symbols=['a','b'],
341    machines=[ m, m_iso ],
342    mode_residency_factor=0.75 
343)
344
345m_star.draw_graph()
346
347m_star.generate_data(
348    file_prefix="./data/iso_012-345", # saves as iso_012-345.parqet
349    n_gen=2_000_000_000, # 2 billion symbols    
350    include_states=True,
351    isomorphic_shifts={1}
352)
353```
354
355<img src="resources/iso.png" alt="iso machine" style="width: 100%; margin-left: 0%;">
356
357The two sequences will be aligned, and with identical local and global statistics. Thus, you can train on one of them, and then analyze activation patterns and residuals on both of them in parallel during inference, to study if and how the neural network learned polysemantic representations that exploit the structural redundancies.
358
359Revisiting the large structured composition example, you can apply multiple shifts. In this case, since we have isomorphic groups over category rotations, with `symbols_per_category` symbols in each category, generate aligned rotated sequences for shifts in { 1, 2, ..., `symbols_per_category`-1 }.
360
361```python
362from pathlib import Path
363import amachine as am
364
365am.srand_global(42)
366
367n_symbol_categories=4
368symbols_per_category=5
369instances_per_component = 2
370component_target_sizes = [ 9, 11 ]
371
372symbol_set = am.StructuredSymbolSet.generate(
373    n_categories=n_symbol_categories,
374    symbols_per_category=[ symbols_per_category ]*n_symbol_categories,
375    rigidity=0.75,
376    within_category_variability=0.1,
377    category_repetition_penalty=0.75,
378    symbol_repetition_penalty=0.75
379)
380
381all_components : dict[str,list[am.HMM] ] = {}
382component_instance_counts : dict[str,list[int]] = {}
383
384for iso_idx, target_size in enumerate( component_target_sizes ) :
385
386    iso_id = f"{iso_idx}" 
387
388    all_components[ iso_id ] = am.full_structured_isomorphic_rotation(
389        isoclass_name=f"{iso_idx}",
390        n_states=target_size,
391        symbol_set=symbol_set,
392        connectedness=0.15,
393        randomness=0.25,
394        star_joined=False,
395        max_search_time=60,
396    )
397
398    component_instance_counts[ iso_id ] = [
399        instances_per_component for _ in all_components[ iso_id ] 
400    ]
401
402m = am.structured_composition(
403    composition_id="0",
404    core_alphabet=symbol_set.symbols,
405    composition_rigidity=0.85,
406    component_groups=all_components,
407    instances_per_component=component_instance_counts,
408    component_residency_factor=0.75,
409    component_repetition_penalty=0.75,
410    composition_connectivity_factor=0.75,
411    instance_cohesion_randomness=0.2,
412    end_of_component_symbols={ ".", "!", "?" },
413    symbol_cohesion=symbol_set.symbol_cohesion,
414    max_search_time=60
415)
416
417m.draw_graph( 
418    output_dir=".",
419    engine='dot',
420    color_nodes_by="cm",         # component instance
421    color_borders_by="isoclass", # equivalent to "g" for group (in this case)
422    color_edges_by="category",   # other option is "symbol"
423    symbol_categories=symbol_set.symbol_categories,
424    no_labels=False,
425    symbols_only=True,
426    show=False,
427    format="svg"
428)
429
430data_dir = Path( "data" ) / "structured_composition"
431data_dir.mkdir( exist_ok=True )
432
433m.generate_data(
434    file_prefix=data_dir/"train",
435    n_gen=4_000_000_000
436)
437
438m.generate_data(
439    file_prefix=data_dir/"test",
440    n_gen=40_000_000, 
441    include_states=True,
442    isomorphic_shifts=set( range( 1, symbols_per_category ) )
443)
444```
445
446## Toy Transformers
447
448A-Machine has [utilities](amachine/am_transformers.html) for generating and training toy transformer models with a matching vocabulary and compatable training configurations.
449
450The model class that is used is [GraniteMoeHybridForCausalLM](https://huggingface.co/docs/transformers/model_doc/granitemoehybrid ) from transformers, which is the basis for IBM Granite 4.0 models like [granite-4.0-h-350m](https://huggingface.co/ibm-granite/granite-4.0-h-350m) and [granite-4.0-350m](https://huggingface.co/ibm-granite/granite-4.0-350m). The rational for using this model class, is that it optionally supports mixture-of-experts (MoE) and Mamba layers, which means we can extend our analysis pipeline to experiment with Mamba and MoE eventually support subsequent, while we can also begin with a pure attention configuration which is simpler and more compatable with existing analys tools. Currently, only attention variants have been tested.
451
452And example, starting from generating an HMM and training data then generating and training an ensemble of toy models:
453
454```python
455from pathlib import Path
456import json
457import subprocess
458import tempfile
459from pathlib import Path
460import amachine as am
461import sys
462
463experiment_id = "unique_iso_17"
464data_dir = Path( "data" ) / experiment_id
465experiment_dir = Path( "experiments" ) / experiment_id
466
467data_dir.mkdir( exist_ok=True )
468experiment_dir.mkdir( exist_ok=True )
469
470am.srand_global(42)
471
472m = am.unique_isomorphic(
473    isoclass_name="i0",
474    n_machines=4,
475    n_states=17, 
476    n_base_symbols=3,
477    connectedness=0.7,
478    randomness=0.35,
479    mode_residency_factor=0.75
480)
481
482print( "Drawing a-machine graph, and saving the a-machine configuration with complexity." )
483
484m.draw_graph( output_dir=experiment_dir, show=False )
485
486m.save_config( 
487    output_dir=experiment_dir,
488    with_complexity=True, 
489    with_non_trivial_complexity=True 
490)
491
492print( "Generating training data." )
493
494m.generate_data(
495    file_prefix=data_dir/"train",
496    n_gen=2_000_000_000, 
497    include_states=False
498)
499
500print( "Generating test data with isomorphic shift." )
501
502m.generate_data(
503    file_prefix=data_dir/"test",
504    n_gen=200_000_000, 
505    include_states=True,
506    isomorphic_shifts={1}
507)
508
509ensemble = [
510    {"width": 768, "depth": 16},
511    {"width": 768, "depth":  8},
512    {"width": 768, "depth":  2}
513    {"width": 384, "depth": 16},
514    {"width": 384, "depth":  8},
515    {"width": 384, "depth":  2}
516    {"width": 128, "depth": 16},
517    {"width": 128, "depth":  8},
518    {"width": 128, "depth":  2}
519    {"width":  64, "depth": 16},
520    {"width":  64, "depth":  8},
521    {"width":  64, "depth":  2}
522]
523
524print( "Generating ensemble of toy models." )
525
526manifest = am.transformers.generate_ensemble(
527    output_dir=experiment_dir/"models/",
528    core_params=ensemble,
529    width_multiplier=1.0,
530    depth_multiplier=1.0,
531    head_dim=64,
532    seq_len=1024
533)
534
535print( "Training the models." )
536
537for model in manifest["models"] :
538
539    model_dir = experiment_dir / "models" / model["name"]
540    checkpoint_directory = model_dir / "checkpoints"
541    checkpoint_directory.mkdir(parents=True, exist_ok=True)
542
543    config_dict = {
544        "data"           : str(data_dir / "train.parquet"),
545        "eval_data"      : str(data_dir / "test.parquet"),
546        "metadata"       : str(data_dir / "train.json"),
547        "model_dir"      : str(model_dir),
548        "output_dir"     : str(checkpoint_directory),
549        "no_grad_ckpt"   : True,
550        "seq_len"        : 1024,
551        "compile"        : True,
552        "batch_size"     : 32,
553        "steps"          : 2800,
554        "log_every"      : 10,
555        "eval_every"     : 400,
556        "eval_steps"     : 10,
557        "save_every"     : 200,
558        "aim_repo"       : ".",
559        "experiment"     : experiment_id 
560    }
561
562    with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as temp_file:
563        json.dump(config_dict, temp_file, indent=2)
564        temp_config_path = temp_file.name
565
566    try:
567        print( f"Training model: {model['name']} ..." )
568        subprocess.run( 
569            [ sys.executable, "-m", "amachine.am_transformers.am_train", temp_config_path ],
570            check=True
571        )
572        print( f"Done training {model['name']}.\n" )
573        
574    except subprocess.CalledProcessError as e:
575        print( f"Training failed with exit code {e.returncode}" )
576        break
577    finally:
578        Path( temp_config_path ).unlink (missing_ok=True )
579
580print( "Finally done!" )
581```
582
583[^1]: Crutchfield, "The calculi of emergence: computation, dynamics and induction.", 1994.
584    <https://csc.ucdavis.edu/~cmg/papers/CalcEmerg.pdf>
585
586[^2]: Shalizi, Rohilla, and Crutchfield. "Computational mechanics: Pattern and prediction, structure and simplicity.", 2001.
587    <https://arxiv.org/abs/cond-mat/9907176>
588
589[^3]: Blackwell, David Harold. "The entropy of functions of finite-state Markov chains.", 1959.
590
591[^4]: Jurgens & Crutchfield "Shannon Entropy Rate of Hidden Markov Processes", 2021.
592    <https://link.springer.com/article/10.1007/s10955-021-02769-3>
593
594[^5]: Shai et al., "Transformers Represent Belief State Geometry in their Residual Stream", 2024. 
595    <https://arxiv.org/abs/2405.15943>
596
597[^6]: Crutchfield, "Exact complexity: The spectral decomposition of intrinsic computation", 2016.
598    <https://csc.ucdavis.edu/~cmg/papers/ec.pdf>
599
600"""
601__version__ = "0.2.1"
602
603from types import SimpleNamespace
604
605from .am_transformers.am_trainer import Trainer
606from .am_transformers.am_training_config import TrainingConfig
607from .am_transformers.am_models import generate_ensemble
608from .am_transformers.am_gpt2_generator import  generate_gpt2_ensemble
609
610transformers = SimpleNamespace(
611    generate_ensemble=generate_ensemble,
612    generate_gpt2_ensemble=generate_gpt2_ensemble,
613    Trainer=Trainer,
614    TrainingConfig=TrainingConfig
615)
616
617del generate_ensemble
618del Trainer
619del TrainingConfig
620del generate_gpt2_ensemble
621
622#----------------------------------------------------
623
624from .am_create import (
625    random_machine,
626    star_composition,
627    isomorphic_to,
628    cyclic_orbit_cluster,
629    disjoint_isomorphic,
630    random_structured,
631    isomorphic_to_with_category_permutations,
632    isomorphic_to_with_superstate_permutations,
633    structured_isomorphic_permutations,
634    structured_isomorphic_permutations_of,
635    structured_composition,
636    random_structured_cyclic_composition
637)
638
639from .am_visualization import (
640    draw_block_entropy_curve,
641    draw_block_measure_curves,
642    draw_graph
643)
644
645from .am_structured_symbol_set import StructuredSymbolSet
646from .am_hmm import HMM
647from .am_random import srand_global
648from .am_vocabulary import Vocabulary
transformers = namespace(generate_ensemble=<function generate_ensemble>, generate_gpt2_ensemble=<function generate_gpt2_ensemble>, Trainer=<class 'amachine.am_transformers.am_trainer.Trainer'>, TrainingConfig=<class 'amachine.am_transformers.am_training_config.TrainingConfig'>)