Metadata-Version: 2.4
Name: sequifier
Version: 2.2.0.0
Summary: Train a transformer model with the command line
Author-email: Leon Luithlen <leontimnaluithlen@gmail.com>
License: BSD 3-Clause
Project-URL: Homepage, https://github.com/0xideas/sequifier
Project-URL: Repository, https://github.com/0xideas/sequifier
Keywords: transformer,sequence classification,machine learning,sequence,sequence modelling,nlp,language,language modelling,torch,pytorch
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: <4.0,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: polars<2.0,>=1.31.0
Requires-Dist: numpy>=1.23
Requires-Dist: torch>=2.5
Requires-Dist: torch_optimizer<0.4.0,>=0.3.0
Requires-Dist: onnx<2.0.0,>=1.15.0
Requires-Dist: onnxruntime>=1.18
Requires-Dist: onnxscript>=0.5.4
Requires-Dist: optuna>=2.10.0
Requires-Dist: polars<2.0.0,>=1.0.0
Requires-Dist: pyyaml<7.0,>=6.0
Requires-Dist: pydantic<3.0,>=2.0
Requires-Dist: pyarrow>=16.1
Requires-Dist: fastparquet<2025.0.0,>=2024.2.0
Requires-Dist: beartype<0.19.0,>=0.18.5
Requires-Dist: psutil<8.0.0,>=7.0.0
Requires-Dist: loguru<1.0.0,>=0.7.3
Requires-Dist: csvkit<2.0,>=1.0
Requires-Dist: plotly>=4.0.0
Provides-Extra: test
Requires-Dist: pytest<8.0,>=7.2; extra == "test"
Requires-Dist: plotly<7.0,>=6.0; extra == "test"
Dynamic: license-file

<p align="center">
  <img src="./design/sequifier.png" alt="Sequifier" width="720">
</p>

# What is sequifier?

Sequifier is the short, efficient, scalable path from tabular sequences to your own transformer model.

It offers three core commands `preprocess`, `train` and `infer`, each of them configurable, fully parallelised and well-tested.

They enable you to go from multivariate sequence data to a model that ingests such data, and emits either (a) this data, (b) a subset, (c) other variables or (d) embeddings.

If input and target variables are the same, it supports full autoregressive inference.

Through configuration, most modern architectural variants such as RoPE, GQA, SwiGLU, and RMSNorm are supported. The ambition is to follow the frontier when industrial-grade implementations become available.

The process looks like this:

<p align="center">
  <img src="./design/sequifier-illustration.png" alt="Sequifier preprocessing, training, and inference workflow" width="900">
</p>

## Examples from the literature

Multivariate tabular transformers have *many* applications.

These independent references illustrate some of them; inclusion does not imply that the cited work used or is affiliated with sequifier.

**Finance**

- Transaction Foundation Models ([Overview](https://sistemalabs.com/blog/shaping-the-problem), [EWE-1](https://sistemalabs.com/blog/introducing-ewe-1))
- Order Book Transformers ([TransLOB](https://arxiv.org/abs/2003.00130))
- Volatility Forecasting ([paper](https://www.sciencedirect.com/science/article/abs/pii/S0952197624003816))
- Macroeconomic Models ([BISTRO](https://www.bis.org/publications/bistro-general-purpose-oracle-macroeconomic-time-series))

**Health/Bio**

- Health Records ([BEHRT](https://www.nature.com/articles/s41598-020-62922-y), [Med-BERT](https://www.nature.com/articles/s41746-021-00455-y))
- Treatment Outcome Prediction ([G-Transformer](https://arxiv.org/abs/2406.05504))
- Health Trajectory Prediction ([ETHOS](https://www.nature.com/articles/s41746-024-01235-0))
- Glucose Forecasting ([GluForecast](https://arxiv.org/html/2606.18640v1))
- ECG rhythm/arrhythmia monitoring ([Heart Language Model](https://www.nature.com/articles/s41598-024-84270-x))

**Cybersecurity**

- Network Intrusion Detection ([paper](https://arxiv.org/pdf/2309.01070))
- Encrypted Traffic Analysis ([Criss-Cross Traffic Transformer](https://ieeexplore.ieee.org/document/11396346))
- Insider Threat Detection ([paper](https://www.nature.com/articles/s41598-025-12063-x))
- In-Vehicle Intrusion Detection ([paper](https://www.sciencedirect.com/science/article/abs/pii/S0950705124007251))
- IoT Anomaly Detection ([paper](https://www.sciopen.com/article/10.32604/cmc.2024.053765))
- System Logs Anomaly Detection ([DeepEAD](https://ieeexplore.ieee.org/document/10279563))

**Industrial & IoT**

- Remaining Useful Life Estimation ([paper](https://www.sciencedirect.com/science/article/abs/pii/S095219762400633X))
- Fault Diagnosis ([paper](https://www.sciencedirect.com/science/article/abs/pii/S0957582026001485))
- Anomaly Detection for Industrial Control Systems ([paper](https://onlinelibrary.wiley.com/doi/10.1155/2024/5459452))
- Soft Sensing ([Debutanizer](https://www.sciencedirect.com/science/article/abs/pii/S1876107024003249))
- Battery State-of-Health Estimation ([DS-transformer](https://www.nature.com/articles/s41598-026-52202-6))
- Production Line Modelling ([paper](https://www.sciencedirect.com/science/article/pii/S2212827126007791?))

**Agriculture & Environment**

- Soil Moisture Forecasting ([paper](https://link.springer.com/chapter/10.1007/978-3-032-19763-4_18))
- Crop Water Demand ([paper](https://www.mdpi.com/2073-4395/16/16/1600))
- Rainfall–runoff modelling ([paper](https://www.sciencedirect.com/science/article/abs/pii/S0022169422003560))
- Drought Forecasting ([paper](https://www.sciencedirect.com/science/article/abs/pii/S1364815225000787))
- Land Surface Dynamics ([paper](https://www.sciencedirect.com/science/article/abs/pii/S0022169425002446))

**Neuroscience**

- Neural Population Dynamics ([Neural Data Transformer](https://pmc.ncbi.nlm.nih.gov/articles/PMC10541112/))
- Neural + Motor Modelling ([Intracortical Motor Decoder](https://www.biorxiv.org/content/10.1101/2025.02.02.634313v1))
- fMRI State Prediction ([paper](https://arxiv.org/abs/2412.19814))
- Seizure Detection ([BIOT](https://arxiv.org/abs/2305.10351))
- Magnetoencephalography Data ([MEG-GPT](https://arxiv.org/abs/2510.18080))

**Animal Communication**

- Sperm Whales ([whale-gpt](https://github.com/0xideas/whale-gpt), [WhAM](https://github.com/Project-CETI/wham/))
- Bengalese Finches ([paper](https://pmc.ncbi.nlm.nih.gov/articles/PMC8746767/))
- Zebra Finches ([ZF-Aim](https://www.biorxiv.org/content/10.64898/2026.02.12.705387v1))

Sequifier aims to standardise model implementation across these fields, to make them comparable, transfer learnings between domains, and converge on optimal solutions faster.

## Value Proposition

For the individual researcher, sequifier cuts the development time of a model significantly: typically, some preprocessing and the subsequent model evaluation are specific to the modelling problem, but all the steps in between are taken care of.

This enables:

- rapid prototyping on a configurable architecture
- trusted implementation (you can't create bugs inadvertently)
- scaling preprocessing across cores and training across GPUs and nodes
- hyperparameter optimization using Optuna (Bayesian, Random, or Grid search)

## The Six Commands

There are six standalone commands within sequifier: `make`, `preprocess`, `train`, `infer`, `hyperparameter-search`, and `visualize-training`.

| Command | Purpose |
| --- | --- |
| `make` | Create a new sequifier project with config templates. |
| `preprocess` | Convert input data into fixed-length subsequences. |
| `train` | Train a model on preprocessed data. |
| `infer` | Generate predictions, probabilities, or embeddings. |
| `hyperparameter-search` | Use Optuna to find optimal configurations across multiple training runs. |
| `visualize-training` | Generate interactive HTML plots from structured training metrics. |

There are documentation pages for each command, except `make`:

- [preprocess documentation](./documentation/configs/preprocess.md)
- [train documentation](./documentation/configs/train.md)
- [infer documentation](./documentation/configs/infer.md)
- [hyperparameter-search documentation](./documentation/configs/hyperparameter-search.md)
- [visualize-training documentation](./documentation/commands/visualize-training.md)

## Other Materials

To get the full documentation, visit [sequifier.com](https://sequifier.com)

## Structure of a Sequifier Project

Sequifier is designed with a specific folder structure in mind:

```text
YOUR_PROJECT_NAME/
├── configs/
│   ├── preprocess.yaml
│   ├── train.yaml
│   └── infer.yaml
├── data/
│   └── (Place your CSV/Parquet files here)
├── models/
├── checkpoints/
├── outputs/
│   ├── embeddings(?)
│   ├── predictions(?)
│   ├── probabilities(?)
│   └── visualization/
├── logs/
├── state/
└── scripts/
```

The `sequifier` commands should typically be run in the project root.

Within YOUR_PROJECT_NAME, you can also add other folders for additional steps, such as `notebooks` or `scripts` for pre- or postprocessing, and `analysis`, `visualizations` or `evals` for files you generate in other, manual steps.

### Data Transformations in Sequifier

The basic input data format is this:

| sequenceId | itemPosition | column1 | column2 | ... |
| --- | --- | --- | --- | --- |
| 0 | 0 | "high" | 12.3 | ... |
| 0 | 1 | "high" | 10.2 | ... |
| ... | ... | ... | ... | ... |
| 1 | 0 | "medium" | 20.6 | ... |
| ... | ... | ... | ... | ... |

The two columns "sequenceId" and "itemPosition" have to be present, and there must be one or more feature columns.

`sequifier preprocess` splits sequences into subsequences, normalises real variables and maps categorical variables to integers/tokens. The subsequence length is the sum of `window_length` and `max_target_offset`.

| sequenceId | subsequenceId | startItemPosition | leftPadLength | inputCol | [Subsequence Length - 1] | [Subsequence Length - 2] | ... | 0 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 0 | 0 | 0 | column1 | "high" | "high" | ... | "low" |
| 0 | 0 | 0 | 0 | column2 | 12.3 | 10.2 | ... | 14.9 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 1 | 0 | 15 | 0 | column1 | "medium" | "high" | ... | "medium" |
| 1 | 0 | 15 | 0 | column2 | 20.6 | 18.5 | ... | 21.6 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... |

Generative inference returns a row-oriented table with the predicted target
columns plus identifiers for the source sequence and model window:

| sequenceId | subsequenceId | windowStartOffset | itemPosition | column1 | column2 | ... |
| --- | --- | --- | --- | --- | --- | --- |
| 0 | 0 | 0 | 963 | "medium" | 8.9 | ... |
| 0 | 0 | 0 | 964 | "low" | 6.3 | ... |
| ... | ... | ... | ... | ... | ... | ... |
| 1 | 4 | 0 | 732 | "medium" | 14.4 | ... |
| ... | ... | ... | ... | ... | ... | ... |

### Complete Example of Training and Inferring a Transformer Model

Once you have your data in the input format described above, you can train a transformer model in a couple of steps:

1. **Install sequifier.** Create and activate an environment with Python >=3.10, then run:

   ```console
   pip install sequifier
   ```

2. **Create a project.** Generate a project folder with config templates in its `configs` subfolder:

   ```console
   sequifier make YOUR_PROJECT_NAME
   ```

3. **Add your data.** Change into the `YOUR_PROJECT_NAME` folder, create a `data` folder, add your data, and adapt `preprocessing_data_path` in `preprocess.yaml` to point to it.

4. **Preprocess the data.** Run:

   ```console
   sequifier preprocess
   ```

5. **Reference the generated metadata.** Preprocessing outputs metadata at `configs/metadata_configs/[INPUT BASENAME].json`. For a single dataset and part, reference that file from `dataset.part.metadata_config_path` in `train.yaml`; named configurations use `dataset_training.<dataset>.parts.<part>.metadata_config_path`. Inference may still use `preprocessing_data_path` or `metadata_config_path`.

6. **Configure and train the model.** Adapt `train.yaml` with your transformer hyperparameters, then run:

   ```console
   sequifier train
   ```

7. **Configure inference.** Point `model_path` in `infer.yaml` at the default ONNX export. Keep the scaffold's explicit contract, or replace it with `training_config_path` and `dataset`; see the [ONNX/PT trade-offs](./documentation/configs/infer.md#onnx-or-pt).

8. **Run inference.** Run:

   ```console
   sequifier infer
   ```

9. **Find your predictions.** They are written to `[PROJECT ROOT]/outputs/predictions/[EXPORTED_MODEL_BASENAME]/part-000.[FORMAT]`, for example `outputs/predictions/your-model-best-3/part-000.csv`.

## Key Features

### Causal Modelling

Causal transformers are the core architecture in sequifier. Their training objective is to 'predict the next token', or, in the multivariate case, the next value for each target variable.

To train a causal transformer, set `training_objective: causal`. If the target value of interest is further in the future, set `target_offset` to a value higher than 1.

### Autoregressive Inference

Autoregressive inference is allowed when the model is causal, all input variables are target variables, `target_offset` equals 1 and `prediction_length` equals 1. It is enabled by setting `autoregressive: true`, and `generation_steps` to the desired integer value. The number of generation_steps will be generated starting from the *first* complete subsequence in a sequence.

It iteratively predicts future values, by returning predictions at step t-1 as input for generating a prediction at t. Predictions for categorical target variables can be made using argmax or sampling.

For dependencies between multiple outputs at the same time step, the
`autoregressive_transformer` decoder models targets in explicit
`target_columns` order. It supports categorical and real targets, parallel
teacher forcing during training and validation, greedy generation during
inference, composite decoder branches, shared categorical tables, and optional
input/output weight tying. Probabilities from this decoder are conditioned on
the preceding greedy same-step predictions, so external per-column sampling is
rejected for its targets.

### Causal Modelling Variants

#### Final-value Causal Modelling

In final-value causal models, the final value of each target variable within the subsequence is projected back in time as target. The idea is that the sequence of events leading up to the final value is a continuous accrual of evidence for an outcome, with the final value being the resolution. For example, the sequence of clicks through an online shop are in search of a product, and the product that is actually purchased at the end is the resolution.

#### Next-occurrence Causal Modelling

Next-occurrence causal modelling is a generalisation of final-value causal modelling: instead of taking the last value of each target variable as target, it takes the next value at a position where another categorical variable matches a criterion value as target. To illustrate this, final-value causal modelling is equivalent to next-occurrence causal modelling where the criterion variable is 'is_last', which is '0' up to the last position, where it is '1', and the criterion value is '1'. The values at the last position are projected 'back' across the subsequence, only now, we also have the option to use a different criterion variable, set it to '1' at multiple locations, and train the model to predict 'next relevant event', rather than just 'last event'.

### Causal Embedding Model

sequifier also supports the export of causal embeddings, instead of predictions. It requires the following settings: `export_embedding_model: true` in the training config and `model_type: embedding` in the inference config. Selected activations are restricted to the configured final `prediction_length` positions and concatenated in configuration order along the feature dimension.

If you are interested in activations *other* than the last backbone layer, you can configure the exact layers you want to contribute to the export using `embedding_layer_names`. You can pass an ordered list, such as `[backbone.layers.1, decoder.branches.default.hidden_blocks.0]`, and the activations of these layers will be concatenated and output.

Layer names follow the network hierarchy using zero-based indices: `backbone.layers.<index>` selects a transformer block output, `backbone.final_norm` the normalized backbone output, and `decoder.branches.<branch>.hidden_blocks.<index>` an MLP decoder hidden-block output; the same scheme applies to BERT embedding models.

Backbone selectors contribute `dim_model` activations. Decoder MLP hidden-block selectors contribute their configured hidden width and receive the same flattened `decoding_support * dim_model` windows used during training. The default, `embedding_layer_names: [backbone.final_norm]`, preserves the final normalised backbone representation.

### BERT Model

Sequifier also supports training and inference of BERT-style masked reconstruction models.

Configuration:

- Preprocessing: Set `max_target_offset: 0` for equal-width input and target windows.
- Training: Set `training_objective: bert`, configure `bert_spec`, and set decoder `prediction_length` equal to `context_length`. Enable generative and/or embedding export according to the desired inference.
- Inference: Set `model_type: generative` to reconstruct explicitly masked input, or `model_type: embedding` to output contextual representations.

Technical Details: BERT-style models use bidirectional attention and learn by reconstructing positions sampled according to `bert_spec`. Inference does not apply random masking; inputs that should be reconstructed must be masked explicitly, for example using `mask_column` during preprocessing. Embedding inference returns one contextual representation for every valid position in the input window.

### Structured Ingestion

Structured ingestion allows the model to learn relationships for configured sets of input variables *before* they are passed to the transformer backbone.

This enables more constrained representation learning within these subspaces, and provides a structural inductive bias and may reduce parameters, depending on configuration. It can be helpful to think of them as smaller submodules that learn local structure before passing the extracted information to the transformer for longer-range dependencies.

The key modalities are self-attention, pooling, 1D, 2D and 3D convolutions, and adding learned or rotary axis embeddings.

#### Repeated child collections

Named depth layouts represent a fixed-capacity collection of child rows at each
outer sequence position. A `depth_transformer` ingestion branch encodes the
valid children, pools them through a CLS token, and emits one vector per outer
position for the temporal transformer backbone. It can be combined with shallow
features in nested composite ingestion branches.

The preprocessing metadata and PT payload carry each layout's capacity and
validity mask. Portable PT and ONNX models retain that execution contract. See
the [preprocessing guide](documentation/configs/preprocess.md#named-depth-layouts),
[training guide](documentation/configs/train.md#depth-encoders-and-nested-composites),
and [inference guide](documentation/configs/infer.md#portable-depth-models-and-dropout-modes).

#### Temporal Convolution

Separately, `temporal_conv` enables temporal convolutions on pass-through or embedded real or categorical variables.

### Multi-Part Datasets

It is often the case that data grows and evolves, and we need the model to be updated using that data. Sequifier supports this practical reality by defining multi-part datasets as sets of data that share the same schema, categorical mappings, normalisation and storage contract, but have distinct metadata configs. In practice, this would look like processing every dataset after the first one with the `metadata_config_path` set to the metadata config created during the first preprocessing execution, to ensure that the properties line up as required. Also `window_length`, `max_target_offset`, normalization mode, dtypes, and file/folder storage form must match the first sequifier preprocess run.

### Composable Configs

Between different training runs and hyperparameter searches, a lot of configuration can get duplicated, and it becomes hard to follow what differentiates them and where they overlap. One approach to address this is to create different config fragments, and compose them into full training and hyperparameter tuning configurations using separate 'top-level' configs, and assemble the fragments by listing them as value in `additional_config_paths`. Fragments can contribute disjoint nested fields, while duplicate fields, recursive fragment inclusion, and repeated files are rejected; command-line overrides are applied after composition.

### Distributed Training

Sequifier supports distributed training using torch `DistributedDataParallel` and `FullyShardedDataParallel`. To make use of multi gpu support, the preprocessing step must write sharded output with `merge_output: false`. `write_format: pt` is the recommended file format; sharded `parquet` is also supported but currently considered beta for distributed training.

For the full guide on how to configure a distributed run, check the [multi-GPU training guide](./documentation/training/multi-gpu-training.md).

### System Requirements

Sequifier currently runs on MacOS and Ubuntu.

## Citation

Please cite with:

```bibtex
@software{sequifier_2025,
  author = {Luithlen, Leon},
  title = {sequifier - transformers for multivariate sequence generation and representation learning},
  year = {2025},
  publisher = {GitHub},
  version = {v2.2.0.0},
  url = {https://github.com/0xideas/sequifier}
}

```
