Metadata-Version: 2.4
Name: mintdim
Version: 0.1.3
Summary: Token unit pipeline for building low-padding binary token shards from JSONL datasets.
Author: MintDim contributors
License: MIT License
        
        Copyright (c) 2026 MintDim contributors
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: dataset,jsonl,shards,tokenizer,training
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Requires-Dist: numpy>=1.24
Requires-Dist: sentencepiece>=0.1.99
Requires-Dist: tokenizers>=0.15
Provides-Extra: dev
Requires-Dist: pytest-cov>=4.1; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Description-Content-Type: text/markdown

# MintDim

MintDim is a token unit pipeline for building low-padding binary token shards from JSONL datasets.

The first public API focuses on one task:

> stream JSONL batches → render templates → tokenize exactly → assign by token length → write binary token unit shards

## Install

~~~bash
pip install mintdim
~~~

For local development:

~~~bash
pip install -e .
~~~

## Quick start

~~~python
from mintdim import pipeline


def build_instruction_units():
    job = (
        pipeline("unit-build")
        .source.jsonl(
            files=[
                "./data/alpaca.jsonl",
            ],
            fields=[
                ["instruction", "input", "output"],
            ],
            templates=[
                "{instruction}\n\n{input}\n\n{output}",
            ],
        )
        .tokenizer.sentencepiece(
            {
                "path": "./tokenizer/tokenizer.model",
            },
        )
        .units(
            sizes=[[128, 192, 256, 320]],
            build_batch=[4096],
        )
        .output.dir(
            "./artifacts/instruction_units",
            samples_per_shard=[10000],
        )
    )

    return job.run()
~~~

## API flow

~~~text
pipeline("unit-build")
→ source
→ tokenizer
→ units
→ output
→ run
~~~

MintDim uses a fluent pipeline API. Each step declares one part of the build job. Nothing is executed until `job.run()` is called.

Batch build flow:

~~~text
stream JSONL sequentially
→ collect source batch
→ render text batch from template
→ hash rendered text batch
→ tokenizer.encode_batch(texts)
→ assign unit by exact token_count inside the batch
→ group records by unit_size
→ writer[unit].write_batch(token_ids)
→ write_many sample/UNK/duplicate index rows
→ update stats and histogram in batch
~~~

## Path rules

All relative paths are resolved from the current working directory.

Example:

~~~python
.source.jsonl(
    files=["./data/train.jsonl"],
    fields=[["text"]],
    templates=["{text}"],
)
~~~

If the script is run from the project root, MintDim resolves the file as:

~~~text
./data/train.jsonl
~~~

MintDim does not scan the repository or search the filesystem automatically. Users must provide explicit paths.

## JSONL source

~~~python
.source.jsonl(
    files=["./data/train.jsonl"],
    fields=[["text"]],
    templates=["{text}"],
)
~~~

Three arguments are required:

~~~text
files      → list of JSONL paths
fields     → list-of-list; fields[i] is the JSONL keys MintDim reads from files[i]
templates  → list of user-defined templates; templates[i] formats samples of files[i]
~~~

Multiple files are supported:

~~~python
.source.jsonl(
    files=[
        "./data/alpaca.jsonl",
        "./data/sharegpt.jsonl",
    ],
    fields=[
        ["instruction", "output"],
        ["prompt", "response"],
    ],
    templates=[
        "{instruction}\n\n{output}",
        "{prompt}\n{response}",
    ],
)
~~~

Each JSONL sample MUST contain every key listed in `fields[i]`. Extra JSONL keys are allowed and ignored.

## Templates

Templates are **user-defined strings**. MintDim does not ship preset templates (no `chatml`, `alpaca`, etc.). The user controls every character.

Template syntax:

~~~text
- placeholders use {field_name}
- placeholder names must match fields[i] exactly as a set
- any other character is treated as a literal (\n, \t, custom separators, etc.)
- placeholders are substituted by the JSONL sample's field values
~~~

Example:

~~~python
templates=[
    "{instruction}\n\n{input}\n\n{output}",
]
~~~

Rules:

~~~text
1. set(placeholders in templates[i]) == set(fields[i])
2. JSONL samples must contain every declared field in fields[i]
3. Declared fields that do not appear in the template are invalid
4. Static parts of templates[i] (every character outside placeholders)
   must encode cleanly with the tokenizer (no UNK)
5. Template rendering is a pure substitution; no extra tokens are injected
~~~

A user may use any character their tokenizer supports. If the tokenizer encodes `\t`, then `"{a}\t{b}"` is valid. If not, MintDim fails pre-flight.

Invalid example:

~~~python
fields=[
    ["prompt", "answer", "separator", "source"],
]
templates=[
    "{prompt}{separator}{answer}",
]
~~~

`source` is declared but not consumed by the template, so MintDim raises
`TemplateFieldMismatchError`.

## Tokenizer

V1 supports:

~~~text
- SentencePiece tokenizer: .model
- Hugging Face tokenizer: .json
~~~

Each tokenizer is configured with a **dict**, not a bare path. The user declares only the tokenizer file path. MintDim loads `unk_id` and `pad_id` from the tokenizer library itself.

SentencePiece:

~~~python
.tokenizer.sentencepiece(
    {
        "path": "./tokenizer/tokenizer.model",
    },
)
~~~

Hugging Face tokenizer JSON:

~~~python
.tokenizer.hf_json(
    {
        "path": "./tokenizer/tokenizer.json",
    },
)
~~~

Required keys:

~~~text
path → tokenizer file path
~~~

Validation rules:

~~~text
- path is a string
- the tokenizer library exposes a valid UNK id
- the tokenizer library exposes a valid PAD id
~~~

Why UNK and PAD are required:

~~~text
UNK
→ used for tracking unknown tokens and dataset integrity statistics

PAD
→ used to pad samples to assigned unit size for fixed-width binary shard layout
~~~

The tokenizer is never modified by MintDim. UNK and PAD ids come from the loaded tokenizer object, not from user-declared values or MintDim-side vocab parsing.

## Units

Shared unit config:

~~~python
.units(
    sizes=[[128, 192, 256, 320]],
    build_batch=[4096],
)
~~~

Mapped unit config:

~~~python
.units(
    sizes=[
        [128, 192, 256, 320],
        [96, 128, 160, 192, 224, 320],
    ],
    build_batch=[
        4096,
        2048,
    ],
)
~~~

Meaning:

~~~text
sizes
→ target shard unit sizes

build_batch
→ number of raw samples processed per tokenizer batch
~~~

MintDim always tokenizes samples exactly. Unit assignment is based on exact token count.

Example:

~~~text
token_count = 143
sizes = [128, 192, 256, 320]
target unit = 192
~~~

If a sample exceeds the largest configured unit size, MintDim fails immediately with `UnitOverflowError`.

MintDim V1 does not:
- truncate samples
- skip overflow samples
- continue partial builds
- cache build progress

## UNK handling

MintDim distinguishes two kinds of UNK and treats them differently.

### Tier 1 — template/tokenizer contract

Before any data is processed, every static part of every template is tokenized once. If any static part produces UNK, MintDim fails immediately with `TemplateTokenizerError`. This means the template's literal characters cannot be cleanly represented by the tokenizer, which is a configuration error, not a data issue.

### Tier 2 — data content UNK

After the template/tokenizer contract is validated, samples are processed. If a sample's field values produce UNK tokens when encoded, MintDim records an entry in `unk_index.jsonl`. Tier 2 UNK does not fail the build — it is a dataset/tokenizer coverage signal for audit.

Summary:

~~~text
template/tokenizer mismatch
→ fail fast (TemplateTokenizerError)

data content produces UNK
→ record in unk_index.jsonl
~~~

## Token storage dtype

MintDim writes tokenized shards as binary token arrays.

V1 supports:

~~~text
uint16
uint32
~~~

Dtype selection:

~~~text
vocab_size <= 65535
→ uint16

vocab_size > 65535
→ uint32
~~~

The selected dtype is written to `manifest.json`.

Example:

~~~json
{
  "token_dtype": "uint16"
}
~~~

## Output

~~~python
.output.dir(
    "./artifacts/instruction_units",
    samples_per_shard=[10000],
)
~~~

Two arguments:

~~~text
path / paths         → output directory path(s)
samples_per_shard    → list of positive ints; samples_per_shard[i] applies to files[i]
~~~

The output directory must be new or empty. MintDim V1 does not support overwrite. Use a new output path for a new build.

### Sharding

Shards are rolled **purely by sample count**:

~~~text
shard_000000  →  samples_per_shard[i] samples
shard_000001  →  samples_per_shard[i] samples
shard_xxxxxx  →  remainder (may be smaller)
~~~

This guarantees deterministic shard cardinality for the train loader. Shard byte size depends on `unit_size × bytes_per_token × samples_per_shard[i]` and varies per `unit_xxx/` directory.

### Shard observability

MintDim emits one line per closed shard:

~~~text
[unit-build] shard=000012 samples=100000 bytes=241.8 MiB unit=512
~~~

If a closed shard exceeds an internal recommended size (512 MiB), MintDim prints a warning but does not stop the build:

~~~text
[unit-build:warning] shard=000012 size=684.3 MiB exceeds recommended 512 MiB.
Build continues because sharding is controlled by samples_per_shard.
~~~

The threshold is an internal heuristic, not a user-facing config. To produce smaller shards, lower `samples_per_shard[i]`.

## Multiple files, fields, templates, tokenizers, units, and outputs

MintDim resolves mapping per config block.

For `N` input files, each list-style config may be either:

~~~text
len == 1
→ shared by every input file

len == N
→ mapped by input file index

anything else
→ fail fast
~~~

This means `fields`, `templates`, `tokenizer`, `sizes`, `build_batch`,
`output.dir`, and `samples_per_shard` can each be shared or mapped
independently.

### Fully shared

~~~python
job = (
    pipeline("unit-build")
    .source.jsonl(
        files=[
            "./data/alpaca.jsonl",
            "./data/sharegpt.jsonl",
        ],
        fields=[
            ["instruction", "output"],
        ],
        templates=[
            "{instruction}\n\n{output}",
        ],
    )
    .tokenizer.sentencepiece(
        {
            "path": "./tokenizer/tokenizer.model",
        },
    )
    .units(
        sizes=[[128, 192, 256, 320]],
        build_batch=[4096],
    )
    .output.dir(
        "./artifacts/instruction_units",
        samples_per_shard=[10000],
    )
)
~~~

All files share one schema, one template, one tokenizer, one unit config, and
one output directory. The files are written into the same output dataset.

### Fully mapped

~~~python
job = (
    pipeline("unit-build")
    .source.jsonl(
        files=[
            "./data/alpaca.jsonl",
            "./data/sharegpt.jsonl",
        ],
        fields=[
            ["instruction", "input", "output"],
            ["prompt", "response"],
        ],
        templates=[
            "{instruction}\n\n{input}\n\n{output}",
            "{prompt}\n{response}",
        ],
    )
    .tokenizer.sentencepiece([
        {
            "path": "./tokenizer/alpaca.model",
        },
        {
            "path": "./tokenizer/sharegpt.model",
        },
    ])
    .units(
        sizes=[
            [128, 192, 256, 320],
            [96, 128, 160, 192, 224, 320],
        ],
        build_batch=[
            4096,
            2048,
        ],
    )
    .output.dir(
        [
            "./artifacts/alpaca_units",
            "./artifacts/sharegpt_units",
        ],
        samples_per_shard=[10000, 5000],
    )
)
~~~

Everything is mapped by input order:

~~~text
files[i]
↔ fields[i]
↔ templates[i]
↔ tokenizer[i]
↔ sizes[i]
↔ build_batch[i]
↔ output_dir[i]
↔ samples_per_shard[i]
~~~

### Mixed shared and mapped

You may map only the parts that need separate outputs while sharing the rest:

~~~python
tokenizer_cfg = {
    "path": "./tokenizer/fineweb_edu_32k.model",
}

job = (
    pipeline("unit-build")
    .source.jsonl(
        files=[
            "./data/train_a.jsonl",
            "./data/train_b.jsonl",
        ],
        fields=[
            ["prompt", "answer", "separator"],
        ],
        templates=[
            "{prompt}{separator}{answer}",
        ],
    )
    .tokenizer.sentencepiece([
        tokenizer_cfg,
        tokenizer_cfg,
    ])
    .units(
        sizes=[[32, 64, 128, 256]],
        build_batch=[64],
    )
    .output.dir(
        [
            "./output1",
            "./output2",
        ],
        samples_per_shard=[256],
    )
)
~~~

Resolved order for this example:

~~~text
file 0 → fields[0], templates[0], tokenizer[0], sizes[0], build_batch[0], output1, samples_per_shard[0]
file 1 → fields[0], templates[0], tokenizer[1], sizes[0], build_batch[0], output2, samples_per_shard[0]
~~~

Passing a single tokenizer dict shares it across files:

~~~python
.tokenizer.sentencepiece(tokenizer_cfg)
~~~

Passing a list maps by file index, even if the entries contain the same values:

~~~python
.tokenizer.sentencepiece([tokenizer_cfg, tokenizer_cfg])
~~~

# message errors

1.
[english]
PipelineValidationError:
Pipeline configuration is incomplete.

Missing:
- source.files
- source.fields
- source.templates
- tokenizer
- units.sizes
- units.build_batch
- output.dir
- output.samples_per_shard

[vietnam]
PipelineValidationError:
Pipeline chưa được cấu hình đầy đủ.

Thiếu:
- source.files
- source.fields
- source.templates
- tokenizer
- units.sizes
- units.build_batch
- output.dir
- output.samples_per_shard

Thiếu cấu hình nào thì báo thiếu cấu hình đó.


2.
[english]
MintDimFileNotFoundError:
Required file was not found.

Context:
source

Missing file:
./data/train.jsonl

[vietnam]
MintDimFileNotFoundError:
Không tìm thấy file cần dùng.

Context:
source

File thiếu:
./data/train.jsonl


3.
[english]
SourceParseError:
JSONL line could not be parsed.

File:
./data/alpaca.jsonl

Line:
18291

Reason:
invalid JSON syntax

[vietnam]
SourceParseError:
Không đọc được dòng JSONL.

File:
./data/alpaca.jsonl

Dòng:
18291

Lý do:
JSON không hợp lệ


4.
[english]
SampleSchemaError:
JSONL sample is missing declared fields.

File:
./data/alpaca.jsonl

Line:
18291

Declared fields:
instruction
input
output

Missing fields:
input

Available keys:
instruction
output
metadata

[vietnam]
SampleSchemaError:
Sample JSONL thiếu fields đã khai báo.

File:
./data/alpaca.jsonl

Dòng:
18291

Fields đã khai báo:
instruction
input
output

Fields bị thiếu:
input

Keys hiện có:
instruction
output
metadata


5.
TokenizerValidationError reports structural tokenizer config problems. Special-token ids are loaded from the tokenizer library; users do not declare `unk_id` or `pad_id`.

Components:

~~~text
config → structural problem (not a dict, missing path, path not string, unknown keys)
~~~

Full ruleset (each component reports only its own rules):

~~~text
config must be a dict with key: path
path must be a string
unk_id and pad_id are loaded from the tokenizer library
~~~

Example — manual pad_id is declared:

[english]
TokenizerValidationError:
Invalid tokenizer config.

Component:
config

Problem:
tokenizer config has unknown keys: ['pad_id']. unk_id and pad_id are loaded from the tokenizer library.

Rules:
- config must be a dict with key: path
- path must be a string
- unk_id and pad_id are loaded from the tokenizer library

[vietnam]
TokenizerValidationError:
Cấu hình tokenizer không hợp lệ.

Thành phần:
config

Vấn đề:
config tokenizer có key không hợp lệ: ['pad_id']. unk_id và pad_id được lấy từ thư viện tokenizer.

Quy tắc:
- config phải là dict có key: path
- path phải là string
- unk_id và pad_id được lấy từ thư viện tokenizer

[english]
TokenizerSpecialTokenError:
Tokenizer library did not expose a required special token.

Special token:
pad_id

[vietnam]
TokenizerSpecialTokenError:
Thư viện tokenizer không cung cấp special token bắt buộc.

Special token:
pad_id


6.
[english]
TokenizerValidationError:
Unsupported tokenizer format.

Supported formats:
- sentencepiece (.model)
- huggingface tokenizer (.json)

[vietnam]
TokenizerValidationError:
Định dạng tokenizer chưa được hỗ trợ.

Các định dạng hiện hỗ trợ:
- sentencepiece (.model)
- huggingface tokenizer (.json)


7.
[english]
TokenizerMappingError:
Received 3 input files but 2 tokenizers.

MintDim supports:
- 1 tokenizer shared across all files
- N tokenizers for N files

[vietnam]
TokenizerMappingError:
Nhận 3 file đầu vào nhưng chỉ có 2 tokenizer.

MintDim hỗ trợ:
- 1 tokenizer dùng chung cho tất cả file
- N tokenizer cho N file


8.
[english]
FieldMappingError:
Received 3 input files but 2 field configs.

MintDim supports:
- 1 field config shared across all files
- N field configs for N files

[vietnam]
FieldMappingError:
Nhận 3 file đầu vào nhưng chỉ có 2 cấu hình fields.

MintDim hỗ trợ:
- 1 fields dùng chung cho tất cả file
- N fields cho N file


9.
[english]
TemplateMappingError:
Received 3 input files but 2 templates.

MintDim supports:
- 1 template shared across all files
- N templates for N files

[vietnam]
TemplateMappingError:
Nhận 3 file đầu vào nhưng chỉ có 2 template.

MintDim hỗ trợ:
- 1 template dùng chung cho tất cả file
- N template cho N file


10.
[english]
TemplateFieldMismatchError:
Source file:
./data/train.jsonl

API declaration for this file:
fields = ['instruction', 'input', 'output']
template = "{instruction}\n\n{prompt}\n\n{output}"

Field causing template mismatch:
prompt
Reason:
template uses {prompt}, but fields does not declare it


[vietnam]
TemplateFieldMismatchError:
File nguồn:
./data/train.jsonl

Khai báo API cho file này:
fields = ['instruction', 'input', 'output']
template = "{instruction}\n\n{prompt}\n\n{output}"

Field gây lệch template:
prompt
Lý do:
template dùng {prompt}, nhưng fields không khai báo field này


11.
[english]
TemplateTokenizerError:
Template static parts produce UNK when encoded by the tokenizer.

File index:
0

Template:
"{a}\n\n{b}"

Problem:
template literal characters are not fully covered by tokenizer vocabulary

Fix:
adjust template literal characters or use a tokenizer that covers them

[vietnam]
TemplateTokenizerError:
Phần literal của template tạo UNK khi tokenize.

Vị trí file:
0

Template:
"{a}\n\n{b}"

Vấn đề:
ký tự literal trong template không được tokenizer bao phủ

Cách sửa:
chỉnh ký tự literal hoặc dùng tokenizer bao phủ được


12.
[english]
UnitMappingError:
Received 3 input files but 2 unit configs.

MintDim supports:
- 1 unit config shared across all files
- N unit configs for N input files

[vietnam]
UnitMappingError:
Nhận 3 file đầu vào nhưng chỉ có 2 unit config.

MintDim hỗ trợ:
- 1 unit config dùng chung cho tất cả file
- N unit config cho N file


13.
[english]
OutputMappingError:
Received 3 input files but 2 output directories.

MintDim supports:
- 1 shared output directory
- N output directories for N input files

[vietnam]
OutputMappingError:
Nhận 3 file đầu vào nhưng chỉ có 2 thư mục output.

MintDim hỗ trợ:
- 1 output dùng chung
- N output cho N file


14.
[english]
UnitValidationError:
Invalid unit config.

Rules:
- sizes must be positive integers
- sizes must be sorted ascending
- build_batch must be a positive integer

[vietnam]
UnitValidationError:
Unit config không hợp lệ.

Quy tắc:
- sizes phải là các số nguyên dương
- sizes phải tăng dần
- build_batch phải là số nguyên dương


15.
[english]
UnitOverflowError:
Sample token length exceeds max unit size.

Sample:
./data/alpaca.jsonl:18291

token_count:
417

max_unit_size:
320

Increase unit sizes and run again.

[vietnam]
UnitOverflowError:
Độ dài token của mẫu vượt quá unit lớn nhất.

Mẫu:
./data/alpaca.jsonl:18291

token_count:
417

max_unit_size:
320

Hãy tăng unit sizes rồi chạy lại.


16.
[english]
OutputValidationError:
Output directory already exists and is not empty.

Directory:
./artifacts/token_shards

Use a new output directory.

[vietnam]
OutputValidationError:
Thư mục output đã tồn tại và không rỗng.

Thư mục:
./artifacts/token_shards

Hãy dùng thư mục output mới.


17.
[english]
TokenDTypeValidationError:
Tokenizer vocabulary is too large for supported token dtypes.

Supported dtypes:
- uint16
- uint32

[vietnam]
TokenDTypeValidationError:
Vocab của tokenizer quá lớn so với dtype token được hỗ trợ.

Các dtype hiện hỗ trợ:
- uint16
- uint32


18.
SamplesPerShardError reports value-level and arity-level problems for `samples_per_shard`.

Example — non-positive value:

[english]
SamplesPerShardError:
Invalid samples_per_shard.

Problem:
samples_per_shard[0] must be a positive int, got 0

Rules:
- each samples_per_shard[i] is a positive int
- len(samples_per_shard) == 1 or len(samples_per_shard) == file_count

[vietnam]
SamplesPerShardError:
samples_per_shard không hợp lệ.

Vấn đề:
samples_per_shard[0] phải là số nguyên dương, hiện tại là 0

Quy tắc:
- mỗi samples_per_shard[i] phải là số nguyên dương
- len(samples_per_shard) == 1 hoặc len(samples_per_shard) == số file


Example — arity mismatch:

[english]
SamplesPerShardError:
Invalid samples_per_shard.

Problem:
Received 3 input files but 2 samples_per_shard entries

Rules:
- each samples_per_shard[i] is a positive int
- len(samples_per_shard) == 1 or len(samples_per_shard) == file_count

[vietnam]
SamplesPerShardError:
samples_per_shard không hợp lệ.

Vấn đề:
Nhận 3 file đầu vào nhưng chỉ có 2 samples_per_shard

Quy tắc:
- mỗi samples_per_shard[i] phải là số nguyên dương
- len(samples_per_shard) == 1 hoặc len(samples_per_shard) == số file


## Repository layout

~~~text
mintdim/
├─ pyproject.toml
├─ README.md
├─ LICENSE
├─ .gitignore
│
├─ mintdim/
│  ├─ __init__.py
│  ├─ pipeline.py
│  ├─ registry.py
│  │
│  ├─ shared/
│  │  ├─ __init__.py
│  │  ├─ paths.py
│  │  ├─ errors.py
│  │  └─ types.py
│  │
│  └─ pipelines/
│     ├─ __init__.py
│     │
│     └─ unit_build/
│        ├─ __init__.py
│        ├─ api.py
│        ├─ config.py
│        ├─ contracts.py
│        ├─ validate.py
│        ├─ runtime.py
│        │
│        ├─ source/
│        │  ├─ __init__.py
│        │  ├─ jsonl.py
│        │  ├─ template.py
│        │  └─ validate.py
│        │
│        ├─ tokenizer/
│        │  ├─ __init__.py
│        │  ├─ sentencepiece.py
│        │  ├─ hf_json.py
│        │  └─ validate.py
│        │
│        ├─ units/
│        │  ├─ __init__.py
│        │  ├─ planner.py
│        │  ├─ shard_writer.py
│        │  └─ validate.py
│        │
│        └─ output/
│           ├─ __init__.py
│           ├─ dir.py
│           ├─ manifest.py
│           ├─ sample_index.py
│           ├─ duplicate_index.py
│           ├─ unk_index.py
│           ├─ histogram.py
│           ├─ stats.py
│           └─ validate.py
│
├─ tests/
│  └─ unit_build/
│     ├─ test_api_chain.py
│     ├─ test_validate.py
│     ├─ test_jsonl_source.py
│     ├─ test_template.py
│     ├─ test_tokenizer.py
│     ├─ test_units.py
│     ├─ test_sample_index.py
│     ├─ test_duplicate_index.py
│     ├─ test_unk_index.py
│     └─ test_output.py
│
└─ examples/
   └─ unit_build_instruction.py
~~~

## Output layout

### Shared output layout

~~~text
./artifacts/instruction_units/
├─ manifest.json
├─ histogram.json
├─ stats.json
├─ sample_index.jsonl
├─ duplicate_index.jsonl
├─ unk_index.jsonl
│
├─ unit_128/
│  ├─ shard_000000.bin
│  ├─ shard_000001.bin
│  └─ ...
│
├─ unit_192/
│  ├─ shard_000000.bin
│  ├─ shard_000001.bin
│  └─ ...
│
├─ unit_256/
│  ├─ shard_000000.bin
│  ├─ shard_000001.bin
│  └─ ...
│
└─ unit_320/
   ├─ shard_000000.bin
   ├─ shard_000001.bin
   └─ ...
~~~

Meaning:

~~~text
manifest.json
→ pipeline metadata, build configuration, tokenizer metadata, and token dtype

histogram.json
→ sample distribution by assigned unit size

stats.json
→ summarized token, shard, duplicate, and UNK statistics

sample_index.jsonl
→ per-sample hash, token count, unit assignment, and shard location

duplicate_index.jsonl
→ grouped duplicate samples by hash

unk_index.jsonl
→ samples containing UNK tokens and their token positions

unit_xxx/
→ binary token shards grouped by unit size

unit_xxx/EMPTY_UNIT.md
→ generated when a declared unit receives no assigned samples

unit_xxx/UNIT_BUILD_FAILED.md
→ generated when the build fails before the unit output is finalized
~~~

### manifest.json

Example:

~~~json
{
  "pipeline": "unit-build",
  "mintdim_version": "0.1.0",
  "tokenizer": {
    "type": "sentencepiece",
    "path": "./tokenizer/tokenizer.model",
    "vocab_size": 32000,
    "unk_id": 0,
    "pad_id": 1
  },
  "token_storage": {
    "dtype": "uint16",
    "bytes_per_token": 2
  },
  "units": {
    "sizes": [128, 192, 256, 320],
    "build_batch": 4096
  },
  "source": {
    "files": ["./data/alpaca.jsonl"],
    "fields": [["instruction", "input", "output"]],
    "templates": ["{instruction}\n\n{input}\n\n{output}"]
  },
  "output": {
    "samples_per_shard": 10000
  }
}
~~~

### histogram.json

Example:

~~~json
{
  "128": 30291,
  "192": 82191,
  "256": 120882,
  "320": 43121
}
~~~

Meaning:

~~~text
histogram.json counts how many samples were assigned to each unit size.
~~~

### stats.json

Example:

~~~json
{
  "total_samples": 1823811,
  "total_tokens": 482918221,
  "token_dtype": "uint16",
  "bytes_per_token": 2,
  "duplicate_groups": 821,
  "duplicate_samples": 1942,
  "samples_with_unk": 321,
  "unk_tokens": 1821,
  "unit_distribution": {
    "128": 582111,
    "192": 891221,
    "256": 301882,
    "320": 48291
  }
}
~~~

### sample_index.jsonl

Each line describes one source sample after tokenization and unit assignment.

Example:

~~~json
{
  "sample_id": 91822,
  "hash": "8f2c1b7d9a9d4a0d...",
  "token_count": 143,
  "unit_size": 192,
  "source_file": "./data/alpaca.jsonl",
  "source_line": 18291,
  "shard_path": "unit_192/shard_000001.bin"
}
~~~

Purpose:

~~~text
sample_index.jsonl enables:
- dataset audit
- sample-level reproducibility
- deduplication
- corruption detection
- shard rebuild/debug
- token count verification
~~~

### duplicate_index.jsonl

Each line groups all duplicated samples sharing the same content hash.

Example:

~~~json
{
  "hash": "8f2c1b7d9a9d4a0d...",
  "count": 3,
  "samples": [
    {
      "sample_id": 12031,
      "source_file": "./data/alpaca.jsonl",
      "source_line": 991
    },
    {
      "sample_id": 91822,
      "source_file": "./data/alpaca.jsonl",
      "source_line": 18291
    },
    {
      "sample_id": 140992,
      "source_file": "./data/sharegpt.jsonl",
      "source_line": 2012
    }
  ]
}
~~~

Purpose:

~~~text
duplicate_index.jsonl enables:
- duplicate audit
- dataset cleanup
- repeated sample detection
- duplicate filtering
- corruption investigation
~~~

### unk_index.jsonl

Each line describes one sample whose **field values** produced UNK tokens. UNK from template literals never reaches this file — that case fails pre-flight with `TemplateTokenizerError`.

Example:

~~~json
{
  "sample_id": 91822,
  "hash": "8f2c1b7d9a9d4a0d...",
  "source_file": "./data/alpaca.jsonl",
  "source_line": 18291,
  "unk_count": 2,
  "unk_positions": [17, 89]
}
~~~

Purpose:

~~~text
unk_index.jsonl enables:
- tokenizer quality audit
- unknown token analysis
- tokenizer coverage inspection
- dataset cleanup
- encoding issue detection
~~~

### Binary token shards

Unit shard files are stored as **fixed-width binary token arrays**.

Each record in `unit_xxx/shard_*.bin` occupies exactly `unit_size` token slots:

~~~text
slots[0 : token_count]        = encoded tokens
slots[token_count : unit_size] = PAD token (from tokenizer.pad_id)
~~~

This guarantees deterministic reader offsets: the k-th sample in a shard starts at byte `k * unit_size * bytes_per_token`.

Example:

~~~text
unit_192/shard_000001.bin
~~~

If `token_dtype` is `uint16`, each token uses 2 bytes (record size = `unit_size * 2`).

If `token_dtype` is `uint32`, each token uses 4 bytes (record size = `unit_size * 4`).

The shard dtype and `pad_id` are recorded in `manifest.json`. Readers may rely on `sample_index.jsonl.token_count` to know the boundary between encoded tokens and padding.

### Multi-output layout

~~~text
./artifacts/
├─ alpaca_units/
│  ├─ manifest.json
│  ├─ histogram.json
│  ├─ stats.json
│  ├─ sample_index.jsonl
│  ├─ duplicate_index.jsonl
│  ├─ unk_index.jsonl
│  │
│  ├─ unit_128/
│  │  ├─ shard_000000.bin
│  │  └─ ...
│  ├─ unit_192/
│  │  ├─ shard_000000.bin
│  │  └─ ...
│  ├─ unit_256/
│  │  ├─ shard_000000.bin
│  │  └─ ...
│  └─ unit_320/
│     ├─ shard_000000.bin
│     └─ ...
│
└─ sharegpt_units/
   ├─ manifest.json
   ├─ histogram.json
   ├─ stats.json
   ├─ sample_index.jsonl
   ├─ duplicate_index.jsonl
   ├─ unk_index.jsonl
   │
   ├─ unit_96/
   │  ├─ shard_000000.bin
   │  └─ ...
   ├─ unit_128/
   │  ├─ shard_000000.bin
   │  └─ ...
   ├─ unit_160/
   │  ├─ shard_000000.bin
   │  └─ ...
   ├─ unit_192/
   │  ├─ shard_000000.bin
   │  └─ ...
   ├─ unit_224/
   │  ├─ shard_000000.bin
   │  └─ ...
   └─ unit_320/
      ├─ shard_000000.bin
      └─ ...
~~~

Each output directory is isolated and mapped by input order when
`output.dir` has one path per input file. Other blocks may still be shared:

~~~text
files[i]
↔ fields[0] or fields[i]
↔ templates[0] or templates[i]
↔ tokenizer[0] or tokenizer[i]
↔ sizes[0] or sizes[i]
↔ build_batch[0] or build_batch[i]
↔ output_dir[i]
↔ samples_per_shard[0] or samples_per_shard[i]
~~~

## Design principles

~~~text
small public API
explicit paths
explicit user-defined templates (no presets)
required-field sample validation (extra JSONL keys are ignored)
fail fast validation
two-tier UNK contract (template fail vs data record)
exact tokenization
direct unit assignment by token length
fixed-width padded binary shards
automatic token dtype selection
no tokenizer mutation
special token ids loaded from tokenizer library
no automatic repo scanning
no overwrite in V1
no build resume/cache
pipeline-specific internals
sample-level reproducibility
dataset quality audit support
~~~

## Version

Initial public version:

~~~text
0.1.0
~~~
