Metadata-Version: 2.4
Name: atrep-cli
Version: 0.3.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Rust
Classifier: Topic :: Text Processing :: Markup
License-File: LICENSE-MIT
License-File: LICENSE-APACHE
Summary: The atrep command-line tool: check, respellings, outline, kanonizo, endo importers, morphisms, exo rendering, collation
Keywords: atrep,markup,semantic,document,dialect,cli
Author: Bojan Đuričković
License-Expression: MIT OR Apache-2.0
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Repository, https://gitlab.com/atrep/atrep-rs

# atrep-rs

Pilot Rust implementation of the **Atrep** core: parser, Dendron
(AST), kanonizo (canonicalization), litosis (content hashing),
metamorphoses (exomorphosis engine; Markdown, HTML, and
reStructuredText endomorphoses), and morphisms (.hom/.iso with
group rules and transitive route resolution, spec v0.11).

## Spec reference

This implementation targets the **Atreptos Platform Specification
v0.12.1** (the Dialektoi Network and Witnesses & Coordinates
freezes plus their addenda), as published in the `atrep/spec`
repository:

- tag: `v0.12.1`
- commit: `3c72861`

Where behavior is underspecified at that commit, the choices made
here are documented under "Spec gaps" below, as input for the next
spec revision.

## Layout

- `crates/atrep` — library: `sigil` (escaping, symbol
  classes), `dialektos` (`.lektos`/`.dia` definitions, local
  inheritance and import), `parser` (document to Dendron),
  `dendron` (AST + canonical serializer), `kanonizo`, `litosis`.
- `crates/atrep-cli` — the `atrep` binary.
- `crates/atrep-lsp` — the `atrep-lsp` language server:
  diagnostics + semantic-token highlighting for `.atd`/`.atk`
  documents and `.dia`/`.lektos` definitions (see its README
  for editor configuration).

## Usage

```
atrep check <file>                 parse + validate a document
                                   (.atd/.atk) or definition
                                   (.lektos/.dia)
atrep kanonizo <file> [-o out]     .atd -> .atk
                                   (+ .atk.tar.gz when media
                                    is bundled);
                                   .dia -> .lektos for
                                   definition files
              [--timeout <secs>]   remote-fetch timeout (30)
              [--retries <n>]      remote-fetch retries (2)
atrep morph <file> <dialektos> [-o out]
                                   map a kanon to another dialektos
                                   via .hom/.iso rules or a
                                   lineage-derived embedding
atrep litos <file> [--save-litos-file]
                                   print the litos ID (SHA-256)
atrep exo <file> <target> [-o out]
                                   render a kanon to an external
                                   format via <dialektos>.<target>.exo
                                   (.atd input is kanonized first)
atrep endo <file> [-o out]         import Markdown (.md), HTML
                                   (.html), reStructuredText
                                   (.rst), or TEI (.xml/.tei) as
                                   at-markdown/at-html/at-rst/
                                   litogramma documents
```

Dialektoi resolve local-only, per the v0.10 spec: the parser looks
for `<dialect-id>.lektos` (falling back to `<dialect-id>.dia`) in
the directory of the file being processed. Remote *resources*
(transclusion targets and media) may be `http(s)` URLs.

`scripts/pandoc-corpus.sh` uses pandoc as a specimen
generator: each Markdown seed converts into every imported
format and runs through `atrep endo`; rejections are coverage
findings, not gated failures.

The crate builds for `wasm32-unknown-unknown` with
`--no-default-features`: the `net` feature (remote resource
fetching) and the `bundle` feature (`.atk.tar.gz` archive
output) are on by default and off for wasm targets.

Milestones and zygosis (spec v0.12): `@("scheme:value")` is
the core global anchor — the quoted sibling of the onym
anchor, kanonizo-exempt (duplicates rejected) and
litosis-surviving. Exos render it through the `*milestone
[<scheme>]` pattern with `@(scheme)`/`@(value)` slots. `atrep
zygo a.atk b.atk --scheme stephanus --as grc,eng -o bi.atd`
weaves witness kanons of one dialektos into the zygoma:
milestone-headed segments in the merged coordinate order
(order-consistency checked, contradictions are errors), each
witness's slice on a `zyg-<id>`-tagged diaphane, blocks and
even inline phrases split cleanly at cut points, headings and
speech prefixes hoisted so they don't repeat. The TEI importer
emits core milestones (scheme from resp, unit as a
presentation genos).

Controlled vocabularies (spec v0.11): a dialektos may define
named term sets with multilingual aliases, bound to a sim's
lemma or genoses; kanonizo canonicalizes aliases
case-insensitively (authoring language cannot fork hashes),
vocabulary-bound genoses survive litosis as semantic content,
and exo rules gain term-keyed variants (`@-> :(auctor)`).
Bibliogramma drives it: Latin canonical terms, ~40 languages of
aliases, biblatex as the lossless export and classic BibTeX as
the lossy one.

Resolution is storage-agnostic behind the `source::Source` trait:
`DirSource` is the classical directory context, `MemorySource`
serves definitions, morphisms, and exos from an in-memory map
(embedded engines, registry services, wasm targets with no
filesystem). The `*_from` variants (`dialektos::resolve_from`,
`morph::resolve_morph_from` / `resolve_route_from`,
`exo::resolve_exo_from`) take any source; the path-based entry
points delegate to `DirSource`.

## Conformance status (spec v0.10)

Implemented:

- The three basic forms (endo-simmere, para-simmere, monosim) with
  onym/genos/taxis annotations; both sigils (`@` / `\`) with the
  inactive-sigil escaping rules (`\@`, `@\`, `\|`); bracket
  matching in episims; core stichoi (`@@@=`); comments; diaphanes;
  axiomata; verbatim and semantic inclusion (inline, block, and
  file forms); enmedia; the deixis (`@<sym>(onym)`, spec v0.10.1) —
  a core pointing reference to an onymized para-simmere, validated
  against its target's sim, surviving litosis with its target's
  onym under fresh declaration-order numbering (so stripped
  metadata cannot influence the litos ID), and rendered via the
  `*deixis <symbol>` exo pattern.
- Dialektos definitions in the `atrep` meta-dialektos: ostensive
  sim definitions (including line-structured grammata via the
  `stichos` keyword), descriptions, parent references, full
  inheritance, sim import (single, aliased, list, exclusion) —
  local-only, with conflict and cycle detection.
- Kanonizo: shebang removal, sigil canonicalization with escape
  recomputation, Unicode NFC, recursive transclusion expansion
  with cycle detection (local paths and URLs), axioma expansion,
  axioma-enlexis resolution against onymized simmeres, onym
  canonicalization (`o1`, `o2`, ...), taxis evaluation and
  validation, comment and unreferenced-diaphane removal,
  paragraph collapsing, whitespace normalization, media bundling
  (`media/m<n>.<ext>`, manifest, deterministic `.atk.tar.gz`).
- Remote resources over `http(s)`: transclusion targets and
  enmedia, with configurable timeout/retry, extension inference
  (source extension, then Content-Type, then magic bytes), and
  source-URL + checksum provenance in the media manifest.
- Definition-file kanonizo (`.dia` -> `.lektos`): inheritance and
  imports expanded, canonical sigil, sim definitions sorted by
  name; re-emitting a canonical `.lektos` is byte-identical.
- Litosis and the litos ID (SHA-256), including enmedia content
  hashing (`@@@@[SHA256:...]`).
- Exomorphosis (spec v0.10.1, optional conformance class): `.exo`
  files in the `atrep-exo` meta-dialektos — symbol-keyed template
  rules with genos-suffix specificity, structural `*` patterns
  (including `*solo` and `*deixis`), sigil slots with raw and
  hanging-prefix forms, conditional sections (`@[ ... ]@`,
  emitted only when every slot inside renders non-empty),
  single-pass escape tables, englossis rule-set switching,
  strict unhandled-pattern errors, deterministic output. Rule
  inheritance follows the dialektos lineage (spec v0.10.2 draft):
  parents' effective rule sets are pulled in declaration order
  (later ops overriding earlier), filtered and alias-remapped
  per import kind, then overlaid by the dialektos's own file; a
  dialektos with no `.exo` of its own still renders through
  inherited rules, and the most derived non-empty escape table
  applies. The spec's worked example (`exempli.html.exo`) is a
  golden test.
- A standard library, embedded in the binary and used when local
  resolution fails: `at-html`, a syntax mapper between a useful
  subset of HTML and atrep (each sim corresponds to one HTML
  element; its exomorphosis and the HTML endomorphosis are
  inverses over the canonical subset), and `at-markdown`
  (realizing the dialektos the spec's inheritance examples treat
  as hypothetical), which is pure lineage: every sim is inherited
  from at-html — containers excluded (markdown is flat), emphasis
  and strong aliased to `*`/`**` — and its HTML rendering is
  acquired entirely through rule inheritance, with no HTML `.exo`
  of its own. The library also carries the literary target
  itself, promoted to core: `litogramma` and `bibliogramma`,
  vendored byte-identical from the normative litogramma repo (a
  drift-guard test compares the copies whenever the sibling
  checkout is present), together with their base exomorphoses
  (`html`, `gemtext`, `latex`; `bib`, `bibtex`) and the
  litogramma↔at-html/at-docbook/at-tei homs — so the
  litogramma-targeting importers (TEI, JATS, BibTeX, atramento)
  have their target resolvable everywhere, with no local
  definition files. The
  publisher-register exos (Loeb, OCT, Penguin, GOST, …) remain
  in the litogramma repo.
- Morphisms (spec v0.11, optional conformance class):
  `.hom` (directional, possibly lossy) and `.iso` (rename-only,
  bijective, one file serving both directions) in the
  atrep-hom/atrep-iso meta-dialektoi — rename with form-
  compatibility validation, unwrap (wrapper dissolves, content
  splices), drop; implicit identity for same-symbol
  form-compatible sims; core forms pass through; deixes follow
  their target sim's rule (removed with it on drop/unwrap); the
  output kanon is re-validated (taxis runs, deixis targets).
  Embedding morphisms are derived from the dialektos lineage
  with no file at all: at-markdown => at-html morphs purely from
  its inheritance declarations, and the std
  `at-html.at-markdown.hom` provides the lossy reverse — the
  flat-content round trip is byte-identical. Renames may append
  genoses (admonition kinds survive as presentation hints on
  generic containers) and the extract rule (`@<# <sym>
  <endo-sym>`) dissolves a lemma-bearing para-simmere into a
  flat heading plus content. Morphisms apply **transitively**:
  when no direct morphism resolves, the shortest route through
  the known dialektoi applies in validated stages (ambiguous
  routes are errors, never silent choices) — at-rst =>
  at-markdown composes through at-html with no direct file. A
  pair may carry several morphisms as named variants
  (`<a>.<b>.<variant>.hom`, `atrep morph ... --variant <name>`)
  — distinct representations of the source in the target. The
  std graph carries its first **isomorphism**: at-djot and
  at-markdown share one sim inventory under different symbols
  (`at-djot.at-markdown.iso`), so the pair converts losslessly
  both ways, round-trips to the identity morphism, and
  collapses into one equivalence class.
  The algebra's one sequence-level action is the **group
  family** (`@>#` / `@><` / `@>-` / `@>:`), the flat-to-nested
  converse of extract: a solo-heading paragraph triggers a
  region spanning the following siblings, wrapped into a
  para-simmere with the trigger content as lemma (or discarded,
  or the region dropped, or the trigger relabeled/deleted, or —
  operand-less `@>#`, the v0.11.1 deplain — stripped to a plain
  paragraph with the region untouched);
  rules nest by file order, the trigger's onym and genoses
  transfer to the wrapper, and deixes follow. The
  group-then-extract round trip recovers the flat document
  exactly. Morphisms are **statically validated at
  resolution**, with no documents present: every rule against
  both dialektoi, plus *totality* — every source sim accounted
  for by a rule, an implicit identity, or a group trigger, with
  `@--` marking deliberate drops (derived embeddings stay
  partial by nature).
  Morphisms **compose symbolically**: the table algebra is
  three actions (rename, drop, dissolve — with the lemma
  discarded, emitted plain, or extracted into a heading endo)
  and is closed under composition, group dispositions chasing
  the second factor's table, so a route fuses into a single
  composite morphism applied in one traversal (`apply_route`
  does this by default; staged application is the reference
  semantics it must match). Grouping bounds fusion: a grouping
  second factor fuses only across a rename-only first factor,
  and never across a non-injective rename onto a trigger (peer
  triggers would share a rank the serialized form cannot
  express; spec v0.11.1) — everywhere else `apply_route` falls
  back to staged application automatically. `atrep compose a b [c ...]` writes
  the composite as an ordinary `.hom` file in normal form —
  group rules in rank order, table rules sorted, implicit
  identities elided, with `@=via` declarations recording the
  hops (provenance, excluded from the `normal_form` equality
  surface) — making morphism equality a byte comparison.
- Endomorphoses (pilot, engine-side): **atramento**, the
  Markdown-like authoring surface for litogramma (`.atr`,
  text-to-text: a litogramma superset whose sugar — typewriter
  emphasis with flanking rules, ATX headings with a declared
  base and implied closure, Markdown lists/quotes/fences,
  autolinks, note callout/definition/inline sugar, verse fences,
  drama auto-closure — compiles away; the compiler is an
  idempotent projection and canonical litogramma passes through
  byte-identical; spec in the atramento repo); CommonMark-subset,
  HTML-subset, and reStructuredText-subset importers into
  `at-markdown`/`at-html`/`at-rst` (strict over the canonical
  subsets; RST footnotes become deixis callouts, fixed adornment
  table for title levels), Org-mode and Djot subset importers into the
  `at-org` and `at-djot` std dialektoi, a DocBook subset
  importer into `at-docbook` (`.dbk`); the `at-tei` std dialektos
  applies the bibliogramma pattern to TEI — ~15 form sims with
  the TEI element identity in vocabulary-bound genoses
  (`@,x,@.persname`), a `tei` exo emitting real TEI XML, and a
  litogramma hom giving litogramma TEI export via morph+exo, a BibTeX importer into
  `bibliogramma` (`.bib`), a JATS subset importer (`.jats`) into
  litogramma with the ref-list embedded as a bibliogramma
  englossis — the BibTeX-isomorphic bibliography
  extension defined beside litogramma (entry lemma = citation
  key, genos = entry type, fields as lemma-named sims) — the first *nested* std
  dialektos: one uniform section sim nests, so
  structure-preserving morphisms to and from litogramma need
  no flattening — (`.org`: star headings, the emphasis
  family, quote/verse/example/src blocks with language genoses,
  pipe tables as stichoi rows, footnotes as deixis callouts,
  links), plus a TEI P5 basic-subset importer
  into `litogramma` (`.xml`/`.tei`): teiHeader front matter,
  nested div sectioning with type/depth laddering, lg/l verse,
  sp/speaker drama with stage directions, epigraphs with
  attribution, and notes as deixis callouts with footnote
  bodies. Roundtrips are tested in all three
  formats: the canonical form of each is a fixed point of
  export∘import.
  The `at-usfm` std dialektos maps scripture (`.usfm`/`.sfm`):
  one dialektos, three export surfaces (`usfm`, `usx`, `osis`).
  Books are vocabulary-bound lemmas (the 66 canonical plus 21
  deuterocanonical USFM codes with English-name aliases, so
  `@# john` and `@# jhn` hash identically); chapters and verses
  are milestone monosims — text flows across verse boundaries,
  the classic overlapping hierarchy; heading markers ride a
  `markers` vocabulary on one typed solo-block sim while
  paragraph-class markers (`\m`, `\pi1`, `\li1`, …) accumulate
  verse flow as typed paragraphs; character markers (`\wj`,
  `\nd`, `\add`, …) are genos-typed phrases, wordlist wrappers
  (`\w …|attrs`, `\+w`) unwrap to their text; footnotes and
  cross-references sit inline at their anchor with `\fr`/`\xo`
  references as typed sub-phrases; poetry lines wrap in
  per-line `q`-level phrases inside stichoi so indent levels
  survive. The OSIS exo maps semantics, not styling (`\wj` →
  `<q who="Jesus">`, `\nd` → `<divineName>`, `\add` →
  `<transChange>`). All three surfaces import as well as
  export: `.usfm`/`.sfm`, `.usx`, and `.osis` files (including
  whole-Bible multi-book files, which become one book block
  per book). The USX surface is information-complete — export
  then import lands on the identical litos; OSIS import
  handles both container and milestone chapter/verse forms,
  Paratext-style sID/eID pairs, and pretty-printed sources
  (insignificant whitespace collapses), while milestone-form
  red-letter `<q sID/>` spans are skipped (the overlapping
  hierarchy OSIS itself avoids in container form).
  The `latex` surface prints it: the exo
  emits a semantic macro layer (`\atbook`, `\atchapter`,
  `\atverse`, `\atmarker{s1}{...}`, `\atchar{wj}{...}`,
  `\atq`, `\atnote`) and carries the `atrep-bible` document
  class as an **asset section** (`@=* <filename> <sentinel>`
  ... sentinel) — a presentation companion delivered
  byte-verbatim as an unconditional auxiliary output, so the
  exo stays one artifact under one content address. The class
  compiles the layer into a two-column dense print Bible:
  drop-number chapters with the first verse suppressed (or
  centered `PSALM n` headings when the book declares `\cl`),
  superscript verses, q-level poetry ladders with right-set
  *Selah*, italic superscriptions, per-column footnotes with
  bold references, running heads from `\h` (falling back to
  `\mt1`), and a `redletter` class option. Genoses interpolate
  as macro arguments, so vocabulary growth touches only the
  class — unknown markers fall through to a visible fallback,
  never an error. Named **exo variants**
  (`<dialektos>.<target>.<variant>.exo`, `atrep exo ...
  --variant <name>`) overlay the base rule set — the
  `redletter` edition redefines only the document rule and
  inherits everything else, class asset included. The class
  sets corner running heads the print way (every verse marks
  `BOOK c:v`; first reference on the verso's outer edge, last
  on the recto's), opens each book at the top of a fresh
  column, sets chapters as lettrine drop numbers, gives
  cross-references their own symbol series beside the numbered
  translation notes, prints a two-column dotted Contents page
  when the document holds more than one book (with PDF
  bookmarks via hyperref), switches to a covering serif for
  Hebrew (right-to-left) and polytonic Greek runs in
  footnotes, and offers a `9pt` option for a denser setting
  and an `a5` exo variant (the classic 148×210 hand-Bible
  format at 9pt). The geometry is calibrated against
  distributed print Bibles (measured block proportions,
  margins, and baseline pitch), and an ink-zone check renders
  every page and asserts nothing prints into the gutter or
  margins.
  `scripts/bible-pdf.sh <src>` typesets any
  USFM/USX/OSIS source (single book or whole Bible) to PDF;
  the complete WEB runs 942 pages, the KJV 853.
  Validated against the complete World English Bible (81
  books through all three surfaces: USFM byte-level fixed
  point with litos identity, USX import litos-identical to
  the USFM-derived kanon, OSIS import→export fixed point,
  well-formed XML throughout) and against the whole-Bible
  KJV OSIS from a foreign toolchain (Haiola, 10 MB, 66 books
  in one file — imports in ~0.2 s and passes the same
  battery).
- Strict error handling: every error condition enumerated in the
  spec's Parsing chapter maps to a typed error with file/line/
  column and, for transcluded content, the inclusion chain.

The spec's worked-example appendix ("From Deltos to Litos") is the
golden test (`crates/atrep/tests/golden.rs`), including a
kanonizo idempotency check. Remote fetching is covered by
`tests/remote.rs` against an ephemeral localhost listener.

Not yet implemented:

- The `.endo` file format (still unspecified by the spec; the
  Markdown and HTML importers are engine-side, as the spec
  allows).

Known Markdown-pilot limitations: exported lists are loose (a
blank line between items); ordered lists must start at 1 (taxis
validation); text paragraphs beginning with a literal list
marker like `1. ` are not escaped and would re-import as items.

Known parsing limitation: paragraph boundaries are detected
line-first, so a verbatim inline (`@@"..."@@`) cannot span a line
that itself looks like a block construct.

## Spec gaps found while implementing

Decisions this pilot had to make where spec v0.10 is silent;
flagged as input for the next spec revision:

1. **Kanon blank-line layout.** "Removes extraneous blank lines"
   does not say which blank lines remain. Pilot rule: one blank
   line between sibling blocks, none around compact single-line
   blocks (enmedia and monosim-shaped forms) — this reproduces
   the spec's worked-example appendix byte-for-byte.
2. **Bracket-matching disablement** has no declared syntax in
   `.lektos` files. The pilot infers it from the ostensive
   definition: an unflipped shown episymbol means matching is
   disabled.
3. **Onym-referencing monosims** are not marked as such in sim
   definitions. The pilot treats any monosim parameter that
   matches a declared onym as a reference (and rewrites it during
   onym canonicalization).
4. **Axioma definitions in the kanon**: the spec says references
   are expanded but not whether definitions remain. The pilot
   removes them (the kanon is fully expanded, so they are dead
   weight).
5. **Dialektos-defined stichoi**: resolved in spec v0.10.1 — the
   `stichos` ostensive keyword (in place of `grammata`) declares
   line-structured grammata, implemented here. Stichos whitespace
   is likewise resolved (litogramma feedback F2): leading and
   internal whitespace is content, preserved through kanonizo and
   litosis (so it participates in the litos ID); only trailing
   whitespace is trimmed. One caveat: a monosim removed by litosis
   mid-stichos leaves its surrounding spaces as written — in verse,
   spacing is content, so no re-normalization is applied.
6. **Version pinning in the kanon**: the pilot preserves the
   declared version specifier verbatim rather than pinning the
   resolved version.
7. **Remote transclusion fetch policy** is specified for media
   only. The pilot applies the media policy to transclusion too:
   same timeout/retry configuration, and an unavailable resource
   fails kanonizo.
8. **Remote documents have no local directory.** Their dialektos
   declarations and any relative transclusion targets resolve
   against the *including* document's directory.
9. **`.lektos` sim ordering**: the spec says "alphabetically by
   sim name" without a collation. The pilot sorts byte-wise on
   the UTF-8 name, with the symbol (byte-wise) as tie-break —
   duplicate names are legal via aliased imports.
10. **Canonical `.lektos` layout** is otherwise unpinned. The
    pilot emits the declaration, then blank-line-separated sim
    blocks; property sims in ASCII symbol order (`"` short
    description, `""` long description, `^` parent); long
    descriptions always in block form with the body verbatim.
11. **Axioma-enlexis to onymized simmeres**: the spec states the
    before-first-reference rule for axiomata only, so the pilot
    allows forward references to simmere onyms. The verbatim copy
    is the simmere's canonical serialization minus its own onym;
    onyms and taxis nested inside the copy keep their source
    form; an onym whose only reference was the expanded enlexis
    is then dropped as unreferenced.
12. **Standard-library resolution**: spec resolution is
    local-only and defines no standard library. The pilot falls
    back to definitions embedded in the binary (`at-markdown`
    and its `md` exomorphosis) when no local file resolves.
13. **The `at-markdown` sim inventory**: the spec's inheritance
    examples pin `#`..`######`, `-`, and `*` (emphasis) but no
    full definition. The pilot adds `**` (strong), `>`
    (blockquote), and `.` (ordered item, required taxis), and
    maps inline code, fenced code, and images to the core
    enlexis/enmedia forms rather than new sims.
14. **Monosim ostensive keyword**: the spec's examples show
    `(param)`, but litogramma uses semantic keywords (`(onym)`,
    `(depth)`, `(key)`). The pilot accepts any identifier except
    the reserved `taxis` and preserves it in the canonical form.
15. **Comments in definition files**: the atrep-meta chapter
    does not say which comment forms `.dia`/`.lektos` files may
    carry. The pilot accepts line (`@@/`) and block
    (`@@@/ ... /@@@`) comments, both removed by kanonizo.

## Development

```
cargo test        # unit + golden + error-condition tests
cargo clippy      # lint (kept warning-free)
cargo fmt         # rustfmt (kept clean)
```

