Metadata-Version: 2.4
Name: asn2rdf
Version: 1.0.0
Summary: Generates OWL, SHACL shapes and SKOS concepts from ASN.1 modules, with the mapping model of xsd2rdf
License-Expression: EUPL-1.2
License-File: LICENSE
Author: Mathias Vanden Auweele
Author-email: mathias@matdata.eu
Requires-Python: >=3.12
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: File Formats
Requires-Dist: asn1tools (>=0.166)
Requires-Dist: xsd2rdf (>=2.1.0,<3)
Project-URL: Changelog, https://gitlab.com/era-europa-eu/private/OPDU/SOT/era-ontology/telematics/asn2rdf/-/blob/main/CHANGELOG.md
Project-URL: Homepage, https://gitlab.com/era-europa-eu/private/OPDU/SOT/era-ontology/telematics/asn2rdf
Project-URL: Repository, https://gitlab.com/era-europa-eu/private/OPDU/SOT/era-ontology/telematics/asn2rdf
Description-Content-Type: text/markdown

# ASN2RDF

[![License](https://img.shields.io/badge/license-EUPL_1.2-blue)](http://data.europa.eu/eli/dec_impl/2017/863/oj)

A tool to convert ASN.1 modules into SHACL shapes, an OWL ontology and SKOS
concept schemes, with the mapping model of [xsd2rdf](https://gitlab.com/era-europa-eu/public/interoperable-data-programme/era-ontology/xsd2rdf).

## Overview

asn2rdf reads ASN.1 modules with [asn1tools](https://pypi.org/project/asn1tools/)
and maps each construct onto the construct xsd2rdf produces for its XML Schema
counterpart: a SEQUENCE is handled like a complexType, a CHOICE like a strict
xs:choice, an ENUMERATED like an enumeration, and so on. It then uses xsd2rdf
2.1 itself for everything after the translation: the shape model, the OWL
derivation from the shapes, the `-k` naming policy, the SKOS writer, the output
file names and the SHACL-SHACL check. The output of both tools can therefore
be merged and treated alike.

asn2rdf knows nothing about a particular domain. Namespaces and naming are
options; domain-specific fixes belong in the pipeline that uses the output.

## Installation

```bash
pip install asn2rdf
```

From source:

```bash
python -m pip install poetry
poetry install
```

## Basic usage

```bash
asn2rdf convert path/to/module.asn -o out
```

This writes, as xsd2rdf does for an XSD:

- `module.asn.shape.ttl` (SHACL shapes)
- `module.asn.owl.ttl` (OWL ontology)
- `module.asn.skos.<Name>.ttl` (SKOS concept schemes, one file per ENUMERATED)
- `module.asn.provenance.ttl` with `--source-provenance`

Several files or folders (their `*.asn` and `*.asn1` files) can be given; each
file is converted on its own, and the other files of the same call are used to
resolve its `IMPORTS`. `python -m asn2rdf` works as well.

### From Python

```python
from asn2rdf import convert

result = convert("module.asn", property_kind_qualified_iris=True, library=["other.asn"])
result.shacl, result.owl  # the graphs, nothing is written yet
result.skos               # one graph per concept scheme
result.write("out")       # writes the same files as the command line
```

`convert` returns xsd2rdf's `Result`. The settings are the fields of
`asn2rdf.Options`; pass them as keywords or as `convert(path, Options(...))`.
`library` lists other ASN.1 files whose modules `IMPORTS` may refer to.
Progress and warnings go to the `asn2rdf` and `xsd2rdf` loggers.

## Command line parameters

The flags are those of `xsd2rdf convert` where the concept exists.

- `-x, --asn-file`: ASN.1 file to be converted
- `-f, --folder`: folder whose ASN.1 files are converted one by one
- `-o, --output-dir`: output directory (default: the folder of the ASN.1 file)
- `-a, --abbreviations-file`: file with abbreviations that labels keep in capitals, one per line
- `-d, --debug`: debug output
- `-q, --quiet`: only print warnings and errors
- `-n, --namespaced-concepts`: concept IRIs `<ns>concepts/<scheme>/<item>` instead of `<ns>concepts/<scheme>_<item>`
- `-u, --namespace`: namespace of the generated IRIs. An ASN.1 module has no namespace, so the default is `http://example.com/`; shapes go in `<namespace>shapes/`
- `-p, --has-prefix-for-properties`: property IRIs start with `has`
- `-s, --skip-shacl-validation`: skip the check of the shapes against SHACL-SHACL
- `--date YYYY-MM-DD`: date written as `dct:created`, `dct:issued` and `dct:modified` (default: the day of `SOURCE_DATE_EPOCH` if set, else today)
- `--strict`: stop with an error when the shapes are not well-formed SHACL
- `-k, --property-kind-qualified-iris`: append `DP` or `OP` to every property IRI according to its OWL property kind (xsd2rdf's naming policy), so that no IRI is both an `owl:DatatypeProperty` and an `owl:ObjectProperty`
- `--functional-properties` / `--no-functional-properties`: declare `owl:FunctionalProperty` for properties whose shapes all have `sh:maxCount 1`. **Off by default** (xsd2rdf has it on): it is a global axiom derived from a local cardinality
- `--source-provenance`: also write `<file>.provenance.ttl` (see below)
- `--language`: language tag of labels and of the comments read from the module (default `en`)
- `--license IRI`: the licence of the output, the object of `cc:license` on each SKOS concept scheme. The default is the EUPL 1.2 that xsd2rdf writes (`http://data.europa.eu/eli/dec_impl/2017/863/oj`); xsd2rdf puts no licence on the ontology header, so neither does asn2rdf. xsd2rdf has no such option

xsd2rdf options without an ASN.1 counterpart are not offered: `--no-pruning`
(every type assignment of the given file is converted; imported types only when
used), `--source-namespaces` and `--namespace-map` (ASN.1 modules have no
namespace IRI) and `--simple-type-subproperties`.

## Mapping rules

Every rule has a fixture in [`tests/fixtures`](tests/fixtures/README.md) with a
minimal module, the golden output and instance data. `ex:` is the namespace,
`nodeshapes:` and `propertyshapes:` are `<ns>shapes/NodeShape/` and
`<ns>shapes/PropertyShape/`.

### Types and components

| ASN.1 | SHACL | OWL / SKOS | xsd2rdf counterpart |
|---|---|---|---|
| `T ::= SEQUENCE { ... }` or `SET { ... }` | `nodeshapes:T` with `sh:targetClass ex:T`, `sh:name`, and `sh:property` for each component | `ex:T a owl:Class` with `rdfs:label` | named complexType with a sequence |
| `T ::= CHOICE { ... }` | `nodeshapes:T` with `sh:xone ( ... )` over the property shapes of the alternatives (each `sh:minCount 1`, `sh:maxCount 1`) | `owl:Class`; each alternative is a property with domain `ex:T` | named complexType with a strict xs:choice |
| component `c` of `T` | `propertyshapes:T/c`, `sh:path ex:c`, `sh:name` | a property `ex:c` with `rdfs:domain` the owner's class (several owners: `owl:unionOf`) | local element |
| component without `OPTIONAL`/`DEFAULT` | `sh:minCount 1`, `sh:maxCount 1` | | `minOccurs="1"` |
| `OPTIONAL` | `sh:minCount 0`, `sh:maxCount 1` | | `minOccurs="0"` |
| `DEFAULT v` | `sh:minCount 0`, `sh:defaultValue v` typed with the datatype, or the concept of an ENUMERATED item | | `default="v"` |
| `SEQUENCE OF X`, `SET OF X` | no `sh:maxCount`, `sh:minCount 0` (an empty list leaves no triple), the constraints of `X` on each value; `SIZE (a..b)` gives `sh:minCount a` (unless `OPTIONAL`) and `sh:maxCount b`; as a CHOICE alternative `sh:minCount 1` | | `maxOccurs="unbounded"` |
| component typed by a SEQUENCE, SET or CHOICE type `U` | `sh:nodeKind sh:IRI`, `sh:node nodeshapes:U` | `owl:ObjectProperty`, `rdfs:range ex:U` | element typed by a named complexType |
| inline `c SEQUENCE { ... }`, `SET`, `CHOICE` | nested `nodeshapes:T/c` (with `sh:nodeKind sh:IRI` and `sh:targetClass ex:C`) and on the component `sh:class ex:C`, `sh:node nodeshapes:T/c`; the class is named after the component | `ex:C a owl:Class`, object property with range `ex:C` | element with an anonymous complexType |
| `COMPONENTS OF U` | the components of `U` in place | | |
| extension marker `...` and extension additions | the marker is ignored, the additions are read like other components; shapes are not closed | | |
| `T ::= INTEGER (0..9)`, `T ::= IA5String (SIZE(3))`, `T ::= SEQUENCE OF X` (a value space) | no shape of its own; a component of type `T` gets the built-in datatype and the constraints of `T`, narrowed by its own | no property of its own | named simpleType (xsd2rdf 2.x default) |
| `IMPORTS` | an imported type is resolved in the other files of the call (`library`) and translated when the module uses it | | xs:import with pruning |

### Built-in types and constraints

| ASN.1 | SHACL | OWL range |
|---|---|---|
| `BOOLEAN` | `sh:datatype xsd:boolean` | `xsd:boolean` |
| `INTEGER` | `xsd:integer` | `xsd:integer` |
| `REAL` | `xsd:double` | `xsd:double` |
| `IA5String`, `UTF8String`, `PrintableString`, `VisibleString`, `GeneralString`, `BMPString`, ... | `xsd:string` | `xsd:string` |
| `NumericString` | `xsd:string`, `sh:pattern "^[0-9 ]*$"` | `xsd:string` |
| `OCTET STRING` | `xsd:hexBinary` | `xsd:hexBinary` |
| `BIT STRING` | `xsd:string`, `sh:pattern "^[01]*$"` | `xsd:string` |
| `OBJECT IDENTIFIER`, `RELATIVE-OID` | `xsd:string`, `sh:pattern` of dot-separated numbers | `xsd:string` |
| `NULL` | `xsd:boolean`, `sh:in ( true )`: a presence marker | `xsd:boolean` |
| `UTCTime`, `GeneralizedTime` | `xsd:string` (their lexical form is not that of `xsd:dateTime`) | `xsd:string` |
| `DATE`, `TIME-OF-DAY`, `DATE-TIME`, `DURATION` | `xsd:date`, `xsd:time`, `xsd:dateTime`, `xsd:duration` | same |
| `(a..b)` | `sh:minInclusive a`, `sh:maxInclusive b`, typed with the datatype; `MIN`/`MAX` leave the bound out | |
| `(v)`, `(v1 \| v2)` | `sh:in ( v1 v2 )` | |
| `(v1 \| a..b)` | `sh:or` with one member per item | `rdfs:range` the datatype |
| `SIZE (n)`, `SIZE (a..b)` on a string | `sh:minLength`, `sh:maxLength`; doubled for `OCTET STRING`, whose hexadecimal form has two digits per octet | |
| `FROM ("A".."Z")` | `sh:pattern "^[A-Z]*$"` | |

A list of lists (`SEQUENCE OF SEQUENCE OF X`) cannot be kept in RDF without
an intermediate node; it is flattened to one list of values, with a warning.
A component whose type is neither built in, defined nor imported gets a
property shape without value constraints, with a warning.

### Enumerations

| ASN.1 | SKOS | SHACL |
|---|---|---|
| `T ::= ENUMERATED { a (0), b (1), ... }` | concept scheme `<ns>concepts/T/T` in its own file `<file>.skos.T.ttl`, with `rdfs:label`, `dct:title`, `dct:issued`, `dct:modified`, `cc:license` as xsd2rdf writes them (`cc:license` the EUPL 1.2, or the `--license` IRI) | |
| item `a (0)` | concept `<ns>concepts/T_a` (`-n`: `<ns>concepts/T/a`), `skos:notation "a"^^xsd:token` (the identifier, used in the value notation and the XML and JSON encodings) and `skos:notation 0` (the number, used in the binary encodings), `skos:prefLabel` from the identifier, `skos:inScheme` and `skos:topConceptOf` the scheme | |
| component typed by `T` | | `sh:nodeKind sh:IRI`, `sh:class skos:Concept` and the SPARQL constraint on `skos:inScheme` that xsd2rdf writes for an enumeration; OWL: `owl:ObjectProperty` with range `skos:Concept` |
| inline `c ENUMERATED { ... }` | a scheme named after the component (`<ns>concepts/c/c`); another inline enumeration with the same component name and other items gets the owner's name in front | as above |

Items without a number get the numbers ASN.1 assigns. Items after the
extension marker are included.

### Documentation

ASN.1 has no annotation syntax, so comments are the documentation of a module.
asn1tools drops them; asn2rdf scans the module once more and attaches:

- the comment lines directly above a type assignment, component or enumeration
  item (a blank line or a line with code ends the block), and
- a comment after the item on the same line, continued by the comment lines
  right below it that are indented at least as far,

as `sh:description` of the shape (and so `rdfs:comment` of the class or
property), or `skos:definition` of a concept scheme or concept. Banner lines
made only of `#`, `=`, `*`, `-` or `.` are dropped. Labels (`sh:name`,
`rdfs:label`, `skos:prefLabel`) are made from the names as in xsd2rdf
(`issuingDay` becomes "Issuing day"; `-a` keeps abbreviations in capitals).

### Same value in two forms

Some modules carry one value in two forms of which only one is to be present,
for example `productOwnerNum INTEGER` and `productOwnerIA5 IA5String`. asn2rdf
maps them as written: two properties, each with its own datatype and
constraints. Merging them into one property is a modelling decision for the
ontology that uses the output; the provenance file gives the ASN.1 identifier
and type of each.

### Ontology

The OWL file declares `<namespace> a owl:Ontology` with `rdfs:label`,
`dct:title` and `dct:description`
`Ontology generated from the ASN.1 module <name>` and the date as `dct:created`, `dct:issued` and `dct:modified`. The
classes, properties, domains and ranges are derived from the shapes by xsd2rdf;
see its README.

## Source provenance

With `--source-provenance`, `<file>.provenance.ttl` links each shape, class,
property and concept scheme to the ASN.1 component it was made from with
`dct:source` (`xsd:anyURI`): the file, relative to the converted file, and the
ASN.1 absolute reference of X.680 clause 14, for example

```
uicRailTicketData_v3.0.3.asn#@ASN-Module-RailTicketData.DocumentData.ticket.reservation
```

where `*` stands for the element of a SEQUENCE OF, as in
`...Document.legs.*.km`. Each property shape also gets the identifier and the
type of its component as written (`asn2rdf:asnIdentifier "productOwnerNum"`,
`asn2rdf:asnType "INTEGER"` or `"SEQUENCE OF TicketLinkType"`), so that data
can be mapped back to the ASN.1 structure. The other output files do not
change.

## The asn2rdf vocabulary

`asn2rdf:asnIdentifier` and `asn2rdf:asnType` are annotation properties in the
namespace `https://w3id.org/asn2rdf#`, defined in
[`vocabulary/asn2rdf.ttl`](vocabulary/asn2rdf.ttl). They are informative only
and have no logical meaning. The provenance file declares them too.

## Tests

```bash
poetry install
poetry run pytest
```

The mapping rules are covered by the fixture suite in
[`tests/fixtures`](tests/fixtures/README.md): one folder per rule with a minimal
module, the golden output and instance data that must (not) conform to the
generated shapes. After an intended change of the output, regenerate the golden
files with `poetry run pytest --update-golden` and review the diff.
[`tests/golden/b12`](tests/golden/b12/README.md) is a golden test on real
modules: the B12 modules (UIC rail ticket data) of the ERA telematics ontology.
`ruff check src tests` and `mypy` must pass.

The fixtures are also published as a documentation site on GitLab Pages,
built by the `pages` job on `main`: one page per case with the ASN.1 source,
the golden output, the instance data and UML-like class diagrams of the source,
the SHACL shapes and the OWL ontology, plus the mapping rules above and a
summary of the B12 golden test. The site is at
[https://era-europa-eu.gitlab.io/private/OPDU/SOT/era-ontology/telematics/asn2rdf/](https://era-europa-eu.gitlab.io/private/OPDU/SOT/era-ontology/telematics/asn2rdf/)
(Deploy → Pages in GitLab shows the address if the project uses a unique Pages
domain; the site is visible to whoever can see the project). To build it
locally (Mermaid then loads from a CDN):

```bash
poetry run python docs/build_site.py public
```

## Releasing

Releases are published to [PyPI](https://pypi.org/project/asn2rdf/) from GitLab CI.

1. Update `CHANGELOG.md`: move the changes to a new `[X.Y.Z] - YYYY-MM-DD`
   section and merge it to `main`.
2. In GitLab, create a release (Deploy → Releases → New release) with a new
   tag `vX.Y.Z` created from `main`. The release notes can go in its
   description.
3. The tag pipeline runs lint and the tests, sets the package version from the
   tag (`v1.2.3` gives `1.2.3`), builds the wheel and the sdist, and uploads
   them to PyPI with trusted publishing: no PyPI password or token is stored in
   GitLab. Only tags of the form `vX.Y.Z` (or `X.Y.Z`) build and publish.

The PyPI project has a GitLab trusted publisher with these settings (before
the first release, when the project does not exist on PyPI yet, add it as a
pending publisher under Your account → Publishing):

| Setting | Value |
|---|---|
| Namespace | `era-europa-eu/private/OPDU/SOT/era-ontology/telematics` |
| Project name | `asn2rdf` |
| Top-level pipeline file path | `.gitlab-ci.yml` |
| Environment name | (empty) |

## License

[EUPL 1.2](http://data.europa.eu/eli/dec_impl/2017/863/oj)

