Metadata-Version: 2.5
Name: livivo-converter
Version: 1.0.0
Summary: Convert preprint metadata exports into the import XML-format for LIVIVO
Project-URL: Homepage, https://codeberg.org/pixls/livivo-converter
Project-URL: Repository, https://codeberg.org/pixls/livivo-converter
Project-URL: Issues, https://codeberg.org/pixls/livivo-converter/issues
Project-URL: Changelog, https://codeberg.org/pixls/livivo-converter/src/branch/main/CHANGELOG.md
Author-email: Benjamin Wolff <wolff@zbmed.de>
License-Expression: MIT
License-File: LICENSE
License-File: LICENSES/CC-BY-4.0.txt
License-File: LICENSES/CC0-1.0.txt
License-File: LICENSES/MIT.txt
Keywords: livivo,metadata,open science,preprints,scholarly communication
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.13
Requires-Dist: glom>=24.11.0
Requires-Dist: pyyaml>=6.0.2
Description-Content-Type: text/markdown

# livivo-converter

[![Python](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/)
[![Version](https://img.shields.io/badge/version-1.0.0-informational.svg)](https://codeberg.org/pixls/livivo-converter)
[![License: MIT](https://img.shields.io/badge/code%20license-MIT-green.svg)](https://codeberg.org/pixls/livivo-converter/src/branch/main/LICENSE)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://black.readthedocs.io/)

Get a new metadata source into LIVIVO by writing a mapping, not code.

`livivo-converter` reads a JSONL export, one record per line, and writes the XML that the
LIVIVO import expects. What makes a given source work is not Python: it is a single YAML
mapping file saying which LIVIVO field gets which value, which records to leave out and
which count as withdrawn. The mapping format is a small DSL of its own, with paths into
the record, format strings and conditions, and it carries the whole translation. So
bringing a new source into LIVIVO is one YAML file, and the converter stays untouched.

```console
$ livivo-converter export.jsonl livivo.xml --source biorxiv
INFO - Wrote livivo.xml.gz: 1284 documents, 3 delete entries, 0 skipped
```

The mappings that ship with the package cover preprint servers, and their input is the
kind of file produced by [pixls-preprints](https://codeberg.org/pixls/pixls-preprints),
which harvests bioRxiv, medRxiv and OSF. The two together form one pipeline: harvest to
JSONL, convert to LIVIVO XML. Nothing in the converter itself is specific to preprints:
any source you can export as JSONL can be mapped the same way.

## Why use it

- **New sources without new code.** A source is a YAML mapping file. Adding one is a
  change to configuration, not to the converter.
- **Filtering is part of the mapping.** A `skip` condition decides what never gets
  imported. The OSF mapping uses it as a life-science relevance filter, keeping some
  preprint providers wholesale and others only for medical and life-science subjects.
- **Withdrawn preprints are handled, not dropped.** Records matching a mapping's
  `withdrawal` condition become delete entries, which tell LIVIVO to remove a record it
  has already imported.
- **Mistakes surface early.** Mappings are schema-checked when loaded, and a missing
  input file or an unknown source ends the run before any input is read, rather than part
  way through a large export.
- **Output the importer accepts.** Field text is cleaned on the way out, with control
  characters removed and line breaks replaced by the marker the import understands.
- **Compressed by default.** Output is gzipped unless you ask for plain XML.

## Requirements

- Python 3.13 or newer (see `requires-python` in [`pyproject.toml`](https://codeberg.org/pixls/livivo-converter/src/branch/main/pyproject.toml))
- [uv](https://docs.astral.sh/uv/) to install from source or to work on the converter

## Installation

Run the package as a module from a source checkout:

```bash
git clone https://codeberg.org/pixls/livivo-converter.git
cd livivo-converter
uv sync
uv run python -m livivo_converter export.jsonl livivo.xml --source biorxiv
```

Installing the package instead provides a `livivo-converter` command, which takes the same
arguments and brings its mappings with it, so there is nothing else to set up:

```bash
uv pip install livivo-converter
livivo-converter export.jsonl livivo.xml --source biorxiv
```

`uv tool install livivo-converter` puts that command on your path without touching the
environment you are in, and for a single conversion `uvx livivo-converter ...` fetches and
runs it in one go, installing nothing at all.

## Usage

Give the converter an input file, an output file and the source to convert:

```bash
livivo-converter export.jsonl livivo.xml --source biorxiv
```

The output is gzip-compressed and `.gz` is appended to the name you passed, so the command
above writes `livivo.xml.gz`. Pass `--no-zip` for plain XML:

```bash
livivo-converter export.jsonl livivo.xml --source osf --no-zip
# -> livivo.xml
```

From a source checkout the same run is `uv run livivo-converter ...`, which reaches the
same entry point as the module form above.

The source name is the name of its mapping file. Three ship with the package: `biorxiv`,
`medrxiv` and `osf`. An unknown source is rejected before any input is read.

### Options

| Option | Description |
| --- | --- |
| `input_file` | **Required.** File to convert, one JSON record per line (JSONL) |
| `output_file` | **Required.** XML file to write, with `.gz` appended unless `--no-zip` is given |
| `--source` | **Required.** Source to convert, named after its mapping file |
| `--no-zip` | Write uncompressed XML instead of gzip-compressed XML |
| `-f`, `--force` | Overwrite the output file if it exists |
| `-v`, `--verbose` | Log at DEBUG level instead of INFO |

An existing output file is not overwritten. The run stops before any input is read unless
`-f` is given, and for a compressed run it is the `.gz` file that is checked, since that is
the one being written.

A run ends with one line counting what was written, deleted and skipped. Which records
those were is DEBUG detail, since a relevance filter is meant to drop the greater part of
an export. Pass `-v` and each skipped record is reported with the values its skip rule
tested, which is usually enough to see why it was left out, and each converted record with
the fields as they were built.

## Mapping files

Each source is described by one YAML file in
[`livivo_converter/mappings/`](https://codeberg.org/pixls/livivo-converter/src/branch/main/livivo_converter/mappings). A mapping says which LIVIVO
field gets which value, which records to leave out and which count as withdrawn:

```yaml
source: biorxiv

settings:
  delimiters: # split these input fields into lists first
    authors: ";"
  skip_values: # values that count as no value
    - "NA"

fields:
  DATABASE:
    val: PIXLS # a constant
  TITLE:
    map: title # copied from the record
  PUBLYEAR:
    map: date
    spec: T[:4] # sliced on the way through
  DOCUMENTURL:
    fstring: # built from a format string
      - "url_oa|https://doi.org/{doi}"
```

Paths such as `attributes.subjects.*.0.text` are [glom](https://glom.readthedocs.io)
expressions into the record, so nested and repeated values are reachable without code.
Conditions for `skip` and `withdrawal` are built from `all`, `any` and `not` around leaves
that test a field with `equals`, `in`, `any_in` or `regex`.

See [`docs/mapping-format.md`](https://codeberg.org/pixls/livivo-converter/src/branch/main/docs/mapping-format.md) for the full reference.

## Output format

A root `add` element holding one `doc` per record, each a flat list of `field` elements
carrying the value and the name of the field it belongs to:

```xml
<add>
  <doc>
    <field name="DATABASE">PIXLS</field>
    <field name="SUBDATABASE">biorxiv</field>
    <field name="TITLE">Toy example study on synthetic cell behavior</field>
    <field name="AUTHOR">Example, A.</field>
    <field name="AUTHOR">Sample, B.</field>
  </doc>
</add>
```

Withdrawn records travel in the same file as delete entries, which name only the record to
remove:

```xml
<doc><entry><field name="id">PIXLS-biorxiv-10.1101/2025.07.15.664985</field></entry></doc>
```

## Development

The [`Makefile`](https://codeberg.org/pixls/livivo-converter/src/branch/main/Makefile) wraps the common tasks:

```bash
make test                  # run the test suite
make reformat              # black, rewriting files
make check-formatting      # black, in check mode
make check-code            # flake8
make check-coverage        # pytest with coverage
make audit                 # formatting + lint + tests, the pre-merge gate
```

### Project layout

```
livivo_converter/
├── cli.py          # argument parsing, logging, checks before work starts
├── mappings.py     # loading and schema validation of a mapping
├── convert.py      # executes a mapping over the input records
├── formatter.py    # resolves glom placeholders inside format strings
├── xml_builder.py  # turns mapped fields into the LIVIVO XML tree
└── mappings/       # one YAML mapping per source
    ├── biorxiv.yaml
    ├── medrxiv.yaml
    └── osf.yaml
```

## Getting help

- **Something broken, or a source converting wrongly?** Open an issue at
  [codeberg.org/pixls/livivo-converter/issues](https://codeberg.org/pixls/livivo-converter/issues).
- **Writing or changing a mapping:** [`docs/mapping-format.md`](https://codeberg.org/pixls/livivo-converter/src/branch/main/docs/mapping-format.md) is
  the reference, and the three mappings that ship with the package are worked examples.
- **A record that did not come out as expected:** run with `--verbose`. Every skipped
  record is logged with the values its skip rule tested, and every converted one with the
  fields as they were built, which is usually enough to see which rule or path is at fault.

## Contributing

Contributions are welcome. Please read [`CONTRIBUTING.md`](https://codeberg.org/pixls/livivo-converter/src/branch/main/CONTRIBUTING.md) for the
workflow, coding conventions and what `make audit` expects before you open a pull request.

## Maintainers

Developed and maintained by Benjamin Wolff (<wolff@zbmed.de>) at
[ZB MED – Information Centre for Life Sciences](https://www.zbmed.de/), as part of the
PIXLS project.

## Funding

<a href="https://gepris.dfg.de/project/492813820/2?lang=en" style="border: 2px solid #26509E;">
  <img src="https://codeberg.org/pixls/livivo-converter/raw/branch/main/img/dfg-funded.jpg" alt="Funded by the Deutsche Forschungsgemeinschaft (DFG)" width="70%">
</a>

Developed in [PIXLS – Preprint Information eXtraction for Life Sciences](https://gepris.dfg.de/project/492813820/2?lang=en),
funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation).

## License

Source code is licensed under the [MIT License](https://codeberg.org/pixls/livivo-converter/src/branch/main/LICENSE). Documentation and other content
are licensed under [CC BY 4.0](https://codeberg.org/pixls/livivo-converter/src/branch/main/LICENSES/CC-BY-4.0.txt), and configuration and generated
files under [CC0 1.0](https://codeberg.org/pixls/livivo-converter/src/branch/main/LICENSES/CC0-1.0.txt). See [`LICENSING.md`](https://codeberg.org/pixls/livivo-converter/src/branch/main/LICENSING.md) for
details.
