Metadata-Version: 2.5
Name: pyimporters-csv
Version: 1.6.50
Summary: Sherpa CSV/Excel/TXT knowledge import plugin
Author-email: Olivier Terrier <olivier.terrier@kairntech.com>
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.12
Requires-Dist: openpyxl
Requires-Dist: pandas>=2.0
Requires-Dist: pyimporters-plugins<1.7.0,>=1.6.0
Requires-Dist: structlog
Requires-Dist: xlrd>=2.0.1
Provides-Extra: sbom
Requires-Dist: cyclonedx-bom; extra == 'sbom'
Requires-Dist: pip-audit; extra == 'sbom'
Provides-Extra: test
Requires-Dist: pip; extra == 'test'
Requires-Dist: pytest; extra == 'test'
Requires-Dist: ruff; extra == 'test'
Description-Content-Type: text/markdown

# pyimporters-csv

Sherpa knowledge import plugin for CSV, Excel and plain text files.

It registers three parsers through the `pyimporters.plugins` entry point group:

| Plugin   | Class                                     | Extensions              |
|----------|-------------------------------------------|-------------------------|
| `text`   | `pyimporters_csv.text:TXTKnowledgeParser`  | `txt`, `text`, `zip`    |
| `csv`    | `pyimporters_csv.csv_parser:CSVKnowledgeParser` | `csv`, `tsv`, `zip` |
| `excel`  | `pyimporters_csv.excel:ExcelKnowledgeParser` | `xls`, `xlsx`         |

Each parser yields `Term` objects from `pyimporters_plugins.base`. Archives
(`zip`) are unwrapped transparently for the text and CSV parsers.

## Requirements

- Python 3.12+
- `pyimporters_plugins` (>=1.6.0,<1.7.0), `pandas` (>=2.0), `structlog`, `openpyxl`, `xlrd`

## Installation

```
pip install pyimporters-csv
```

## Usage

```python
from pathlib import Path

from progress.bar import Bar

from pyimporters_csv.csv_parser import CSVKnowledgeParser, CSVOptionsModel

parser = CSVKnowledgeParser()
options = CSVOptionsModel(
    encoding="utf-8",
    identifier_col="ID",
    preferredForm_col="prefLabel_en",
    altForms_cols="altLabel_en",
    multivalue_separator="|",
)
terms = list(parser.parse(Path("terms.csv"), options.model_dump(), Bar("Processing")))
```

### Options

`TXTOptions` (text parser):

| Option     | Default   | Description            |
|------------|-----------|------------------------|
| `encoding` | `utf-8`   | Encoding of the file   |

`CSVOptions` extends `TXTOptions`:

| Option                  | Default          | Description                                                                |
|-------------------------|------------------|----------------------------------------------------------------------------|
| `separator`             | `,`              | Field separator                                                            |
| `quotechar`             | `"`              | Quote character                                                            |
| `multivalue_separator`  | `\|`             | Separator used to split multivalued columns                                |
| `header`                | `0`              | Row number (0-indexed) used as column names, blank if there is no header   |
| `identifier_col`        | `identifier`     | Name of the column used as the concept identifier                          |
| `preferredForm_col`     | `preferredForm`  | Name of the column used as the preferred form                              |
| `altForms_cols`         | `altForms`       | Name of the column used as alternative forms                               |

`ExcelOptions` is the same as `CSVOptions` without `separator` and `quotechar`.

The three `*_col` options take a **column name**, not an index: an unknown name
falls back to the default (column 0 for the identifier, the identifier column for
the preferred form, no alternative forms at all) and logs a warning. Values in
`altForms_cols` are split on `multivalue_separator`, so a single column can hold
several alternative forms.

## Development

The build is driven by [Task](https://taskfile.dev) and [uv](https://docs.astral.sh/uv/),
with the shared stages coming from the `python-archetype` submodule:

```
git submodule update --init
```

```
task stages          # print the pipeline stages, in order
task                 # run the pipeline up to (but excluding) py:publish
task -- --skip-tests # same, without the test stage
task up-to -- lint   # run the pipeline up to and including one stage
task jenkins         # run every stage, exactly what Jenkins runs
```

Individual stages:

| Task                        | Description                                          |
|-----------------------------|------------------------------------------------------|
| `task py:sync`              | Install the workspace and its dependencies (uv sync) |
| `task py:lint`              | `ruff check` and `ruff format --check`               |
| `task py:format`            | Reformat the code with ruff                          |
| `task py:test`              | Run the test suite                                   |
| `task py:test-marker -- <m>`| Run the tests carrying one pytest marker             |
| `task py:sbom`              | Generate a CycloneDX SBOM of the resolved environment|
| `task py:check-vulnerabilities` | Check for known CVEs                             |
| `task py:check-updates`     | Check for dependency updates                         |
| `task py:build`             | Build the wheel and sdist (uv build)                 |
| `task py:publish`           | Publish the distributions (uv publish)               |

`uv.lock` is not versioned here (the Jenkinsfile removes it before building), so
`py:sync` always resolves dependencies from scratch.

### SBOM & vulnerability check

`task py:sbom` and `task py:check-vulnerabilities` wrap the underlying tools; to
run them by hand:

```
uv sync --extra sbom
uv run cyclonedx-py environment -o sbom.cdx.json --output-format json
uv run pip-audit --format json --output audit-report.json
uv run pip-audit --strict   # fail on any known vulnerability
```
