Metadata-Version: 2.4
Name: eurlex-corpus
Version: 0.1.0
Summary: EU financial regulation as structured, cross-referenced, multilingual data
Author: Naveen
License: MIT
Project-URL: Homepage, https://github.com/chenjigaram/eurlex-corpus
Project-URL: Source, https://github.com/chenjigaram/eurlex-corpus
Keywords: eur-lex,regulation,compliance,regtech,mifid,legal,nlp
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Legal Industry
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28
Requires-Dist: beautifulsoup4>=4.11
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Dynamic: license-file

# eurlex-corpus

EU financial regulation as structured, cross-referenced, multilingual data.

**[What I found building it →](https://chenjigaram.github.io/eurlex-corpus/)**

EUR-Lex publishes every act as HTML behind a SPARQL endpoint. Getting the text out is
solved — [`eurlex`](https://github.com/kevin91nl/eurlex) parses any CELEX document into
a DataFrame and does it well. What is not solved is everything after that: the same act
in 24 languages, aligned paragraph by paragraph, and the citation graph that connects
the acts to each other. That is what this is.

```bash
pip install eurlex-corpus
```

```python
from eurlex_corpus import load

mifid = load("MiFID2", lang="en")

mifid.article("16").title                    # 'Organisational requirements'
mifid.article("16").paragraph(7).text        # 'Records shall include the recording of telephone...'
mifid.article("16").url                      # deep link to the paragraph on EUR-Lex
mifid.article("16").citation                 # '32014L0065 Article 16'
mifid.article("16").heading                  # chapter/section chain + title

mifid.article("16").translations["nl"].paragraph(7).text
# 'Het bijhouden van gegevens omvat het opnemen van telefoongesprekken...'

mifid.article("16").references
# [Reference('Article 17' -> 32014L0065/17),
#  Reference('Article 4 of Directive 2006/73/EC' -> 32006L0073/4),
#  Reference('Regulation (EU) No 600/2014' -> 32014R0600/None), ...]
```

Documents are cached on first use under `~/.cache/eurlex`, so everything after the
first call is local.

## Coverage

| Key | Act | CELEX | Articles | Paragraphs |
|---|---|---|---:|---:|
| `mifid2` | MiFID II — Directive 2014/65/EU | 32014L0065 | 97 | 406 |
| `mifir` | MiFIR — Regulation (EU) No 600/2014 | 32014R0600 | 55 | 231 |
| `priips` | PRIIPs — Regulation (EU) No 1286/2014 | 32014R1286 | 34 | 89 |
| `ucits` | UCITS — Directive 2009/65/EC | 32009L0065 | 119 | 319 |
| `aifmd` | AIFMD — Directive 2011/61/EU | 32011L0061 | 71 | 363 |
| `dora` | DORA — Regulation (EU) 2022/2554 | 32022R2554 | 64 | 254 |
| `mica` | MiCA — Regulation (EU) 2023/1114 | 32023R1114 | 149 | 732 |
| `sfdr` | SFDR — Regulation (EU) 2019/2088 | 32019R2088 | 20 | 48 |

609 articles, 2,442 numbered paragraphs. Any other act works too if you pass its CELEX
id directly — `load("32016R0679")` — you just do not get the short name.

Consolidated texts work the same way: `load("02014L0065-20240328")`.

## The alignment is exact, not approximate

EUR-Lex assigns the same structural ids to every language version of an act. Article 16
is `art_16` in all of them, and its seventh paragraph is `016.007` whether you asked for
English, Dutch, German or French.

That means aligning translations needs no sentence matching and no heuristics. Checked
across all eight acts in all 24 languages — 181 act-language pairs:

| | Pairs | |
|---|---|---|
| Article ids identical to English | **181/181** | 100% |
| Article *and* paragraph ids identical | **179/181** | 98.9% |

```bash
python tools/check_alignment.py
```

The two exceptions are UCITS and AIFMD in Croatian. Croatia joined the EU in 2013, so
acts adopted before then appear in Croatian as a special edition using the older EUR-Lex
layout, which has no structural ids at all. Those parse to the right 119 and 71 articles
but carry no paragraph ids, so they can be aligned by article and not below it.

MiFID II, UCITS and AIFMD have no Irish version — the Irish language derogation ran
until 2022.

```python
en = load("dora", lang="en").article("11").paragraph(1).text
de = load("dora", lang="de").article("11").paragraph(1).text
```

Cellar serves all 24 official EU languages, and `lang=` accepts any of them.

## References work in every language

An EU act is numbered identically in all 24 versions, so the citation itself is
language-independent — only the word in front of it changes, and it inflects:
`Verordnung`, `Verordening`, `nařízení`, `asetuksen`, `рeгламент`. Extraction keys off
the number shape and treats the surrounding word as a hint rather than a requirement.

Because the article ids align exactly, that extractor can be checked without any
annotation: the same article in Dutch must cite the same acts as in English, and any
disagreement is a bug. Across MiFID II and DORA — 161 articles per language:

```bash
python tools/check_languages.py
```

| | Languages |
|---|---|
| 100% | cs, el, en, es, et, fr, nl, pt, ro, sv |
| 98–99% | bg, da, de, fi, hr, it, lt, mt, pl, sk, sl |
| 95–97% | ga, lv |
| below | hu (86%) |

Mean 98.5%, with 23 of 24 languages at or above 95%. Hungarian is the outlier: it
writes `1093/2010/EU rendelet`, putting the act word after the number and the number
before the year, which collides with the directive form used everywhere else.

## The citation graph

References only get you outbound edges. The useful question is usually the reverse one —
*what cites this?* — and EUR-Lex does not answer it.

```python
import eurlex_corpus

g = eurlex.graph()                       # all eight acts
len(g)                                   # 3662 citations, 184 act-to-act edges

g.inbound("32014L0065", "16")            # what cites MiFID II Article 16
g.outbound("32014R0600", "26")           # what MiFIR Article 26 cites
g.most_cited(5, external_only=True)      # most cited from outside their own act
g.dangling()                             # 83 acts cited but not loaded
```

The most-cited provisions across these eight acts, counting only citations from a
different act, are not the famous ones:

| Provision | Cited by other acts |
|---|---:|
| ESMA Regulation Article 19 — binding mediation | 15 |
| ESMA Regulation Article 15 — implementing technical standards | 14 |
| ESMA Regulation Article 16 — guidelines and recommendations | 10 |
| EBA Regulation Article 16 — guidelines and recommendations | 7 |

EU financial regulation leans on the supervisor's rule-making powers more than on any
substantive provision.

```bash
eurlex-corpus graph --top 10
eurlex-corpus graph --cited-by mifid2:16
```

## Command line

```bash
eurlex-corpus list                                  # known instruments
eurlex-corpus fetch                                 # cache all eight, English
eurlex-corpus fetch mifid2 dora --lang en --lang nl
eurlex-corpus show mifid2 16 --paragraph 7
eurlex-corpus show mifid2 16 --json
```

## What it does not do

- No search or retrieval. This is the data layer; ranking is your problem.
- Recitals, annexes and tables are not split out yet — article bodies only.
- Article-level references (`Article 16(7)` and its 23 translations) are extracted
  but not measured; only the act-level citations above are verified.
- Enumerations are expanded ("Articles 10 to 13" counts as four citations), but only
  where the connector is a recognised range word and the span is under 40 articles.
- MiFID II has no Irish version — the Irish derogation ran until 2022, so acts from
  before then are not available in `ga`.
- Not every act titles its articles. PRIIPs and UCITS give none at all, so `.title`
  is empty for all 153 of their articles. That is what the Official Journal says, not
  a parsing gap — use `.heading`, which falls back to the enclosing chapter and
  section titles.
- Cross-references are resolved by pattern, not by an official citation graph.
  Internal references and numbered EU acts resolve reliably; prose like
  "the Directive referred to in the preceding paragraph" does not.

## The dataset

The aligned corpus is published as a dataset in its own right — 56,838 rows, 2,442
provisions across 24 languages, CC BY 4.0:

**[chenjigaram/eu-financial-regulation-aligned](https://huggingface.co/datasets/chenjigaram/eu-financial-regulation-aligned)**

```bash
python tools/build_dataset.py     # rebuild it
python tools/check_alignment.py   # verify the alignment claim
```

## Licence and attribution

The code is MIT. The texts are not mine to license:

> Source: EUR-Lex, © European Union. Reused under Commission Decision 2011/833/EU;
> editorial content is licensed CC BY 4.0.

Available as `eurlex.ATTRIBUTION`. Only the authentic printed Official Journal has legal
value — do not use this for anything where that distinction matters.

## Development

```bash
pip install -e ".[dev]"
pytest
ruff check .
```

Tests run offline against committed HTML fixtures.
