Metadata-Version: 2.4
Name: edgalaxydata-loader
Version: 0.1.0
Summary: Lazily load archived EDDN events from files on https://edgalaxydata.space/EDDN.
License-Expression: MIT
Project-URL: Homepage, https://codeberg.org/jdlbt/edgalaxydata-loader
Project-URL: Issue tracker, https://codeberg.org/jdlbt/edgalaxydata-loader/issues
Project-URL: Source, https://codeberg.org/jdlbt/edgalaxydata-loader
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests
Provides-Extra: dev
Requires-Dist: edgalaxydata-loader; extra == "dev"
Requires-Dist: black==26.5.1; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: flake8==7.3.0; extra == "dev"
Requires-Dist: isort==8.0.1; extra == "dev"
Requires-Dist: pre-commit; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# edgalaxydata-loader

Lazily load archived EDDN events from files on https://edgalaxydata.space/EDDN.

⭐ If you find this tool useful, it's most likely because of the dataset it leverages. Consider [supporting](https://www.patreon.com/EDGalaxyData) the [Elite Dangerous Galaxy Data](https://edgalaxydata.space) project. ⭐

## Installation

    pip install edgalaxydata-loader

By default, the Python standard library's `json` module will be used to parse JSON events (see [Events Output Format](#events-output-format) below). When [orjson](https://pypi.org/project/orjson/) is installed in the environment, it will be used instead of the standard libary module. orjson can be installed alongside edgalaxydata-loader:

    pip install edgalaxydata-loader orjson

## Usage

Define a `EDGalaxyDataLoader` object.

```python
>>> from edgalaxydata_loader import EDGalaxyDataLoader
>>> loader = EDGalaxyDataLoader()
```

Get an iterator over the events of the specified types and the specified date range.

```python
>>> data_types = ["Journal.FSDJump", "Journal.Docked", "Commodity"]
>>> events = loader.events(
...     data_types=event_types,
...     start_date="2026-01-01",
...     end_date="2026-01-04",
... )
```

At this point no data is loaded. The resulting `events` is an iterator that can be consumed inside of any `for` loop.

Getting the first event in the iterator with `next` will load the content of the first data file and yield the first event. The next data file will be loaded only when all the items of the first file have been consumed.

```python
>>> next(events)
{'$schemaRef': 'https://eddn.edcd.io/schemas/journal/1', 'header': {'gamebuild': 'r322188/r0 ', 'gameversion': '4.3.0.1', 'gatewayTimestamp': '2026-01-01T00:00:00.270694Z', 'softwareName': 'EDDiscovery', 'softwareVersion': '19.0.15.0', 'uploaderID': '8b7ecaf6170297bd96744a0d9a82e0d4e5a0903a'}, 'message': {'Body': 'Tyriedgoea IW-Q b32-0 A', 'BodyID': 2, 'BodyType': 'Star', 'Population': 0, 'StarPos': [6430.78125, 43.03125, 2214.84375], 'StarSystem': 'Tyriedgoea IW-Q b32-0', 'SystemAddress': 757122410777, 'SystemAllegiance': '', 'SystemEconomy': '$economy_None;', 'SystemGovernment': '$government_None;', 'SystemSecondEconomy': '$economy_None;', 'SystemSecurity': '$GAlAXY_MAP_INFO_state_anarchy;', 'event': 'FSDJump', 'horizons': True, 'odyssey': True, 'timestamp': '2025-12-31T23:59:56Z'}}
```

Example: Use a `Counter` to count the number of events by ("$schemaRef", "message.event") value pairs.

```python
>>> from collections import Counter
>>> c = Counter((e["$schemaRef"], e["message"].get("event")) for e in events)
>>> c
Counter({('https://eddn.edcd.io/schemas/journal/1', 'FSDJump'): 473693, ('https://eddn.edcd.io/schemas/commodity/3', None): 277061, ('https://eddn.edcd.io/schemas/journal/1', 'Docked'): 262854})
```

### Events output format

By default, the loader returns the events as python dictionaries after decoding the binary data to text and parsing it with `json.loads`. The raw binary format can be obtained by passing `output_format="raw"` when initializing the loader. Alternatively, the decoded JSON string can be obtained by passing `output_format="string"`.

```python
>>> raw_loader = EDGalaxyDataLoader(output_format="raw")
>>> raw_events = raw_loader.events(
...     data_types=event_types,
...     start_date="2026-01-01",
...     end_date="2026-01-04",
... )
>>> next(raw_events)
b'{"$schemaRef": "https://eddn.edcd.io/schemas/journal/1", "header": {"gamebuild": "r322188/r0 ", "gameversion": "4.3.0.1", "gatewayTimestamp": "2026-01-01T00:00:00.270694Z", "softwareName": "EDDiscovery", "softwareVersion": "19.0.15.0", "uploaderID": "8b7ecaf6170297bd96744a0d9a82e0d4e5a0903a"}, "message": {"Body": "Tyriedgoea IW-Q b32-0 A", "BodyID": 2, "BodyType": "Star", "Population": 0, "StarPos": [6430.78125, 43.03125, 2214.84375], "StarSystem": "Tyriedgoea IW-Q b32-0", "SystemAddress": 757122410777, "SystemAllegiance": "", "SystemEconomy": "$economy_None;", "SystemGovernment": "$government_None;", "SystemSecondEconomy": "$economy_None;", "SystemSecurity": "$GAlAXY_MAP_INFO_state_anarchy;", "event": "FSDJump", "horizons": true, "odyssey": true, "timestamp": "2025-12-31T23:59:56Z"}}\n'
```
