Metadata-Version: 2.4
Name: ftm-lakehouse
Version: 0.8.2
Summary: Data standard, storage and retrieval for structured and unstructured FollowTheMoney data
License: AGPLv3+
License-File: LICENSE
License-File: NOTICE
Author: Data and Research Center
Author-email: hi@dataresearchcenter.org
Requires-Python: >=3.12,<3.15
Classifier: License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Provides-Extra: api
Provides-Extra: azure
Provides-Extra: gcs
Provides-Extra: http
Provides-Extra: postgres
Provides-Extra: s3
Requires-Dist: adbc-driver-postgresql (>=1.12,<2) ; extra == "postgres"
Requires-Dist: adlfs ; extra == "azure"
Requires-Dist: aiohttp (!=4.0.0a0,!=4.0.0a1) ; extra == "api"
Requires-Dist: aiohttp (!=4.0.0a0,!=4.0.0a1) ; extra == "http"
Requires-Dist: anystore (>=1.4.0,<2.0.0)
Requires-Dist: fastapi (>=0.138.2,<0.139.0) ; extra == "api"
Requires-Dist: followthemoney (>=4.11.0,<5.0.0)
Requires-Dist: ftmq[lake] (>=5.1.0,<6.0.0)
Requires-Dist: gcsfs (>2024.2.0) ; extra == "gcs"
Requires-Dist: granian (>=2.7.7,<3.0.0) ; extra == "api"
Requires-Dist: httpx (>=0.28.1,<0.29.0)
Requires-Dist: psycopg2-binary (>=2.9,<3) ; extra == "postgres"
Requires-Dist: putfs (>=0.4.0,<0.5.0) ; extra == "api"
Requires-Dist: putfs (>=0.4.2,<0.5.0)
Requires-Dist: s3fs (>2024.2.0) ; extra == "s3"
Requires-Dist: uvloop (>=0.22.1,<0.23.0) ; extra == "api"
Requires-Dist: zfs-agent (>=0.0.1,<0.0.2)
Project-URL: Documentation, https://openaleph.org/docs/lib/ftm-lakehouse
Project-URL: Homepage, https://openaleph.org/docs/lib/ftm-lakehouse
Project-URL: Issues, https://github.com/openaleph/ftm-lakehouse/issues
Project-URL: Repository, https://github.com/openaleph/ftm-lakehouse
Description-Content-Type: text/markdown

[![Docs](https://img.shields.io/badge/docs-live-brightgreen)](https://openaleph.org/docs/lib/ftm-lakehouse)
[![ftm-lakehouse on pypi](https://img.shields.io/pypi/v/ftm-lakehouse)](https://pypi.org/project/ftm-lakehouse/)
[![PyPI Downloads](https://static.pepy.tech/badge/ftm-lakehouse/month)](https://pepy.tech/projects/ftm-lakehouse)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/ftm-lakehouse)](https://pypi.org/project/ftm-lakehouse/)
[![Python test and package](https://github.com/openaleph/ftm-lakehouse/actions/workflows/python.yml/badge.svg)](https://github.com/openaleph/ftm-lakehouse/actions/workflows/python.yml)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)
[![Coverage Status](https://coveralls.io/repos/github/openaleph/ftm-lakehouse/badge.svg?branch=main)](https://coveralls.io/github/openaleph/ftm-lakehouse?branch=main)
[![AGPLv3+ License](https://img.shields.io/pypi/l/ftm-lakehouse)](./LICENSE)
[![Pydantic v2](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/pydantic/pydantic/main/docs/badge/v2.json)](https://pydantic.dev)

# ftm-lakehouse

`ftm-lakehouse` provides a _data standard_ and _archive storage_ for leaked data, private and public document collections and structured [FollowTheMoney](https://followthemoney.tech) data. The concepts and implementations are originally inspired by [mmmeta](https://github.com/simonwoerpel/mmmeta), [Aleph's servicelayer archive](https://github.com/alephdata/servicelayer) and [OpenSanctions](https://opensanctions.org) work on dataset catalog metadata.

`ftm-lakehouse` acts as a multi-tenant storage and retrieval mechanism for structured entity data, documents and their metadata. It can be used by _tenants_ to produce and/or consume data such as [investigraph](https://docs.investigraph.dev), [memorious](https://docs.investigraph.dev/lib/memorious/) and the full suites of various search and analysis platforms, such as [_OpenALeph_](https://openaleph.org), [_ICIJ Datashare_](https://datashare.icij.org/) or [_Liquid Investigations_](https://github.com/liquidinvestigations/)

## Installation

Requires python 3.12 or later.

```bash
pip install ftm-lakehouse
```

Remote storage backends are optional extras – install the one matching your archive/lake URI:

```bash
pip install "ftm-lakehouse[s3]"     # S3-compatible object storage (s3fs)
pip install "ftm-lakehouse[gcs]"    # Google Cloud Storage (gcsfs)
pip install "ftm-lakehouse[azure]"  # Azure Blob Storage (adlfs)
pip install "ftm-lakehouse[http]"   # HTTP(S)-backed api stores (aiohttp)
```

Extras combine, e.g. `pip install "ftm-lakehouse[s3,gcs]"`.

## Documentation

[openaleph.org/docs/lib/ftm-lakehouse](https://openaleph.org/docs/lib/ftm-lakehouse)

## Development

This package is using [poetry](https://python-poetry.org/) for packaging and dependencies management, so first [install it](https://python-poetry.org/docs/#installation).

Clone [this repository](https://github.com/openaleph/ftm-lakehouse) to a local destination.

Within the repo directory, run

    poetry install --with dev

This installs a few development dependencies, including [pre-commit](https://pre-commit.com/) which needs to be registered:

    poetry run pre-commit install

Before creating a commit, this checks for correct code formatting (isort, black) and some other useful stuff (see: `.pre-commit-config.yaml`)

### Testing

`ftm-lakehouse` uses [pytest](https://docs.pytest.org/en/stable/) as the testing framework.

    make test

## Background

The design grew out of the [FollowTheMoney data lake RFC discussion](https://discuss.openaleph.org/t/rfc-followthemoney-data-lake/37) and prior art in [mmmeta](https://github.com/simonwoerpel/mmmeta), [Aleph's servicelayer archive](https://github.com/alephdata/servicelayer), [OpenSanctions](https://opensanctions.org) dataset metadata and [nomenklatura statements](https://followthemoney.tech/docs/statements/).

For contributing, development setup and testing see the [repository README](https://github.com/openaleph/ftm-lakehouse#development).


## License and Copyright

`leakrfc` (_predecessor_), (c) 2024 [investigativedata.io](https://investigativedata.io)

`ftm-lakehouse`, (c) 2024 [investigativedata.io](https://investigativedata.io)

`ftm-lakehouse`, (c) 2025-2026 [Data and Research Center – DARC](https://dataresearchcenter.org)

`ftm-lakehouse` is licensed under the AGPLv3 or later license.

