Metadata-Version: 2.4
Name: fsdata
Version: 0.0.9
Summary: Simple data access layer over fsspec
Keywords: data-access,pathlib,fsspec
Author: Furechan
Author-email: Furechan <furechan@xsmail.com>
License-Expression: MIT
License-File: LICENSE.txt
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Dist: universal-pathlib
Requires-Dist: pyarrow
Requires-Dist: click
Requires-Dist: typing-extensions
Requires-Dist: adlfs>=2025.1.0 ; extra == 'adl'
Requires-Dist: gcsfs>=2025.1.0 ; extra == 'gcs'
Requires-Dist: pandas ; extra == 'pandas'
Requires-Dist: polars ; extra == 'polars'
Requires-Dist: s3fs>=2025.1.0 ; extra == 's3'
Requires-Python: >=3.10
Provides-Extra: adl
Provides-Extra: gcs
Provides-Extra: pandas
Provides-Extra: polars
Provides-Extra: s3
Description-Content-Type: text/markdown

# Simple data catalog library for python

This project is a trivial attempt at offering basic catalog functionality for structured datasets stored in local or remote folders. The library uses `universal_pathlib` to access remote storage locations like S3, Google Cloud Storage, etc ... The library reads a config file called `fsdata.ini` which defines a list of collections, one per section. Each collection corresponds to a local or remote folder containing data files, homogeneous in format: `parquet` collections hold DataFrames (pandas or polars), `json` collections hold plain lists/dicts — declared per collection in the config (`format = json`; parquet is the default). Consumers pick the accessor for the shape they want (`.pandas()`, `.polars()`, `.json()`); the wrong accessor for the collection format raises. The library uses local caching to avoid fetching the same data multiple times.

> **Warning** This project is for exploration only, the interface can change.

## Configuration

The configuration file `fsdata.ini` has one section for each collection, with the section name for name and with a `path` key pointing to its location. The file is resolved in order: the `FSDATA_CONFIG` environment variable, then upward search from the current directory (a repo-local config), then the standard XDG config directory `XDG_CONFIG_HOME` (or ~/.config). The first match wins; configs never merge.

Each collection declares its format with a `format` key (`parquet` or `json`); `parquet` is the default when the key is omitted.

Values support strict `${VAR}` environment-variable interpolation, so a committed repo-local config can declare the collection layout while the storage root stays in the environment (e.g. provisioned by direnv). Unset variables raise a clear error on access — never a silent literal path.

```ini
# fsdata.ini

[samples]
path = ${MY_LAKE}/samples

[datasets]
path = ${MY_LAKE}/datasets
format = parquet

[tickers]
path = ${MY_LAKE}/tickers
format = json
```

To assert that the active config defines the collections your code depends on, fail fast with `fsdata.require("tickers", "datasets")`, or from the shell:

```bash
python -m fsdata check            # validate all configured collections
python -m fsdata check tickers    # validate specific collections
```


## Usage

To access a collection use the `collection` function, or — for collection names that are valid identifiers — plain attribute access on the module.

```python
import fsdata

samples = fsdata.collection("samples")
samples = fsdata.samples                  # same thing
```

To list the configured collections

```python
fsdata.collection_names()
```

To list items in a collection (item names are bare names, without extension)

```python
samples.items()
samples.has("my-sample")
```

To load data, pick the accessor for the shape you want. Each accessor has a single concrete return type, and raises if the collection format does not match.

```python
samples.pandas("my-sample")     # -> pandas.DataFrame   (parquet collections)
samples.polars("my-sample")     # -> polars.DataFrame   (parquet collections)

tickers = fsdata.tickers
tickers.json("DOW30")           # -> plain list or dict (json collections)
```

To save data use the `save` method — the object type must match the collection format: pandas/polars DataFrames go to parquet collections, plain lists and dicts go to json collections. Anything else raises.

```python
samples.save("my-sample", df)          # DataFrame -> .parquet
tickers.save("DOW30", ["MMM", "AXP"])  # list -> .json
samples.remove("my-sample")            # delete an item
```

To inspect a parquet item without loading it, use `metadata` — it reads only the parquet footer, so it stays fast on large files. For the raw artifact bytes, use `read_bytes`.

```python
meta = samples.metadata("my-sample")
meta.num_rows, meta.schema, meta.num_row_groups

raw = samples.read_bytes("my-sample")
```

## Caching

Remote collections (`s3`, `gs`, `az`) keep a local copy of each item they read, so repeated loads do not re-fetch. Local collections are read in place and never cached.

Cached files live in `$XDG_CACHE_HOME/fsdata/<collection>/` (usually `~/.cache/fsdata/<collection>/`), one file per item. The directory is created on first use, and deleting it is always safe.

A cached item is reused without contacting storage for `check_interval` seconds (one day by default). After that, fsdata compares timestamps and re-fetches only if the stored artifact actually changed. Saving or removing an item drops its cached copy immediately.

```python
samples = fsdata.collection("samples")
samples.pandas("my-sample", refresh=True)   # ignore the cache, re-fetch now
```

Set the window per collection when constructing one directly — `check_interval=0` checks storage on every access.

```python
from fsdata import Collection

samples = Collection("samples", "s3://my-bucket/samples", check_interval=0)
```

## Deprecated APIs

The following functions still work but emit a `DeprecationWarning`; new code should use the replacements.

| Deprecated | Use instead |
|---|---|
| `fsdata.collections()` | `fsdata.collection_names()` |
| `fsdata.load(name, item, backend=...)` | `fsdata.collection(name).pandas(item)` / `.polars(item)` / `.json(item)` |
| `Collection.load(item, backend=...)` | `Collection.pandas(item)` / `.polars(item)` / `.json(item)` |

The `backend=` parameter is superseded by the accessor names: instead of selecting the return type with an argument, call the accessor that returns what you want.

## Installation

You can install the package with `pip`

```shell
pip install fsdata
```

The frame backends are extras — install the one(s) matching the accessors you use: `pandas`, `polars`. Cloud storage backends are extras as well: `s3`, `gcs`, `adl` install the required `fsspec` backend.

```shell
pip install "fsdata[pandas,s3]"
```

## Requirements

- pandas and/or polars (each needed only by its own accessor — see extras)
- pyarrow
- universal_pathlib
- fsspec backends like s3fs, etc ... as applicable (see extras)


## Related Projects and Resources
- [intake](https://github.com/intake/intake) - Lightweight package for finding, investigating, loading and disseminating data.
- [pins](https://github.com/rstudio/pins-python) - Publish data sets, models, and other python objects, making it easy to share them across projects and with your colleagues.
- [quilt](https://github.com/quiltdata/quilt) - Quilt is a data mesh for connecting people with actionable data
- [pystore](https://github.com/ranaroussi/pystore) - Fast data store for Pandas time-series data
- [pandas](https://github.com/pandas-dev/pandas) - Flexible and powerful data analysis / manipulation library for Python
- [pyarrow](https://github.com/apache/arrow) - Universal columnar format and multi-language toolbox
- [parquet](https://github.com/apache/parquet-format) - Apache Parquet Format
- [fsspec](https://github.com/fsspec/filesystem_spec) - Filesystem interfaces for Python
- [universal_pathlib](https://github.com/fsspec/universal_pathlib) - pathlib api extended to use fsspec backends

