Metadata-Version: 2.4
Name: fsdata
Version: 0.0.7
Summary: Simple data access layer over fsspec
Keywords: data-access,pathlib,fsspec
Author: Furechan
Author-email: Furechan <furechan@xsmail.com>
License-Expression: MIT
License-File: LICENSE.txt
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Dist: universal-pathlib
Requires-Dist: pyarrow
Requires-Dist: click
Requires-Dist: typing-extensions
Requires-Dist: adlfs>=2025.1.0 ; extra == 'adl'
Requires-Dist: gcsfs>=2025.1.0 ; extra == 'gcs'
Requires-Dist: pandas ; extra == 'pandas'
Requires-Dist: polars ; extra == 'polars'
Requires-Dist: s3fs>=2025.1.0 ; extra == 's3'
Requires-Python: >=3.10
Provides-Extra: adl
Provides-Extra: gcs
Provides-Extra: pandas
Provides-Extra: polars
Provides-Extra: s3
Description-Content-Type: text/markdown

# Simple data catalog library for python

This project is a trivial attempt at offering basic catalog functionality for structured datasets stored in local or remote folders. The library uses `universal_pathlib` to access remote storage locations like S3, Google Cloud Storage, etc ... The library reads a config file called `fsdata.ini` which defines a list of collections, one per section. Each collection corresponds to a local or remote folder containing data files, homogeneous in format: `parquet` collections hold DataFrames (pandas or polars), `json` collections hold plain lists/dicts — declared per collection in the config (`format = json`; parquet is the default). Consumers pick the accessor for the shape they want (`.pandas()`, `.polars()`, `.json()`); the wrong accessor for the collection format raises. The library uses local caching to avoid fetching the same data multiple times.

> **Warning** This project is for exploration only, the interface can change.

## Configuration

The configuration file `fsdata.ini` has one section for each collection, with the section name for name and with a `path` key pointing to its location. The config file should be located in the the standard XDG config directory `XDG_CONFIG_HOME` (or ~/.config).

Each collection declares its format with a `format` key (`parquet` or `json`); `parquet` is the default when the key is omitted.

```ini
# fsdata.ini

[samples]
path = s3://my-bucket/samples

[datasets]
path = s3://my-bucket/datasets
format = parquet

[tickers]
path = s3://my-bucket/tickers
format = json
```


## Usage

To access a collection use the `collection` function, or — for collection names that are valid identifiers — plain attribute access on the module.

```python
import fsdata

samples = fsdata.collection("samples")
samples = fsdata.samples                  # same thing
```

To list the configured collections

```python
fsdata.collection_names()
```

To list items in a collection (item names are bare names, without extension)

```python
samples.items()
samples.has("my-sample")
```

To load data, pick the accessor for the shape you want. Each accessor has a single concrete return type, and raises if the collection format does not match.

```python
samples.pandas("my-sample")     # -> pandas.DataFrame   (parquet collections)
samples.polars("my-sample")     # -> polars.DataFrame   (parquet collections)

tickers = fsdata.tickers
tickers.json("DOW30")           # -> plain list or dict (json collections)
```

To save data use the `save` method — the object type must match the collection format: pandas/polars DataFrames go to parquet collections, plain lists and dicts go to json collections. Anything else raises.

```python
samples.save("my-sample", df)          # DataFrame -> .parquet
tickers.save("DOW30", ["MMM", "AXP"])  # list -> .json
```

## Deprecated APIs

The following functions still work but emit a `DeprecationWarning`; new code should use the replacements.

| Deprecated | Use instead |
|---|---|
| `fsdata.collections()` | `fsdata.collection_names()` |
| `fsdata.load(name, item, backend=...)` | `fsdata.collection(name).pandas(item)` / `.polars(item)` / `.json(item)` |
| `Collection.load(item, backend=...)` | `Collection.pandas(item)` / `.polars(item)` / `.json(item)` |

The `backend=` parameter is superseded by the accessor names: instead of selecting the return type with an argument, call the accessor that returns what you want.

## Installation

You can install the package with `pip`

```shell
pip install fsdata
```

You can specify any of the extra dependencies `s3`, `gcs`, `adl` to install the required `fsspec` backends.

```shell
pip install "fsdata[s3]"
```

## Requirements

- pandas and/or polars (each needed only by its own accessor)
- pyarrow
- universal_pathlib
- fsspec backends like s3fs, etc ... as applicable


## Related Projects and Resources
- [intake](https://github.com/intake/intake) - Lightweight package for finding, investigating, loading and disseminating data.
- [pins](https://github.com/rstudio/pins-python) - Publish data sets, models, and other python objects, making it easy to share them across projects and with your colleagues.
- [quilt](https://github.com/quiltdata/quilt) - Quilt is a data mesh for connecting people with actionable data
- [pystore](https://github.com/ranaroussi/pystore) - Fast data store for Pandas time-series data
- [pandas](https://github.com/pandas-dev/pandas) - Flexible and powerful data analysis / manipulation library for Python
- [pyarrow](https://github.com/apache/arrow) - Universal columnar format and multi-language toolbox
- [parquet](https://github.com/apache/parquet-format) - Apache Parquet Format
- [fsspec](https://github.com/fsspec/filesystem_spec) - Filesystem interfaces for Python
- [universal_pathlib](https://github.com/fsspec/universal_pathlib) - pathlib api extended to use fsspec backends

