Metadata-Version: 2.4
Name: rida-format
Version: 0.1.0
Summary: Range Indexed Data Access (RIDA) static data format and library
Author: Syed Ali Hamza Chishty
Maintainer-email: Qasra Foundation <admin@qasra.org>
License-Expression: LicenseRef-Proprietary
Keywords: rida,json,r2,range-requests,static-data
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# RIDA Python library — 0.1

RIDA means **Range Indexed Data Access**. It is an installable Python library
and terminal tool that builds one `.rida` file from a collection of JSON
records. The file contains:

- a fixed RIDA header;
- small, readable UTF-8 metadata at the start of the file;
- a compressed dense or sparse address catalog; and
- UTF-8 JSON values stored in a data region.

The reader loads the catalog once and then reads only the requested field bytes.
It works locally and over an HTTP server that supports byte-range requests.

## Install locally

From this project folder:

```sh
python3 -m pip install .
```

After that, the `rida` command is available in the same Python environment.
The library uses only Python's standard library at runtime: no database, cloud
account, or server process is required.

After the package is published, another person can install it with:

```sh
python3 -m pip install rida-format
```

The public package name is provisionally `rida`; publishing to PyPI will
happen only after the RIDA file format is frozen. The command and Python import
name are both simply `rida`.

### Library project files

Keep the source project in one folder, for example `rida/`:

```text
rida/
├── pyproject.toml          package name and install command
├── MANIFEST.in             files included in the public source archive
├── RIDA-README.md           usage instructions
├── RIDA-DESIGN-0.1.md       RIDA format design
├── rida-metadata.html       optional browser metadata opener
└── src/
    └── rida/
        ├── __init__.py     RIDA builder and reader library
        └── __main__.py     enables: python -m rida
```

`src/rida/` is internal library code. After installation, Python copies it into
its own `site-packages/rida/` location. A developer never needs to run an
individual `.py` file: they use `import rida`, `rida ...`, or `python -m rida`.

## Requirements and downloads

- Python **3.10 or newer** and `pip` are required.
- RIDA has **zero third-party runtime dependencies**. It uses only Python's
  built-in `json`, `lzma`, `hashlib`, `urllib`, and related standard modules.
- The `rida` command has its own progress display for loading, scanning, value
  writing, catalog compression, and final file writing. It does not require
  `tqdm`.
- It does not install or require AWS SDK, Cloudflare SDK, SQLite, Node.js,
  Rust, a database server, or a Worker.
- When installing this source project, `pip` may temporarily download
  `setuptools` (version 68 or newer) to build the package. It is an install
  tool, not a RIDA runtime dependency.
- Official Python installers for macOS and Windows include the required `lzma`
  support. On an unusually minimal Linux Python build, install the operating
  system's `python3-lzma` package before using RIDA.

## Build

Scan first. It reports the field density and recommends dense or sparse mode:

```sh
rida scan users.json
```

Build a RIDA file. The command asks for confirmation unless `--yes` is supplied:

```sh
rida build users.json users.rida \
  --title "Users dataset" \
  --description "Public sample data" \
  --reader-url "https://example.com/rida-explorer" \
  --yes
```

During a build, the terminal shows RIDA's own progress, for example:

```text
Reading JSON bytes: 217,221,288/217,221,288 (100%)
Parsing JSON: 1/1 (100%)
Scanning field names: 39,098/39,098 (100%)
Measuring values: 39,098/39,098 (100%)
Writing value data: 39,098/39,098 (100%)
Compressing catalog: 1/1 (100%)
Writing RIDA file: 217,221,288/217,221,288 (100%)
```

For a folder of JSON record files, loading progress counts record files. For
one large JSON file, it counts source bytes while reading, then shows a
separate parsing stage. RIDA 0.1 still keeps a single JSON source in memory
while parsing; the planned streaming importer is for very large single files.

Input can be:

- an object that maps IDs to JSON record objects;
- an array of JSON record objects, with `--id-key`; or
- a folder containing individual JSON record files. Filenames become IDs.

```sh
rida build records/ records.rida --yes
rida build users-array.json users.rida --id-key id --yes
```

Use `--mode dense` or `--mode sparse` to override the recommendation. The builder does not overwrite an existing output file.

## Read locally

```sh
rida inspect users.rida
rida get users.rida ali city
```

## Read from R2 or another public HTTP URL

After uploading the `.rida` file to an HTTP byte-range server:

```sh
rida remote-get https://data.example.com/users.rida ali city
```

The remote reader first fetches the 128-byte header and catalog, then fetches only the requested field range. For browser readers, configure CORS to allow `GET`, `HEAD`, and the `Range` request header, and expose `Content-Range`, `Content-Length`, and `ETag`.

One `RemoteRidaFile` object keeps that catalog in RAM, so later `.get()` calls on
the same object fetch only the requested value range. The `rida remote-get`
terminal command creates a new object each time, so persistent on-disk catalog
cache is a later RIDA Reader feature.

## Metadata opener page

Open `rida-metadata.html` in a browser and choose any `.rida` file. It reads only the embedded header and metadata on the local computer, then displays the title, record count, index mode, and RIDA Explorer download URL. It never uploads the file.

## Use from Python

```python
import rida

# Scan first if you want to show the publisher the recommendation.
report = rida.scan("users.json")
print(report.recommended_mode)  # "dense" or "sparse"

# Build once. The source JSON stays unchanged.
rida.build(
    "users.json",
    "users.rida",
    title="Users dataset",
    reader_url="https://example.com/rida",
)

# Read only one stored value from a local RIDA file.
with rida.RidaFile("users.rida") as data:
    print(data.get("ali", "city"))

# The same primary lookup works against an R2/public URL with Range support.
remote = rida.RemoteRidaFile("https://data.example.com/users.rida")
print(remote.get("fatima", "languages"))
```

A desktop app or another Python program can show the same stages in its own
interface by passing a callback:

```python
def progress(stage, completed, total):
    print(f"{stage}: {completed}/{total}")

rida.build("users.json", "users.rida", progress=progress)
```

## Current scope

RIDA 0.1 implements primary lookup: `record ID + field name → value`.

The approved design for optional exact-match, numeric-range, and Unicode full-text indexes is in [RIDA-DESIGN-0.1.md](RIDA-DESIGN-0.1.md). Those search-index blocks are the next format layer; they are not claimed as implemented by this library yet.

## License

RIDA is distributed under the RIDA Proprietary License in the `LICENSE` file.
You may download and modify RIDA, and use it in free or paid applications.
You may not sell or redistribute RIDA as a standalone package or library.
Applications that include RIDA must show the attribution notice specified in
the license.
