Metadata-Version: 2.4
Name: rida-format
Version: 0.2.1
Summary: Range Indexed Data Access (RIDA) static data format and library
Author: Syed Ali Hamza Chishty
Maintainer-email: Qasra Foundation <admin@qasra.org>
License-Expression: LicenseRef-Proprietary
Project-URL: PyPI, https://pypi.org/project/rida-format/
Project-URL: Changelog, https://pypi.org/project/rida-format/#history
Keywords: rida,json,r2,range-requests,static-data
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# RIDA Python library — 0.2.1

RIDA means **Range Indexed Data Access**. It is an installable Python library
and terminal tool that builds one `.rida` file from a collection of JSON
records. The file contains:

- a fixed 128-byte RIDA header;
- small, readable UTF-8 metadata at the start of the file;
- a compressed dense or sparse address catalog; and
- field values stored in a data region.

The reader loads the catalog once and then reads only the requested field bytes.
It works locally and over an HTTP server that supports byte-range requests.

RIDA 0.2.0 writes the **RIDA 2** layout by default: every stored field value is
its own gzip member. A reader still fetches exactly one value range, then
inflates that one member. On a 27,647-record, 34-field hadith dataset the data
region drops from 191,378,148 to 110,924,973 bytes, and the average stored value
drops from 244 to 143 bytes. Older **RIDA 1** files stay readable, and
`--value-codec plain` still writes the original RIDA 1 layout byte for byte.

## Install locally

From this project folder:

```sh
python3 -m pip install .
```

After that, the `rida` command is available in the same Python environment.
The library uses only Python's standard library at runtime: no database, cloud
account, or server process is required.

Install the published package with:

```sh
python3 -m pip install rida-format
```

The PyPI package name is `rida-format`. After installation, the terminal
command and Python import name are both `rida`. Check the installed version with:

```sh
rida --version
```

### Library project files

Keep the source project in one folder, for example `rida/`:

```text
rida/
├── pyproject.toml           package name and install command
├── MANIFEST.in              files included in the public source archive
├── RIDA-README.md           usage instructions
├── RIDA-FORMAT.md           byte-level file format specification
├── CHANGELOG.md             release history
├── LICENSE                  RIDA license
├── docs/
│   └── rida-metadata.html   optional local metadata helper
├── tests/
│   ├── test_rida.py         test suite
│   ├── make_fixtures.py     regenerates the RIDA 1 golden fixtures
│   └── fixtures/            RIDA 1 files written by the 0.1.1 library
└── src/
    └── rida/
        ├── __init__.py      RIDA builder and reader library
        └── __main__.py      enables: python -m rida
```

`src/rida/` is internal library code. After installation, Python copies it into
its own `site-packages/rida/` location. A developer never needs to run an
individual `.py` file: they use `import rida`, `rida ...`, or `python -m rida`.

## Requirements and downloads

- Python **3.10 or newer** and `pip` are required.
- RIDA has **zero third-party runtime dependencies**. It uses only Python's
  built-in `json`, `gzip`, `lzma`, `hashlib`, `urllib`, and related standard
  modules.
- The `rida` command has its own progress display for loading, scanning, value
  writing, catalog compression, and final file writing. It does not require
  `tqdm`.
- It does not install or require AWS SDK, Cloudflare SDK, SQLite, Node.js,
  Rust, a database server, or a Worker.
- When installing this source project, `pip` may temporarily download
  `setuptools` (version 77.0.3 or newer) to build the package. It is an install
  tool, not a RIDA runtime dependency.
- Official Python installers for macOS and Windows include the required `lzma`
  support. On an unusually minimal Linux Python build, install the operating
  system's `python3-lzma` package before using RIDA.

## Build

Scan first. It reports the field density and recommends dense or sparse mode:

```sh
rida scan users.json
```

Build a RIDA file. The command asks for confirmation unless `--yes` is supplied:

```sh
rida build users.json users.rida \
  --title "Users dataset" \
  --description "Public sample data" \
  --reader-url "https://example.com/rida-explorer" \
  --yes
```

During a build, the terminal shows RIDA's own progress, for example:

```text
Reading JSON bytes: 217,221,288/217,221,288 (100%)
Parsing JSON: 1/1 (100%)
Scanning field names: 39,098/39,098 (100%)
Measuring values: 39,098/39,098 (100%)
Writing value data: 39,098/39,098 (100%)
Compressing catalog: 1/1 (100%)
Writing RIDA file: 217,221,288/217,221,288 (100%)
```

For large inputs, low-memory progress reports the record count as it reads.
When the total is not known in advance, updates appear every 1,000 records.

Input can be:

- an object that maps IDs to JSON record objects;
- an array of JSON record objects, with `--id-key`; or
- a folder containing individual JSON record files. Filenames become IDs.

```sh
rida build records/ records.rida --yes
rida build users-array.json users.rida --id-key id --yes
```

If the input folder has JSON files in more than one place, RIDA lists the
groups and asks which folder contains the records. For example, if a folder
has `book_details.json` at its top level and thousands of record files inside
`hadiths/`, choose the `hadiths/` group. For a command that runs without a
prompt, select it explicitly:

```sh
rida scan "/path/to/musnad-ahmed" --records-dir hadiths
rida build "/path/to/musnad-ahmed" musnad-ahmed.rida \
  --records-dir hadiths --yes
```

`--records-dir` is a folder inside the input folder. RIDA searches that folder
and its subfolders for `.json` files; it does not mix separate groups unless
they are all inside the selected folder. The scan prints the number of records
before a build starts.

Use `--mode dense` or `--mode sparse` to override the recommendation. The
builder does not overwrite an existing output file.

### Choosing a value codec

`--value-codec` selects the stored value layout:

| Value | Header | Stored value | When to use it |
| --- | --- | --- | --- |
| `gzip-per-field` (default) | `RIDA0002` | one gzip member per value | Normal use. Cuts file size and per-lookup transfer when values are large enough. |
| `plain` | `RIDA0001` | plain UTF-8 JSON | Small values, already-compressed values, or when a 0.1.1 or other RIDA 1 reader must open the file. |

```sh
rida build users.json users.rida --value-codec plain --yes
```

Each gzip member carries a CRC-32 and a length, so a damaged RIDA 2 value fails
to inflate instead of returning wrong data. Values that are only a few bytes
long can be larger when gzipped, because a gzip member has about 20 bytes of
overhead; `scan` measures the real stored size for the codec you select, so its
estimates and recommendation match the file you are about to write.

## Memory modes for large inputs

The default `--memory-mode auto` estimates the input's working-memory needs
and checks currently available RAM. Choose a mode explicitly when needed:

```sh
rida build records.json records.rida --id-key id --memory-mode low-memory --yes
rida build records/ records.rida --memory-mode fast --yes
```

`low-memory` reads one JSON record at a time instead of keeping the whole JSON
input in RAM. For one JSON file, supported layouts are an object mapping IDs
to record objects, or an array of record objects with `--id-key`. For a folder,
RIDA reads one record file at a time. The current record and the address
catalog still use RAM, so this is not a zero-memory mode; an individual record
must fit in memory.

When you pass an explicit `--mode dense` or `--mode sparse`, the low-memory
builder stages only that one index layout instead of both, which lowers its
peak catalog memory and CPU.

If the input folder contains separate JSON groups, RIDA asks which group to
use. For non-interactive commands, pass `--records-dir`, for example:

```sh
rida build "/path/to/musnad-ahmed" musnad-ahmed.rida \
  --records-dir hadiths --memory-mode low-memory --yes
```

## Read locally

```sh
rida inspect users.rida
rida get users.rida ali city
```

`inspect` prints the header and the embedded metadata as JSON. `get` prints one
value and exits with status 1 and a one-line message if the record ID or field
name does not exist.

## Check a file for damage

Reading one field never hashes the whole data region, so a normal lookup trusts
the transport. `verify` is the explicit, slower check: it streams every data
byte and compares it with the SHA-256 digest recorded in the header.

```sh
rida verify users.rida
```

```json
{
  "file": "users.rida",
  "format_version": 2,
  "value_codec": "gzip-per-field",
  "record_count": 27647,
  "data_bytes": 110924973,
  "size_ok": true,
  "metadata_ok": true,
  "catalog_ok": true,
  "data_ok": true,
  "ok": true
}
```

`verify` exits with status 1 when `ok` is false. From Python, use
`rida.verify("users.rida")`, or `RidaFile.verify_data()` on an already open file.

## Where RIDA can read data

RIDA 0.2.0 can read a `.rida` file from the local computer or from a direct
HTTP/HTTPS file URL. For remote reads, RIDA sends ordinary HTTP byte-range
requests; it does not use a cloud-provider SDK. This makes it compatible with
object storage that returns the requested bytes as HTTP `206 Partial Content`
and includes the matching `Content-Range` header.

Examples expected to work because their object APIs support byte-range reads
include Cloudflare R2, Amazon S3, Google Cloud Storage, Azure Blob Storage,
Backblaze B2, and Bunny.net through a Pull Zone. See their docs: [R2](https://developers.cloudflare.com/r2/api/s3/api/),
[S3](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html),
[Google Cloud Storage](https://docs.cloud.google.com/storage/docs/downloading-objects),
[Azure Blob Storage](https://learn.microsoft.com/en-us/rest/api/storageservices/get-blob),
[Backblaze B2](https://www.backblaze.com/apidocs/b2-download-file-by-id), and
[Bunny.net](https://bunny.net/faq/).
This is protocol-based expected compatibility, not a separate integration
with each provider. The exact URL, permissions, and any CDN in front of a
bucket still need to allow byte-range responses; the providers have not all
been individually tested with this RIDA release.

For Bunny Storage, connect the Storage Zone to a Pull Zone and use the Pull
Zone URL (or its custom hostname) for the file. Bunny documents range support
for cached content by default; enable Cache Slicing in the Pull Zone for
uncached content.

The current reader sends a `Range` header and a `rida/<version>` User-Agent, but
does not add provider-specific authentication headers. Use a public object URL
or a signed URL that permits the range request. RIDA does not upload files to
cloud storage: upload the `.rida` file using that storage provider, then give
RIDA the file URL.

## Read from R2 or another HTTP URL

After uploading the `.rida` file to an HTTP byte-range server:

```sh
rida remote-get https://data.example.com/users.rida ali city
```

The remote reader fetches the 128-byte header, then the metadata and catalog,
then only the requested field range. It checks that the server returns HTTP
`206` and the correct `Content-Range`; if the server sends the whole file
instead, RIDA stops with an error rather than silently downloading it all.

### Catalog cache

The metadata and catalog are cached on disk, keyed by URL. Every open re-fetches
the 128-byte header and uses it to validate the cache, because the header already
contains the metadata length, the catalog length and the catalog digest. The
result is that a warm lookup needs two requests instead of three and no longer
downloads the catalog at all. For the 27,647-record Musnad Ahmed file, whose
compressed catalog is 536,472 bytes, that removes roughly 537 KB from every
lookup.

```sh
rida remote-get https://data.example.com/users.rida ali city
rida remote-get https://data.example.com/users.rida ali city --no-cache
rida remote-get https://data.example.com/users.rida ali city --timeout 10
rida cache            # show the cache folder and size
rida cache --clear    # delete cached catalogs
```

The cache lives in the user cache folder (`~/.cache/rida` on Linux and macOS).
Set `RIDA_CACHE_DIR`, or pass `--cache-dir`, or pass `cache_dir=` to
`rida.RemoteRidaFile`, to move it. A remote object that changes invalidates its
own cache entry automatically, because the header no longer matches.

Network reads fail after `--timeout` seconds (default 30) instead of hanging.

### Connection reuse

A range request sent over a fresh TLS connection pays a full handshake, which on
a CDN costs several times more than the lookup itself. The remote reader
therefore keeps one HTTP(S) connection open per reading thread and reuses it, so
a warm lookup spends its time on the value, not on the handshake.

Measured against Bunny CDN on the 27,647-record Musnad Ahmed file:

| | average | median |
| --- | ---: | ---: |
| 0.2.1 with connection reuse (default) | **25.9 ms** | 20.9 ms |
| 0.2.0, fresh connection per lookup | 89.0 ms | 76.9 ms |

Use `--no-keepalive`, or `keepalive=False` in Python, to get the old behaviour.

## Use from Python

```python
import rida

# Scan first if you want to show the publisher the recommendation.
report = rida.scan("users.json")
print(report.recommended_mode)  # "dense" or "sparse"

# Build once. The source JSON stays unchanged.
rida.build(
    "users.json",
    "users.rida",
    title="Users dataset",
    reader_url="https://example.com/rida-explorer",
)

# Read only one stored value from a local RIDA file.
with rida.RidaFile("users.rida") as data:
    print(data.get("ali", "city"))

# Check the whole file against its stored digests.
print(rida.verify("users.rida")["ok"])

# The same primary lookup works against an R2/public URL with Range support.
remote = rida.RemoteRidaFile("https://data.example.com/users.rida")
print(remote.get("fatima", "languages"))
print(remote.cache_hit)  # True when the cached catalog was reused
```

A desktop app or another Python program can show the same stages in its own
interface by passing a callback:

```python
def progress(stage, completed, total):
    print(f"{stage}: {completed}/{total}")

rida.build("users.json", "users.rida", progress=progress)
```

Useful names:

| Name | Purpose |
| --- | --- |
| `rida.build`, `rida.scan` | Build or analyse input; `value_codec=` and `memory_mode=` select behaviour |
| `rida.RidaFile` | Local reader: `get(record_id, field)`, `verify_data()` |
| `rida.RemoteRidaFile` | HTTP byte-range reader: `cache_hit`, `timeout=`, `cache_dir=`, `keepalive=`, `close()` |
| `rida.verify` | Full integrity report for a local file |
| `rida.inspect_metadata` | Header and metadata as a dictionary |
| `rida.VALUE_CODECS` | `("gzip-per-field", "plain")` |
| `rida.clear_remote_cache`, `rida.remote_cache_directory` | Manage the catalog cache |

## Run the tests

```sh
python3 -m unittest discover -s tests -v
```

The suite needs no third-party packages and does not use the network. It starts
its own local range server for the remote tests. `tests/fixtures` contains RIDA
1 files written by the published 0.1.1 library; the suite checks that this
version still reads them and still reproduces them byte for byte with
`--value-codec plain`. Regenerate those fixtures only with
`tests/make_fixtures.py` pointed at an older library.

## Current scope

RIDA 0.2.0 implements primary lookup: `record ID + field name → value`, plus a
whole-file integrity check. Optional exact-match, numeric-range, and Unicode
full-text search indexes are not part of this release, and neither is a
JavaScript reader or a persistent catalog cache for the browser.

## License

RIDA is distributed under the RIDA Proprietary License in the `LICENSE` file.
You may download and modify RIDA, and use it in free or paid applications.
You may not sell or redistribute RIDA as a standalone package or library.
Applications that include RIDA must show the attribution notice specified in
the license.
