Metadata-Version: 2.4
Name: rida-format
Version: 0.1.1
Summary: Range Indexed Data Access (RIDA) static data format and library
Author: Syed Ali Hamza Chishty
Maintainer-email: Qasra Foundation <admin@qasra.org>
License-Expression: LicenseRef-Proprietary
Keywords: rida,json,r2,range-requests,static-data
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# RIDA Python library — 0.1.1

RIDA means **Range Indexed Data Access**. It is an installable Python library
and terminal tool that builds one `.rida` file from a collection of JSON
records. The file contains:

- a fixed RIDA header;
- small, readable UTF-8 metadata at the start of the file;
- a compressed dense or sparse address catalog; and
- UTF-8 JSON values stored in a data region.

The reader loads the catalog once and then reads only the requested field bytes.
It works locally and over an HTTP server that supports byte-range requests.

## Install locally

From this project folder:

```sh
python3 -m pip install .
```

After that, the `rida` command is available in the same Python environment.
The library uses only Python's standard library at runtime: no database, cloud
account, or server process is required.

Install the published package with:

```sh
python3 -m pip install rida-format
```

The PyPI package name is `rida-format`. After installation, the terminal
command and Python import name are both `rida`.

### Library project files

Keep the source project in one folder, for example `rida/`:

```text
rida/
├── pyproject.toml          package name and install command
├── MANIFEST.in             files included in the public source archive
├── RIDA-README.md           usage instructions
├── RIDA-DESIGN-0.1.md       RIDA format design
├── LICENSE                  RIDA license
├── docs/
│   └── rida-metadata.html   optional local metadata helper
└── src/
    └── rida/
        ├── __init__.py     RIDA builder and reader library
        └── __main__.py     enables: python -m rida
```

`src/rida/` is internal library code. After installation, Python copies it into
its own `site-packages/rida/` location. A developer never needs to run an
individual `.py` file: they use `import rida`, `rida ...`, or `python -m rida`.

## Requirements and downloads

- Python **3.10 or newer** and `pip` are required.
- RIDA has **zero third-party runtime dependencies**. It uses only Python's
  built-in `json`, `lzma`, `hashlib`, `urllib`, and related standard modules.
- The `rida` command has its own progress display for loading, scanning, value
  writing, catalog compression, and final file writing. It does not require
  `tqdm`.
- It does not install or require AWS SDK, Cloudflare SDK, SQLite, Node.js,
  Rust, a database server, or a Worker.
- When installing this source project, `pip` may temporarily download
  `setuptools` (version 77.0.3 or newer) to build the package. It is an install
  tool, not a RIDA runtime dependency.
- Official Python installers for macOS and Windows include the required `lzma`
  support. On an unusually minimal Linux Python build, install the operating
  system's `python3-lzma` package before using RIDA.

## Build

Scan first. It reports the field density and recommends dense or sparse mode:

```sh
rida scan users.json
```

Build a RIDA file. The command asks for confirmation unless `--yes` is supplied:

```sh
rida build users.json users.rida \
  --title "Users dataset" \
  --description "Public sample data" \
  --reader-url "https://example.com/rida-explorer" \
  --yes
```

During a build, the terminal shows RIDA's own progress, for example:

```text
Reading JSON bytes: 217,221,288/217,221,288 (100%)
Parsing JSON: 1/1 (100%)
Scanning field names: 39,098/39,098 (100%)
Measuring values: 39,098/39,098 (100%)
Writing value data: 39,098/39,098 (100%)
Compressing catalog: 1/1 (100%)
Writing RIDA file: 217,221,288/217,221,288 (100%)
```

For large inputs, low-memory progress reports the record count as it reads.
When the total is not known in advance, updates appear every 1,000 records.

Input can be:

- an object that maps IDs to JSON record objects;
- an array of JSON record objects, with `--id-key`; or
- a folder containing individual JSON record files. Filenames become IDs.

```sh
rida build records/ records.rida --yes
rida build users-array.json users.rida --id-key id --yes
```

If the input folder has JSON files in more than one place, RIDA lists the
groups and asks which folder contains the records. For example, if a folder
has `book_details.json` at its top level and thousands of record files inside
`hadiths/`, choose the `hadiths/` group. For a command that runs without a
prompt, select it explicitly:

```sh
rida scan "/path/to/musnad-ahmed" --records-dir hadiths
rida build "/path/to/musnad-ahmed" musnad-ahmed.rida \
  --records-dir hadiths --yes
```

`--records-dir` is a folder inside the input folder. RIDA searches that folder
and its subfolders for `.json` files; it does not mix separate groups unless
they are all inside the selected folder. The scan prints the number of records
before a build starts.

Use `--mode dense` or `--mode sparse` to override the recommendation. The builder does not overwrite an existing output file.

## Memory modes for large inputs

The default `--memory-mode auto` estimates the input's working-memory needs
and checks currently available RAM. Choose a mode explicitly when needed:

```sh
rida build records.json records.rida --id-key id --memory-mode low-memory --yes
rida build records/ records.rida --memory-mode fast --yes
```

`low-memory` reads one JSON record at a time instead of keeping the whole JSON
input in RAM. For one JSON file, supported layouts are an object mapping IDs
to record objects, or an array of record objects with `--id-key`. For a folder,
RIDA reads one record file at a time. The current record and the address
catalog still use RAM, so this is not a zero-memory mode; an individual record
must fit in memory.

If the input folder contains separate JSON groups, RIDA asks which group to
use. For non-interactive commands, pass `--records-dir`, for example:

```sh
rida build "/path/to/musnad-ahmed" musnad-ahmed.rida \
  --records-dir hadiths --memory-mode low-memory --yes
```

## Read locally

```sh
rida inspect users.rida
rida get users.rida ali city
```

## Where RIDA can read data

RIDA 0.1.1 can read a `.rida` file from the local computer or from a direct
HTTP/HTTPS file URL. For remote reads, RIDA sends ordinary HTTP byte-range
requests; it does not use a cloud-provider SDK. This makes it compatible with
object storage that returns the requested bytes as HTTP `206 Partial Content`
and includes the matching `Content-Range` header.

Examples expected to work because their object APIs support byte-range reads
include Cloudflare R2, Amazon S3, Google Cloud Storage, Azure Blob Storage,
Backblaze B2, and Bunny.net through a Pull Zone. See their docs: [R2](https://developers.cloudflare.com/r2/api/s3/api/),
[S3](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html),
[Google Cloud Storage](https://docs.cloud.google.com/storage/docs/downloading-objects),
[Azure Blob Storage](https://learn.microsoft.com/en-us/rest/api/storageservices/get-blob),
 [Backblaze B2](https://www.backblaze.com/apidocs/b2-download-file-by-id), and
[Bunny.net](https://bunny.net/faq/).
This is protocol-based expected compatibility, not a separate integration
with each provider. The exact URL, permissions, and any CDN in front of a
bucket still need to allow byte-range responses; the providers have not all
been individually tested with this RIDA release.

For Bunny Storage, connect the Storage Zone to a Pull Zone and use the Pull
Zone URL (or its custom hostname) for the file. Bunny documents range support
for cached content by default; enable Cache Slicing in the Pull Zone for
uncached content.

The current reader sends a `Range` header but does not add provider-specific
authentication headers. Use a public object URL or a signed URL that permits
the range request. RIDA does not upload files to cloud storage: upload the
`.rida` file using that storage provider, then give RIDA the file URL.

## Read from R2 or another HTTP URL

After uploading the `.rida` file to an HTTP byte-range server:

```sh
rida remote-get https://data.example.com/users.rida ali city
```

The remote reader first fetches the 128-byte header and catalog, then fetches
only the requested field range. It checks that the server returns HTTP `206`
and the correct `Content-Range`; if the server sends the whole file instead,
RIDA stops with an error rather than silently downloading it all.

One `RemoteRidaFile` object keeps that catalog in RAM, so later `.get()` calls on
the same object fetch only the requested value range. The `rida remote-get`
terminal command creates a new object each time, so persistent on-disk catalog
cache is a later RIDA Reader feature.

## Use from Python

```python
import rida

# Scan first if you want to show the publisher the recommendation.
report = rida.scan("users.json")
print(report.recommended_mode)  # "dense" or "sparse"

# Build once. The source JSON stays unchanged.
rida.build(
    "users.json",
    "users.rida",
    title="Users dataset",
    reader_url="https://example.com/rida",
)

# Read only one stored value from a local RIDA file.
with rida.RidaFile("users.rida") as data:
    print(data.get("ali", "city"))

# The same primary lookup works against an R2/public URL with Range support.
remote = rida.RemoteRidaFile("https://data.example.com/users.rida")
print(remote.get("fatima", "languages"))
```

A desktop app or another Python program can show the same stages in its own
interface by passing a callback:

```python
def progress(stage, completed, total):
    print(f"{stage}: {completed}/{total}")

rida.build("users.json", "users.rida", progress=progress)
```

## Current scope

RIDA 0.1.1 implements primary lookup: `record ID + field name → value`.
Optional exact-match, numeric-range, and Unicode full-text search indexes are
not part of this release.

## License

RIDA is distributed under the RIDA Proprietary License in the `LICENSE` file.
You may download and modify RIDA, and use it in free or paid applications.
You may not sell or redistribute RIDA as a standalone package or library.
Applications that include RIDA must show the attribution notice specified in
the license.
