Metadata-Version: 2.4
Name: rida-format
Version: 0.3.3
Summary: Range Indexed Data Access (RIDA) static data format and library
Author: Syed Ali Hamza Chishty
Maintainer-email: Qasra Foundation <admin@qasra.org>
License-Expression: LicenseRef-Proprietary
Project-URL: PyPI, https://pypi.org/project/rida-format/
Project-URL: Changelog, https://pypi.org/project/rida-format/#history
Keywords: rida,json,r2,range-requests,static-data
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: encryption
Requires-Dist: cryptography>=42.0; extra == "encryption"
Dynamic: license-file

# RIDA Python library — 0.3.3

RIDA means **Range Indexed Data Access**. It is an installable Python library
and terminal tool that builds a RIDA dataset from a collection of JSON records.

A dataset is **four files, and neither the data nor the index carries a header**:

| File | Holds | Where it goes |
|---|---|---|
| `NameData.rida` | field values, and nothing else | the publisher's server or CDN |
| `NameIndex.rida` | the address catalog plus record count, value codec, data length, flags and one digest | with the reader |
| `NameMap.json` | the folder names behind the numbered record IDs and fields | with the reader |
| `NameKey.rida` | the 256-bit key, written only for an encrypted build | kept private |

The data file is the only thing served. The index and the map travel with the
reader, so opening a dataset costs **no request at all** and every lookup costs
exactly **one** — the range of bytes holding that value.

The map is what lets a reader ask for a record by a name a person recognises, as
`shamela/Bukhari/5`, rather than by its number. A dataset whose record names are
already unique has no map, and needs none.

Every stored value is its own deflate stream, so a reader fetches one range and
inflates one member. On RIDA's reference dataset (27,647 records, 34 fields) the
data region drops from 191,378,148 to 95,476,203 bytes, which is 18 bytes per
value less than a gzip member would take.

With `--encrypt` each value is sealed with AES-256-GCM after compression, which
costs 28 bytes per value (a 12-byte nonce and a 16-byte tag) and keeps one value
per request. On the reference dataset the data file grows from 106.3 MB to
128.7 MB and the build takes 34.7 s instead of 29.3 s.

The figures in this document that say "the reference dataset" all come from that
one measured file. It is the dataset RIDA was built against; it is not a special
case, and any JSON collection of the same shape behaves the same way.

## Install locally

From this project folder:

```sh
python3 -m pip install .
```

After that, the `rida` command is available in the same Python environment.
The library uses only Python's standard library at runtime: no database, cloud
account, or server process is required.

Install the published package with:

```sh
python3 -m pip install rida-format
```

The PyPI package name is `rida-format`. After installation, the terminal
command and Python import name are both `rida`. Check the installed version with:

```sh
rida --version
```

### Library project files

Keep the source project in one folder, for example `rida/`:

```text
rida/
├── pyproject.toml           package name and install command
├── MANIFEST.in              files included in the public source archive
├── RIDA-README.md           usage instructions
├── RIDA-FORMAT.md           byte-level file format specification
├── CHANGELOG.md             release history
├── LICENSE                  RIDA license
├── docs/
│   └── rida-metadata.html   optional local metadata helper
├── tests/
│   ├── test_rida.py         test suite
│   ├── make_fixtures.py     regenerates the RIDA 1 golden fixtures
│   └── fixtures/            RIDA 1 files written by the 0.1.1 library
└── src/
    └── rida/
        ├── __init__.py      RIDA builder and reader library
        └── __main__.py      enables: python -m rida
```

`src/rida/` is internal library code. After installation, Python copies it into
its own `site-packages/rida/` location. A developer never needs to run an
individual `.py` file: they use `import rida`, `rida ...`, or `python -m rida`.

## Requirements and downloads

- Python **3.10 or newer** and `pip` are required.
- RIDA has **no required third-party runtime dependencies**. It uses only
  Python's built-in `json`, `zlib`, `lzma`, `hashlib`, `urllib`, and related
  standard modules.
- **Value encryption is the one optional extra.** It needs the
  `cryptography` package, so install it only when you build or read encrypted
  files:

  ```sh
  python3 -m pip install "rida-format[encryption]"
  ```

  Everything else — building unencrypted files, local reads, remote reads,
  verification — works without it. `rida.encryption_available()` reports
  whether it is importable in the current environment.
- The `rida` command has its own progress display for loading, scanning, value
  writing, catalog compression, and final file writing. It does not require
  `tqdm`.
- It does not install or require AWS SDK, Cloudflare SDK, SQLite, Node.js,
  Rust, a database server, or a Worker.
- When installing this source project, `pip` may temporarily download
  `setuptools` (version 77.0.3 or newer) to build the package. It is an install
  tool, not a RIDA runtime dependency.
- Official Python installers for macOS and Windows include the required `lzma`
  support. On an unusually minimal Linux Python build, install the operating
  system's `python3-lzma` package before using RIDA.

## Build

Scan first. It reports the field density and recommends dense or sparse mode:

```sh
rida scan users.json
```

Build a RIDA file. The command asks for confirmation unless `--yes` is supplied:

```sh
rida build users.json UsersData.rida \
  --url https://data.example.org/UsersData.rida \
  --yes
```

During a build, the terminal shows RIDA's own progress, for example:

```text
Scanning field names: starting…
Scanning field names: 1,000 files [00:01, 900 files/s]
Scanning field names: 100%|██████████████████████████████| 479,552/479,552 [...]
Measuring values: 100%|██████████████████████████████| 479,552/479,552 [...]
Compressing catalog: 1/1 (100%)
Writing RIDA data: 100%|██████████████████████████████| 217,221,288/217,221,288 [...]
```

For large inputs, progress reports the record count as it reads. When the total
is not known in advance, updates appear every 1,000 records.

Input can be:

- an object that maps IDs to JSON record objects;
- an array of JSON record objects, with `--id-key`; or
- a folder containing individual JSON record files. Filenames become IDs.

```sh
rida build records/ RecordsData.rida --yes
rida build users-array.json UsersData.rida --id-key id --yes
```

`--url` records where the data file will be served from. The index is then
written next to the data file **and** installed in the reader cache folder, so a
reader on the same machine finds it with no setup.

If the input folder has JSON files in more than one place, RIDA lists the
groups and asks which folder contains the records. For example, if a folder
has `config.json` at its top level and thousands of record files inside
`items/`, choose the `items/` group. For a command that runs without a
prompt, select it explicitly:

```sh
rida scan "./my-data" --records-dir items
rida build "./my-data" my-data.rida --records-dir items --yes
```

`--records-dir` is a folder inside the input folder. RIDA searches that folder
and its subfolders for `.json` files; it does not mix separate groups unless
they are all inside the selected folder. The scan prints the number of records
before a build starts.

#### When several groups should be one file

A folder holding several groups — `shamela/`, `shia/`, `sunni/`, say — may hold
one dataset that was split across folders, or several separate ones. RIDA
settles it by comparing a sample of each group's field names, and prints what it
found:

```text
JSON groups found: 3
Distinct fields across all groups: 34
Fields shared by every group: 28

group                              records   fields  not shared
---------------------------------------------------------------
shamela                                300       34           6
shia                                   300       28           0
sunni                                  300       34           6

  field 'f28': not in shia
```

It then lays out the three ways to build, each with a size:

```text
Three ways to build this folder (900 records in all)

  1) one file for each group, each keeping its own schema
       group                       records  fields         data      catalog
       shamela                         300      34     135.7 KB      33.1 KB
       shia                            300      28     111.3 KB      27.7 KB
       sunni                           300      34     135.7 KB      33.1 KB
       in all                          900             382.7 KB      93.9 KB

  2) one file, on the fields every group shares
     only the 28 fields every group has; the rest are not stored
     900 records, 28 fields, dense, data about 333.8 KB, catalog about 82.8 KB
     6 field(s) would not be stored: 'f28', 'f29', 'f30', 'f31', 'f32', 'f33'
     Field density would be 100.0%.

  3) one file, widened to every field any group uses
     all 34 fields any group uses; a record that lacks one stores nothing for it
     900 records, 34 fields, dense, data about 382.7 KB, catalog about 98.7 KB
     Field density would be 94.1%.
```

Option 2 keeps only the fields every group has, so a group's own fields are not
stored at all: the smallest file, at the cost of data. Option 3 keeps every
field, so a record that lacks one stores nothing for it and the density drops.
Which is better depends on whether those extra fields matter, which is why both
are measured rather than assumed.

Non-interactively:

```sh
rida build "./old" union.rida  --combine-groups --yes
rida build "./old" common.rida --combine-groups --combined-schema common --yes
```

Only a few hundred records per group are read to compare the schemas, so the
answer arrives before a build commits to anything.

A combined build numbers its records, so the four files it writes are:

```text
old.rida          the values, which go to the CDN
oldIndex.rida     the catalog, which stays with the reader
oldMap.json       the folder names behind the numbers
oldKey.rida       the key, when the values are encrypted
```

That is the whole deliverable. Where the four files live afterwards is the
publisher's choice: a CDN, a web server, a laptop. RIDA reads a data file over
HTTP byte ranges and does not care who serves it.

#### Record IDs

Every book numbers its records from `1`. Three folders holding `1.json` therefore
produce three records called `1`, and no file name can tell them apart:

```text
shamela/Bukhari/hadiths/1.json     ->  "1"
shia/Al-Kafi/hadiths/1.json        ->  "1"
sunni/Musnad/hadiths/1.json        ->  "1"
```

RIDA identifies a record by where it sits rather than what it is called. Each
folder's children are numbered from 1 in sorted order, so an ID has one number
per level:

```text
shamela/Bukhari/hadiths/1.json     ->  1/1/1/1
shia/Al-Kafi/hadiths/1.json        ->  2/1/1/1
sunni/Musnad/hadiths/1.json        ->  3/1/1/1
```

Positions cannot repeat. The same ID can never be produced twice, at any depth,
whatever the folders are called and however many there are -- three folders or
ten crore.

##### The map file

The numbers mean nothing on their own, so a build writes a fourth file beside the
data and index:

```json
{
  "version": 1,
  "records": 479552,
  "paths": {
    "1": "shamela",
    "1/1": "Bukhari",
    "1/1/1": "hadiths"
  },
  "fields": {
    "1": "english",
    "2": "urdu",
    "3": "arabic"
  }
}
```

`paths` turns a numbered prefix back into the folder names it stands for, so a
reader can find a record by the name a person knows:

```python
data.get("1/1/1/5", "english")             # by number
data.get("shamela/Bukhari/5", "english")   # by name, using the map

data.resolve("shamela", "Bukhari")   # -> "1/1"
data.records_under("1/1")            # -> ["1/1/1/1", "1/1/1/2", ...]
data.folders()                       # -> every numbered prefix and its name
```

The name form does not need the `hadiths` level: `shamela/Bukhari/5` finds
`1/1/1/5`. Naming only a folder, `shamela/Bukhari`, gives that folder's first
record.

The map is optional. A file whose IDs are already its record names reads without
one, and a reader that cannot find a map simply expects numbers.

##### Keeping the plain names

`--id-scheme named` keeps the old behaviour: the file name is the ID, and repeats
are numbered `1-2`, `1-3`. No map is written. Use it when the names are already
meaningful and unique, such as `Bukhari_1.json` and `Muslim_1.json`.

| | `numbered` (default) | `named` |
| --- | --- | --- |
| ID | `1/1/1/1` | `1`, `1-2` |
| Map file | written | not written |
| Lookup by name | yes | no |
| Collisions | impossible | handled with a suffix |

Records are ordered numerically rather than alphabetically, so `10.json` is the
third record and not the second.

Use `--mode dense` or `--mode sparse` to override the recommendation. The
builder does not overwrite an existing output file.

### Choosing a value codec

`--value-codec` selects the stored value layout:

| Value | Header | Stored value | When to use it |
| --- | --- | --- | --- |
| `deflate-per-field` (default) | one raw deflate stream per value | Normal use. Cuts file size and per-lookup transfer when values are large enough. |
| `plain` | plain UTF-8 JSON | Small values or already-compressed values. |

```sh
rida build users.json UsersData.rida --value-codec plain --yes
```

The stream is raw deflate, without the gzip wrapper. That wrapper would add 18
bytes to every value: a magic number and modification time that say nothing
(the codec is already in the index), and a CRC-32 that would repeat the check
AES-256-GCM already performs. Values that are only a few bytes long can still be
larger when compressed; `scan` measures the real stored size for the codec you
select, so its estimates and recommendation match the file you are about to write.

## Building large inputs

RIDA processes one JSON record at a time, so it does not load the whole
collection into RAM. For one JSON file, supported layouts are an object mapping
IDs to record objects, or an array of record objects with `--id-key`. For a
folder, RIDA reads one record file at a time. The current record and the address
catalog still use RAM, so an individual record must fit in memory.

```sh
rida build records.json RecordsData.rida --id-key id --yes
rida build records/ RecordsData.rida --yes
```

When you pass an explicit `--mode dense` or `--mode sparse`, the builder stages
only that index layout instead of both, which lowers its peak catalog memory
and CPU.

If the input folder contains separate JSON groups, RIDA asks which group to
use. For non-interactive commands, pass `--records-dir`, for example:

```sh
rida build "./my-data" MyDataData.rida \
  --records-dir items --yes
```

## Read locally

```sh
rida inspect UsersData.rida
rida fields UsersData.rida
rida get UsersData.rida ali city
```

`inspect` prints the index as JSON: record count, codec, lengths and field
names. `get` prints one value and exits with status 1 and a one-line message if
the record ID or field name does not exist.

A numbered dataset takes either form of record ID. The name form needs the map
beside the data file; the number form does not:

```sh
rida get old.rida 1/1/1/5 english --key "$(cat oldKey.rida)"
rida get old.rida shamela/Bukhari/5 english --key "$(cat oldKey.rida)"
```

To see what a numbered dataset holds, ask it for the folder names:

```sh
rida folders old.rida
```

`fields` answers "which key do I ask for?" — the field names live in the index,
so nothing has to be guessed or documented separately:

```text
$ rida fields UsersData.rida
Records: 27,647
Fields: 34
   1  id
   2  name
   3  city
   4  joined
   ...
```

It also accepts an http(s) URL, and `--json` prints just the array.

## Check a file for damage

Reading one field never hashes the whole data region, so a normal lookup trusts
the transport. `verify` is the explicit, slower check: it streams every data byte
and compares it with the digest recorded in the index.

```sh
rida verify UsersData.rida
```

```json
{
  "file": "UsersData.rida",
  "format_version": 3,
  "value_codec": "deflate-per-field",
  "record_count": 27647,
  "data_bytes": 110924973,
  "actual_bytes": 110924973,
  "size_ok": true,
  "catalog_ok": true,
  "data_ok": true,
  "encrypted": false,
  "ok": true
}
```

`verify` exits with status 1 when `ok` is false. For an encrypted dataset pass
`--key`. From Python, use `rida.verify("UsersData.rida")`, or
`RidaFile.verify_data()` on an already open file.

## Where RIDA can read data

RIDA 0.3.3 can read a data file from the local computer or from a direct
HTTP/HTTPS file URL. For remote reads, RIDA sends ordinary HTTP byte-range
requests; it does not use a cloud-provider SDK. This makes it compatible with
object storage that returns the requested bytes as HTTP `206 Partial Content`
and includes the matching `Content-Range` header.

Examples expected to work because their object APIs support byte-range reads
include Cloudflare R2, Amazon S3, Google Cloud Storage, Azure Blob Storage,
Backblaze B2, and Bunny.net through a Pull Zone. See their docs: [R2](https://developers.cloudflare.com/r2/api/s3/api/),
[S3](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html),
[Google Cloud Storage](https://docs.cloud.google.com/storage/docs/downloading-objects),
[Azure Blob Storage](https://learn.microsoft.com/en-us/rest/api/storageservices/get-blob),
[Backblaze B2](https://www.backblaze.com/apidocs/b2-download-file-by-id), and
[Bunny.net](https://bunny.net/faq/).
This is protocol-based expected compatibility, not a separate integration
with each provider. The exact URL, permissions, and any CDN in front of a
bucket still need to allow byte-range responses; the providers have not all
been individually tested with this RIDA release.

The current reader sends a `Range` header and a `rida/<version>` User-Agent, but
does not add provider-specific authentication headers. Use a public object URL
or a signed URL that permits the range request. RIDA does not upload files to
cloud storage: upload the data file with that provider, then give RIDA the URL.

## Read from R2 or another HTTP URL

Upload only the data file. The index stays with your app:

```sh
rida remote-get https://data.example.com/UsersData.rida ali city \
  --index UsersIndex.rida
```

The reader never fetches the index. It checks that the server returns HTTP `206`
and the correct `Content-Range`; if the server sends the whole file instead, RIDA
stops with an error rather than silently downloading it all.

### The index

The index is found in this order: the `index=` argument, then the reader cache
folder keyed by URL. Building with `--url` fills the cache folder for you, so on
the machine that built the dataset no options are needed at all.

```sh
rida remote-get https://data.example.com/UsersData.rida ali city
rida remote-get https://data.example.com/UsersData.rida ali city --no-cache
rida remote-get https://data.example.com/UsersData.rida ali city --timeout 10
rida cache            # show the cache folder and size
rida cache --clear    # delete cached catalogs
```

The cache lives in the user cache folder (`~/.cache/rida` on Linux and macOS).
Set `RIDA_CACHE_DIR`, or pass `--cache-dir`, or pass `cache_dir=` to
`rida.RemoteRidaFile`, to move it. A remote object that changes invalidates its
own cache entry automatically, because the header no longer matches.

Network reads fail after `--timeout` seconds (default 30) instead of hanging.

### Connection reuse

A range request sent over a fresh TLS connection pays a full handshake, which on
a CDN costs several times more than the lookup itself. The remote reader
therefore keeps one HTTP(S) connection open per reading thread and reuses it, so
a warm lookup spends its time on the value, not on the handshake.

Measured against Bunny CDN on the reference dataset:

| | average | median |
| --- | ---: | ---: |
| 0.2.1 with connection reuse (default) | **25.9 ms** | 20.9 ms |
| 0.2.0, fresh connection per lookup | 89.0 ms | 76.9 ms |

Use `--no-keepalive`, or `keepalive=False` in Python, to get the old behaviour.

### Asking for several fields at once

A request header costs more than a small payload. Measured against Bunny, every
response carries about **660 bytes of headers**, while a stored field value in
the reference dataset averages a few hundred bytes. Many small single-field
lookups therefore spend most of their bytes on headers.

Because a record's values are stored next to each other, `get_many` and
`get_record` fetch several fields in one range request:

| | Requests | Bunny bill |
| --- | ---: | ---: |
| A whole 34-field record, field by field | 34 | ~26,800 B |
| The same record with `get_record()` | **1** | ~5,000 B |

Two fields are merged into one request when the bytes between them are cheaper
than the request header they save (`MERGE_GAP_LIMIT`, 660 bytes), and stay
separate otherwise — a field asked on its own is still exactly one request.

### Cost at scale

RIDA's main payoff is the size of the delivered response: the reference dataset
averages about **7.69 KB** per complete record but only about **143 bytes** per
stored RIDA field value. When the CDN bills per GB transferred, every lookup
that asks for one field instead of the whole record saves roughly **98 %** of
the delivery bytes.

The table shows a 1-billion-lookup month on the reference dataset. A file of the
same shape and size with a different subject would cost the same.

| Provider | Billing | Normal JSON (1B lookups) | RIDA field-deflate (1B lookups) |
| --- | --- | ---: | ---: |
| Bunny CDN — Standard (Asia) | per GB delivered | **$230.79**/mo | **$4.29**/mo |
| Bunny CDN — Volume (first tier) | per GB delivered | **$38.46**/mo | **$0.71**/mo |
| AWS CloudFront — US / Europe | per GB delivered | **$653.90**/mo | **$12.15**/mo |
| AWS CloudFront — India | per GB delivered | **$1,307.80**/mo | **$24.31**/mo |
| Google Cloud CDN — US / Europe | per GB delivered | **$615.44**/mo | **$11.44**/mo |
| Google Cloud CDN — Asia | per GB delivered | **$923.16**/mo | **$17.16**/mo |
| Azure CDN — Zone 1 (US / EU) | per GB delivered | **$623.13**/mo | **$11.58**/mo |
| Azure CDN — Zone 2 (India / Asia) | per GB delivered | **$1,038.55**/mo | **$19.30**/mo |
| Backblaze B2 + Bunny CDN | per GB delivered | **$76.93**/mo | **$1.43**/mo |
| Cloudflare R2 | per read (Class B) | **$356.40**/mo | **$356.40**/mo |

Cloudflare R2 costs the same either way because it bills per read, not per GB,
and its egress is free. CDN providers that bill per GB all show the same 98 %
reduction — a result of the file format, not of the provider.

This is a planning estimate, not a provider quote. Provider rates, regional mix,
caching, taxes, and account plans change the final bill. Storage adds roughly
$0.02–$0.37/mo for a 5 GB file and $0.20–$3.70/mo for a 50 GB file across the
providers listed (Cloudflare R2 gives the first 10 GB-month free).

## Encrypt the data

A data file that is served from a CDN is public. Encrypting it means a scraper
that downloads the file gets bytes it cannot read.

```sh
# Needs the optional extra: pip install "rida-format[encryption]"
rida build ./my-data ItemsData.rida --encrypt
```

Three files come out:

| File | What it is |
|---|---|
| `ItemsData.rida` | Field values, each deflate-compressed and then sealed with AES-256-GCM. |
| `ItemsIndex.rida` | The address catalog. Plain, and it stays with your app. |
| `ItemsKey.rida` | The 256-bit AES key, URL-safe base64 (44 characters). Keep it private. |

Read it back by passing the key — from a string, or straight from the file:

```python
import rida
from pathlib import Path

key = Path("ItemsKey.rida").read_text().strip()

with rida.RidaFile("ItemsData.rida", key=key) as data:
    print(data.get("ali", "city"))

# The same works over HTTP with byte ranges.
with rida.RemoteRidaFile("https://cdn.example.com/ItemsData.rida", key=key) as data:
    print(data.get_record("ali"))
```

And from the command line:

```sh
rida get ItemsData.rida ali city --key "$(cat ItemsKey.rida)"
rida fields ItemsData.rida --key "$(cat ItemsKey.rida)"
rida remote-get https://cdn.example.com/ItemsData.rida ali city --key "$(cat ItemsKey.rida)"
```

### Why compression runs before AES

Encrypted bytes do not compress. Sealing first would throw away the whole
benefit of the codec, so every value is compressed and then sealed. Encryption
costs 28 bytes per value — a 12-byte nonce and a 16-byte tag — and adds about
8.6 microseconds per value to a build.

### The key is the checksum

No separate checksum is needed. If a single byte of the data changes, the GCM
tag no longer verifies and the value fails to decrypt. `verify()` therefore
checks the data length, the digest stored in the index and that the catalog
decodes; the key does the real work.

### What encryption does and does not do

It stops anyone who does not hold the key from reading any value. It does not
protect you from a reader who already has the key: the key has to reach your
client somehow, and a client that holds a key can be studied. Looking after the
key is the publisher's job, exactly like a password.

For access control tied to a user account, keep the key on a server and hand it
out per session. `rida.encryption_available()` tells you whether the optional
dependency is present before you promise encryption to a caller.

## Use from Python

```python
import rida

# Scan first if you want to show the publisher the recommendation.
report = rida.scan("users.json")
print(report.recommended_mode)  # "dense" or "sparse"

# Build once. The source JSON stays unchanged.
rida.build("users.json", "UsersData.rida", url="https://data.example.org/UsersData.rida")

# Read only one stored value from a local RIDA file.
with rida.RidaFile("UsersData.rida") as data:
    print(data.get("ali", "city"))

# Read several fields, or the whole record, in one request.
with rida.RidaFile("UsersData.rida") as data:
    print(data.get_many("ali", ["name", "city"]))
    print(data.get_record("ali"))

# Check the data file against its index.
print(rida.verify("UsersData.rida")["ok"])

# The same lookups work against a public URL with Range support.
remote = rida.RemoteRidaFile("https://data.example.com/UsersData.rida")
print(remote.get("fatima", "languages"))
print(remote.get_many("fatima", ["languages", "city"]))
```

`get_many` raises `KeyError` for a field name the file does not have, so a typo
never passes silently. A field the file has but this record lacks is simply left
out of the returned dictionary.

### Reading a collection that numbers its records

A folder whose books all number their records from `1` gets numbered IDs, and a
map beside the data file turns a name into one:

```python
with rida.RidaFile("old.rida", key=open("oldKey.rida").read().strip()) as data:
    # By name, the way a person would say it.
    print(data.get("shamela/Bukhari/5", "english"))

    # By number, which needs no map and works from a bare index.
    print(data.get("1/1/1/5", "english"))

    # What does this file hold?
    for prefix, name in data.folders().items():
        print(prefix, name)

    # Everything under one book.
    for record_id in data.records_under(data.resolve("shamela", "Bukhari")):
        print(record_id, data.get(record_id, "english"))
```

An operator who does not know the names yet can ask the file:

```sh
rida folders old.rida
```

A desktop app or another Python program can show the same stages in its own
interface by passing a callback:

```python
def progress(stage, completed, total):
    print(f"{stage}: {completed}/{total}")

rida.build("users.json", "UsersData.rida", progress=progress)
```

Useful names:

| Name | Purpose |
| --- | --- |
| `rida.build`, `rida.scan` | Build or analyse input; `value_codec=` selects the stored value layout |
| `rida.open_reader` | Open a local path or an http(s) URL with one call |
| `rida.RidaFile` | Local reader: `get`, `get_many`, `get_record`, `verify_data()`, `folders()`, `resolve()`, `records_under()`; `index=` to name the index file |
| `rida.RemoteRidaFile` | HTTP byte-range reader: same lookups, plus `index=`, `timeout=`, `cache_dir=`, `keepalive=`, `close()` |
| `rida.verify` | Integrity report for a local data file |
| `rida.inspect_metadata` | The index contents as a dictionary |
| `rida.MERGE_GAP_LIMIT` | How many skipped bytes still beat a request header (660) |
| `rida.VALUE_CODECS` | `("deflate-per-field", "plain")` |
| `rida.clear_remote_cache`, `rida.remote_cache_directory` | Manage the index cache folder |
| `rida.read_map_folders` | The numbered folders beside a data file, without opening it |
| `rida.ID_SEPARATOR` | The separator between numbered levels (`"/"`) |

## Run the tests

```sh
python3 -m unittest discover -s tests -v
```

The suite needs no third-party packages and does not use the network. It starts
its own local range server for the remote tests. `tests/fixtures` freezes one
build; the suite checks that this version still reads it and still reproduces its
catalog and data bytes. Regenerate it with `tests/make_fixtures.py` whenever the
layout changes.

## Current scope

RIDA 0.3.3 implements primary lookup — `record ID + field name → value` — plus
multi-field reads (`get_many`, `get_record`), numbered folder IDs with a map that
turns a name into one, a data-file integrity check, and optional AES-256-GCM
encryption of every stored value. Optional exact-match,
numeric-range, and Unicode full-text search indexes are not part of this release,
and neither is a JavaScript reader.

## License

RIDA is distributed under the RIDA Proprietary License in the `LICENSE` file.
You may download and modify RIDA, and use it in free or paid applications.
You may not sell or redistribute RIDA as a standalone package or library.
Applications that include RIDA must show the attribution notice specified in
the license.
