Metadata-Version: 2.5
Name: relib
Version: 1.3.18
Project-URL: Repository, https://github.com/Reddan/relib.git
Author: Hampus Hallman
License: Copyright 2018-2025 Hampus Hallman
        
        Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
License-File: LICENSE
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# relib

Small Python utilities for dictionaries, iterables, files, async tasks and local
text search. Python 3.10+. The package has no required third-party dependencies.

## Install

```sh
uv add relib
```

Most utilities are exported from `relib`. Import the search classes from
`relib.search`.

```python
from relib import distinct, group, read_json, roll_tasks
from relib.search import SearchIndex
```

## Utilities

| Module | What it provides |
| --- | --- |
| `dict_utils` | Pick, omit, group, remap, flatten and rebuild dictionaries |
| `iter_utils` | Chunk, partition, deduplicate, interleave and seek through iterables |
| `io_utils` | Read/write JSON and remove empty directories |
| `processing_utils` | Clamp values, filter strings and convert arrays to data frames |
| `runtime_tools` | Async task limits, thread wrappers, timings and progress |
| `type_utils` | Small helpers for optional values and tuples |
| `search` | Ranked dictionary search backed by a local index file |

### Dictionaries and iterables

```python
from relib import distinct, group, partition, pick

assert distinct(["Orion", "Moon", "Orion"]) == ["Orion", "Moon"]
assert group([("clear", "Monday"), ("cloudy", "Tuesday"), ("clear", "Friday")]) == {
    "clear": ["Monday", "Friday"],
    "cloudy": ["Tuesday"],
}
assert pick({"object": "Orion", "exposure": 30, "note": "clear"}, ["object", "exposure"]) == {
    "object": "Orion", "exposure": 30,
}
assert partition((seconds >= 30, seconds) for seconds in [10, 30, 60]) == ([30, 60], [10])
```

### JSON files

```python
from pathlib import Path
from relib import read_json, write_json

path = Path("settings.json")
settings = read_json(path, default={"exposure": 30})
write_json(path, settings, indent=2)
```

`read_json` uses the default only when the file is absent. `write_json` writes to
an existing parent directory. `clear_directory` removes `.DS_Store` files and
empty directories; it does not recursively delete ordinary files.

### Async tasks

```python
import asyncio
from pathlib import Path
from relib import as_async, roll_tasks

@as_async(workers=4)
def read_text(path: Path) -> str:
    return path.read_text()

async def main():
    texts = await roll_tasks(
        (read_text(path) for path in Path("notes").glob("*.txt")),
        workers=4,
    )
    print(len(texts))

asyncio.run(main())
```

`as_async` runs a synchronous function in a thread pool. `roll_tasks` limits how
many awaitables run concurrently and returns results in input order.
`async_limit(workers=4)` decorates an existing async function with a concurrency
limit. `measure_duration("name")` is a context manager that prints elapsed time.

Progress bars require `tqdm`; `df_from_array` requires `numpy` and `pandas`. These
packages are imported only by the features that use them.

## Search dictionaries from disk

`SearchIndex` builds from JSON-compatible dictionaries. Parsing CSV, XML or other
sources belongs to the caller. Records can arrive through a regular one-pass
generator; the index does not retain the complete dictionaries in memory.

The examples below use an astronomy observation log.

### Build and search

This is a complete runnable example:

```python
import asyncio
from relib.search import SearchIndex

observations = [
    {
        "id": "000021",
        "title": "Orion nebula red 80 mm",
        "description": "Hydrogen emission long exposure",
        "observer": "Anaïs",
        "categories": ["Deep sky", "Deep sky > Nebulae"],
        "aperture": "80mm",
        "image": "images/orion-red.fits",
    },
    {
        "id": "000022",
        "title": "Orion nebula blue 80mm",
        "description": "Oxygen emission long exposure",
        "observer": "Anaïs",
        "categories": ["Deep sky", "Deep sky > Nebulae"],
        "aperture": "80mm",
        "image": "images/orion-blue.fits",
    },
    {
        "id": "000031",
        "title": "Lunar mosaic",
        "description": "High resolution crater survey",
        "observer": "Sam",
        "categories": ["Solar system", "Solar system > Moon"],
        "aperture": "200mm",
        "image": "images/moon.fits",
    },
]

async def main():
    index = await SearchIndex.build(
        "observations.idx",
        iter(observations),
        fields={"id": 8, "title": 4, "observer": 2, "categories": 2, "description": 1, "aperture": 1},
        filter_fields=["observer", "categories"],
    )
    with index:
        for hit in await index.search("anais red 80mm"):
            print(hit.record["id"], hit.record["title"], hit.matched_terms)
            # 000021 Orion nebula red 80 mm ('anais', 'red', '80', 'mm')

asyncio.run(main())
```

`fields` selects searchable fields and their relative weights. Passing a list
of names instead gives every field weight 1. Searchable values can be scalars or
lists. Other fields, such as `image`, remain in the stored record without being
indexed. Keep identifiers as strings to preserve leading zeros.

Each result is a `SearchHit` with the original `record` dictionary, a numeric
`score` and normalized `matched_terms`. Only records on the returned page are
decoded, during the search call. `hit.record` is already a dictionary.

### Query options

Reuse an open index across queries. Within an async function:

```python
with SearchIndex("observations.idx") as index:
    by_id = await index.search("000021")
    words = await index.search("hydrogen exposure")
    completion = await index.search("Orion neb", prefix=True)
    category = await index.search(filters={"categories": "Deep sky"})
    filtered = await index.search("nebula", filters={"observer": "anais"})
    next_page = await index.search("nebula", offset=1, limit=1)
    alternatives = await index.search("Orion ultraviolet", match="any")
```

- All words must match by default, across any indexed fields. Unknown words are
  not silently removed. `match="any"` permits partial matches and ranks the
  number of matched words before relevance.
- Case and accents normalize: `anais` matches `Anaïs`. Letters and numbers split,
  so `80mm` matches `80 mm`. `000021` and `21` remain different tokens.
- `prefix=True` completes only the final query word. Punctuation is text
  separation, not query syntax.
- Filters match complete values, ignoring case and accents. A list of values
  means OR within one field; separate fields combine with AND. Declare filter
  fields when building. Category trees have no special meaning: include parent
  paths in each record if you want to filter by them.
- Empty queries browse filtered records in source order. Without filters, an
  empty query returns no results. `len(index)` gives the record count.
- Ranking combines field weights, word frequencies, field lengths and how rare
  each word is in the collection. Equal scores use source order as the tie-break.
- There is no stemming, automatic translation, typo correction or semantic
  matching. Add aliases to your input dictionaries when needed.

### Storage and lifetime

An index is one file containing records, term/filter postings, record offsets
and metadata. Opening it memory-maps the file and reads the term/filter catalog;
it does not decode every record. Query caches retain up to 128 text lookups and
128 filter values.

Builds create a new snapshot and replace the target file after success. Failed
record streams leave the previous file intact. Existing open readers keep their
snapshot; new readers see the replacement. Updates require rebuilding the index.

Building retains the vocabulary and compact postings in memory, so its memory
usage still grows with the collection. Broad partial-match queries can inspect
many more candidates than selective queries. Concurrent reads are supported;
close the index only after its queries finish.
