Metadata-Version: 2.4
Name: aethisdb
Version: 1.0.1
Summary: A lightweight, pure-Python document database with a custom binary file format
Author: Aethis Studio
License: MIT
Keywords: database,nosql,document,pure-python,embedded
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database
Classifier: License :: OSI Approved :: MIT License
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"

# AethisDB

A lightweight, pure-Python document database with a custom binary file format (`.aethisdb`). Zero external dependencies — uses only the Python standard library.

## Features

- **Custom binary file format** with a 64-byte header and append-friendly data section
- **Hash-based indexing** for fast exact-match queries on indexed fields
- **Full CRUD operations**: insert, find, update, delete
- **Rich query operators**: `$eq`, `$ne`, `$gt`, `$lt`, `$gte`, `$lte`, `$in`, `$nin`, `$exists`, `$and`, `$or`
- **Auto-commit or manual commit** modes
- **File compaction** to reclaim space from deleted records
- **Automatic compaction** based on tombstone ratio — ideal for small/embedded storage
- **zlib-compressed index block** for minimal on-disk footprint
- **Compact JSON serialization** (no whitespace in record data)
- **Space statistics** reporting via `db.stats()`
- **Context manager** support (`with Database(...) as db:`)
- **Persistent across sessions** — collections and indexes survive reopens

## Installation / Setup

AethisDB has no third-party dependencies. Simply place the `aethisdb/` package directory on your Python path, or install it in development mode:

```bash
pip install aethisdb
```

Then import:

```python
from aethisdb import Database, Record
```

## Quick Start

### Initializing a Database

```python
from aethisdb import Database

# Creates a new .aethisdb file (or opens an existing one)
db = Database("mydata.aethisdb")

# Create a collection
users = db.create_collection("users")

# List collections
print(db.list_collections())  # ['users']
```

### Inserting Records

```python
# Insert a single record — returns the auto-assigned record ID
uid = users.insert({"name": "Alice", "email": "alice@example.com", "age": 30})

# Insert multiple records at once
ids = users.insert_many([
    {"name": "Bob",   "email": "bob@example.com",   "age": 25},
    {"name": "Carol", "email": "carol@example.com", "age": 35},
])

# Get a record by ID
alice = users.get(0)
print(alice["name"])  # Alice
```

### Querying Records

```python
# Find by exact match
results = users.find({"name": "Alice"})
print(results[0]["email"])  # alice@example.com

# Find one (returns None if no match)
bob = users.find_one({"email": "bob@example.com"})

# Range query
adults = users.find({"age": {"$gte": 30}})

# Multiple conditions (implicit AND)
results = users.find({"age": {"$gte": 25}, "age": {"$lte": 35}})

# $in operator
results = users.find({"name": {"$in": ["Alice", "Bob"]}})

# $or operator
results = users.find({"$or": [{"age": 25}, {"age": 35}]})

# Get all records
all_users = users.all()

# Count records
print(users.count())  # 3
```

### Updating Records

```python
# Update matching records
count = users.update({"name": "Alice"}, {"age": 31})
print(count)  # 1

# Update by record ID
users.update(record_id=1, update_data={"age": 26})
```

> **Note:** Updates in AethisDB use a copy-on-write strategy — the old record is marked as deleted and a new record is appended with the updated data. Record IDs are preserved.

### Deleting Records

```python
# Delete by query
count = users.delete({"name": "Bob"})

# Delete by record ID
count = users.delete(record_id=2)

# Delete all records (use with care!)
users.delete()

# Count after deletion
print(users.count())  # 2
```

### Indexing

Hash indexes provide O(1) exact-match lookups:

```python
# Create an index on a field
users.create_index("email")

# Now queries on the indexed field use the index automatically
bob = users.find_one({"email": "bob@example.com"})  # fast

# List indexes
print(users.indexes)  # ['email']

# Drop an index
users.drop_index("email")

# Rebuild an index after bulk operations
users.create_index("email")
users.rebuild_index("email")
```

### Compaction

Deleted records leave tombstones in the file. Use `compact()` to reclaim space:

```python
# After many deletions
file_size_before = os.path.getsize("mydata.aethisdb")

db.compact()

file_size_after = os.path.getsize("mydata.aethisdb")
# File is smaller — tombstones removed
```

### Transaction / Auto-commit Control

```python
# Manual commit mode — changes are not persisted until commit()
db = Database("mydata.aethisdb", auto_commit=False)

users = db.get_collection("users")
users.insert({"name": "Alice"})
users.insert({"name": "Bob"})

# Verify data is not yet persisted
db.close()  # Without commit, changes are lost

# Correct usage:
db = Database("mydata.aethisdb", auto_commit=False)
users = db.get_collection("users")
users.insert({"name": "Alice"})
users.insert({"name": "Bob"})
db.commit()  # persist to disk
db.close()
```

### Context Manager

```python
with Database("mydata.aethisdb") as db:
    users = db.get_collection("users")
    users.insert({"name": "Alice"})
    alice = users.find_one({"name": "Alice"})
# db is automatically closed
```

### Space Efficiency (Small Storage)

AethisDB includes several features to minimize disk usage, making it ideal for embedded or small-storage applications:

#### Compact JSON

All record data and the index block use compact JSON serialization (no whitespace, no separator padding). This reduces file size compared to standard `json.dumps()`.

#### Compressed Index Block

The index block at the end of the file is automatically zlib-compressed when compression reduces its size. Compression is transparent — no configuration required.

#### Auto-Compaction

When records are deleted, they are soft-deleted (marked as tombstones). Over time, tombstones waste space. AethisDB can **automatically compact** the file on close when the tombstone ratio exceeds a configurable threshold:

```python
from aethisdb import Database

# auto_compact=True: compact on close if deleted ratio > compact_ratio
# compact_ratio=0.3 (default): compact when 30%+ of records are deleted
db = Database("small.aethisdb", auto_compact=True, compact_ratio=0.25)

users = db.create_collection("users")
for i in range(10):
    users.insert({"name": f"user_{i}"})
for i in range(5):
    users.delete({"name": f"user_{i}"})
# 50% deleted > 25% threshold → file is compacted automatically on close
db.close()
```

#### Manual Commit Mode

For workloads with many writes, use manual commit mode to reduce index write frequency:

```python
# Only write the index to disk when commit() or close() is called
db = Database("small.aethisdb", auto_commit=False)

users = db.create_collection("users")
for i in range(1000):
    users.insert({"name": f"user_{i}"})

db.commit()  # one index write instead of 1000
db.close()
```

#### Space Statistics

```python
db = Database("small.aethisdb")
stats = db.stats()
print(stats)
# {
#   'file_size': 4384,
#   'header_size': 64,
#   'data_size': 3415,
#   'index_size': 905,
#   'record_count': 50,
#   'deleted_count': 0,
#   'deleted_bytes_estimate': 0,
#   'collections': 1,
#   'auto_compact': False,
#   'auto_commit': True
# }
db.close()
```

#### Manual Compaction

Force a compaction at any time:

```python
db = Database("small.aethisdb")
users = db.get_collection("users")
users.delete({"age": {"$lt": 18}})
db.compact()  # immediately rewrite file, removing tombstones
db.close()
```

## Full Example

```python
from aethisdb import Database

with Database("company.aethisdb") as db:
    employees = db.create_collection("employees")
    employees.create_index("email")

    # Seed data
    employees.insert_many([
        {"name": "Alice",   "email": "alice@corp.com", "department": "Engineering", "salary": 90000},
        {"name": "Bob",     "email": "bob@corp.com",   "department": "Sales",       "salary": 65000},
        {"name": "Charlie", "email": "carol@corp.com", "department": "Engineering", "salary": 85000},
    ])

    # Query with index
    alice = employees.find_one({"email": "alice@corp.com"})

    # Range query
    high_earners = employees.find({"salary": {"$gte": 80000}})

    # Update
    employees.update({"email": "bob@corp.com"}, {"salary": 70000})

    # Delete
    employees.delete({"department": "Sales"})

    # Verify
    remaining = sorted(e["name"] for e in employees.all())
    print(remaining)  # ['Alice', 'Charlie']
```

---

## File Format: `.aethisdb`

### Overview

The `.aethisdb` file format is a custom binary layout designed for append-friendly record storage with a persistent JSON index at the end of the file. The format consists of three sections:

```
┌──────────────────┬──────────────────┬─────────────────────┐
│   Header (64B)   │  Data Section    │  Index Block (JSON) │
│  (fixed size)    │ (append-only)    │   (end of file)     │
├──────────────────┼──────────────────┼─────────────────────┤
│                  │ Collection       │ {                   │
│  Magic number    │   Markers        │   "version": 2,     │
│  Version         │ Data records     │   "collections": {  │
│  Flags           │                  │     "users": {      │
│  Header size     │                  │       ...           │
│  Index offset    │                  │     }               │
│  Index length    │                  │   }                 │
│  Data start      │                  │ }                   │
│  Collection count│                  │                     │
│  Reserved        │                  │                     │
└──────────────────┴──────────────────┴─────────────────────┘
```

### 1. Header (exactly 64 bytes)

The header is a fixed-size binary structure at the beginning of the file:

| Offset | Size | Type   | Field            | Description                              |
|--------|------|--------|------------------|------------------------------------------|
| 0      | 8    | `8s`   | Magic number     | Always `b"AETHISDB"`                    |
| 8      | 1    | `uint8`| Version          | Format version (currently `2`)          |
| 9      | 1    | `uint8`| Flags            | Reserved flags (currently `0`)         |
| 10     | 2    | `uint16`| Header size     | Always `64`                             |
| 12     | 4    | `uint32`| Reserved        | Zero                                     |
| 16     | 8    | `uint64`| Index offset     | Byte offset to the index block          |
| 24     | 8    | `uint64`| Index length     | Size of the index block in bytes        |
| 32     | 8    | `uint64`| Data start       | Always `64` (where data begins)         |
| 40     | 8    | `uint64`| Collection count | Number of collections                  |
| 48     | 16   | `16s`  | Reserved         | Zero-padded                             |

All integers are little-endian (struct format character `<`).

### 2. Data Section

The data section starts at byte offset 64 (right after the header). It contains a sequence of block entries. Two block types are supported:

#### Collection Marker Block

Written when a collection is created. Identifies the start of a collection's data.

| Offset | Size   | Type    | Field        | Description                        |
|--------|--------|---------|--------------|------------------------------------|
| 0      | 1      | `uint8` | Type         | `1` (COLLECTION_MARKER_TYPE)      |
| 1      | 1      | `uint8` | Name length  | Length of collection name (bytes) |
| 2      | var    | `bytes` | Name         | UTF-8 encoded collection name      |
| 2+n    | 8      | `uint64`| Next ID      | Next auto-increment record ID      |

#### Record Entry Block

Written for each record insert. Records are appended sequentially.

| Offset | Size   | Type    | Field       | Description                        |
|--------|--------|---------|-------------|------------------------------------|
| 0      | 1      | `uint8` | Type        | `2` (RECORD_ENTRY_TYPE)           |
| 1      | 1      | `uint8` | Status      | `0` = deleted, `1` = active       |
| 2      | 8      | `uint64`| Record ID   | Unique record identifier          |
| 10     | 4      | `uint32`| Data length | Length of JSON data (bytes)       |
| 14     | var    | `bytes` | Data        | JSON-encoded record payload       |

### 3. Index Block (JSON, end of file)

The index block is a JSON document appended after the last data byte. The header's `index_offset` and `index_length` fields locate it. On every commit, the entire index is rewritten.

Structure:

```json
{
  "version": 2,
  "collections": {
    "users": {
      "name": "users",
      "marker_offset": 64,
      "next_id": 3,
      "record_count": 2,
      "deleted_count": 1,
      "record_positions": {
        "0": 92,
        "1": 130
      },
      "indexes": {
        "email": {
          "type": "hash",
          "field": "email",
          "values": {
            "alice@example.com": [0],
            "bob@example.com": [1]
          }
        }
      }
    }
  }
}
```

**Key fields:**

| Field              | Type     | Description                                            |
|--------------------|----------|--------------------------------------------------------|
| `marker_offset`    | integer  | Byte offset of the collection marker in the data section|
| `next_id`          | integer  | Next auto-increment record ID                          |
| `record_count`     | integer  | Number of active (non-deleted) records                  |
| `deleted_count`    | integer  | Number of soft-deleted records                         |
| `record_positions` | object   | Maps `record_id` (as string) to byte offset in file    |
| `indexes`          | object   | Hash indexes for indexed fields                        |

### How Writes Work

AethisDB uses an **append-friendly** strategy:

1. **Insert**: A record is appended to the end of the data section. Its position is recorded in `record_positions`.
2. **Update**: The old record's status byte is set to `0` (deleted) in-place. A new record with the updated data is appended at the end. The `record_positions` map is updated to point to the new record. The old record remains as a tombstone.
3. **Delete**: The record's status byte is set to `0` (deleted) in-place. It is removed from `record_positions` and from all hash indexes.
4. **Commit**: The entire index block is serialized as JSON and written after the last data byte. The 64-byte header is updated with the new `index_offset` and `index_length`.

### How Reads Work

1. The header is read (first 64 bytes) to locate the index block.
2. The index JSON is parsed to reconstruct collection metadata and hash indexes in memory.
3. For single-record lookups (`get(id)`), the byte offset is retrieved from `record_positions`, the file is seeked to that position, and the record is read.
4. For indexed exact-match queries, the hash index directly maps field values to record IDs, avoiding any file scanning.
5. For non-indexed or range queries, all active records are scanned and filtered using the query engine.

### How Compaction Works

When `compact()` is called:

1. A temporary file is created.
2. All active records (status = `1`) are read from the current file and rewritten sequentially to the temp file.
3. Deleted records (tombstones) are skipped — they are not copied.
4. A fresh, compact index is written at the end of the temp file.
5. The temp file replaces the original file atomically (`os.replace`).
6. The result is a file with no wasted space from tombstones or orphaned index blocks.

## Query Operators

| Operator | Type      | Description                                          | Example                         |
|----------|-----------|------------------------------------------------------|---------------------------------|
| (default)| Exact match| Field equals value                                   | `{"name": "Alice"}`             |
| `$eq`    | Equality  | Field equals value                                   | `{"age": {"$eq": 30}}`          |
| `$ne`    | Inequality| Field does not equal value                           | `{"age": {"$ne": 30}}`          |
| `$gt`    | Comparison| Field greater than value                             | `{"age": {"$gt": 25}}`          |
| `$lt`    | Comparison| Field less than value                                | `{"age": {"$lt": 35}}`          |
| `$gte`   | Comparison| Field greater than or equal to value                 | `{"age": {"$gte": 30}}`         |
| `$lte`   | Comparison| Field less than or equal to value                    | `{"age": {"$lte": 30}}`         |
| `$in`    | Set       | Field value is in list                               | `{"role": {"$in": ["a", "b"]}}` |
| `$nin`   | Set       | Field value is not in list                           | `{"role": {"$nin": ["a"]}}`     |
| `$exists`| Existence | Field exists (true) or doesn't exist (false)         | `{"email": {"$exists": true}}`  |
| `$and`   | Logical   | All sub-conditions must match                        | `{"$and": [cond1, cond2]}`      |
| `$or`    | Logical   | At least one sub-condition must match                | `{"$or": [cond1, cond2]}`       |

## Running Tests

```bash
cd /path/to/aethisdb
python -m unittest discover -s tests -v
```

## Architecture

```
aethisdb/
├── __init__.py      Package exports (Database, Collection, Record, etc.)
├── database.py      Top-level Database class — file lifecycle & collection mgmt
├── collection.py    Collection class — CRUD operations & query optimization
├── storage.py       StorageEngine — binary file format I/O (header, records, index)
├── index.py         Hash-based Index & IndexManager for fast lookups
├── record.py        Record class — dict-like document with an ID
├── query.py         Query engine — condition parsing, evaluation, update application
├── utils.py         Constants and struct helpers for the binary format
└── exceptions.py    Custom exception hierarchy
```
