Metadata-Version: 2.4
Name: scistack-db
Version: 0.1.11
Summary: A lightweight database framework for scientific computing with versioning and provenance tracking
License: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.9
Requires-Dist: duckdb>=0.9.0
Requires-Dist: numpy>=1.20.0
Requires-Dist: pandas>=1.3.0
Requires-Dist: scicanonicalhash>=0.1.0
Requires-Dist: sciduckdb>=0.1.0
Requires-Dist: scifor>=0.1.0
Requires-Dist: scipathgen>=0.1.0
Requires-Dist: scistacklog>=0.1.0
Provides-Extra: dev
Requires-Dist: matplotlib; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Description-Content-Type: text/markdown

# SciDB

Database operations layer for SciStack. Provides abstractions for defining typed variables, configuring the database, and saving/loading data by metadata.

## Named Filters (value-based, reusable)

`Filter` objects are first-class values — assign them to a variable and reuse
them in any `where=` clause, or compose them with `&` / `|` / `~`:

```python
# Define once (this IS a ColumnFilter — no special wrapper needed)
clean_gr = GAITRiteLoadedCycle["StepLengths_GR"] != 0

# Use anywhere
for_each(mean_change_from_reference,
         inputs={"baseline": Fixed(GAITRiteLoadedCycle["StepLengths_GR"], session="BL"),
                 "value":    GAITRiteLoadedCycle["StepLengths_GR"]},
         outputs=[DeltaStepLength],
         where=clean_gr & (UAStartFoot() == "A"))

# Compose named filters
clean_and_unilateral = clean_gr & (UAStartFoot() == "U")
for_each(..., where=clean_and_unilateral)
```

MATLAB equivalent:

```matlab
clean_gr = GAITRiteLoadedCycle("StepLengths_GR") ~= 0;
scidb.for_each(@meanChangeFromReference, ..., where=clean_gr & (UAStartFoot() == "A"));
```

## Permanent Schema-Level Exclusions

For data that should be excluded from **every** analysis (e.g., a failed
recording session), use the exclusion registry instead of per-call filters.

```python
# Mark a specific trial as excluded (persisted in the database)
scidb.exclude_schema(subject=1, trial=2,
                     reason="equipment malfunction during recording")

# Exclude an entire subject (trial omitted = wildcard)
scidb.exclude_schema(subject=3, reason="participant withdrew")

# Inspect currently-excluded combinations
exclusions_df = scidb.list_exclusions()

# Re-include (logged; the original exclusion row is preserved)
scidb.include_schema(subject=1, trial=2,
                     reason="re-reviewed video, recording was valid")
```

MATLAB equivalent:

```matlab
scidb.exclude_schema("equipment malfunction", 'subject', 1, 'trial', 2)
scidb.include_schema("re-reviewed, recording was valid", 'subject', 1, 'trial', 2)
tbl = scidb.list_exclusions()
```

Exclusions are applied automatically by `for_each` before the iteration loop.
The full exclusion table is hashed and stored in `version_keys`
(`__schema_overrides_hash`) so that adding or removing an exclusion
invalidates cached results.

```python
from scidb import configure_database, BaseVariable
import numpy as np

db = configure_database("experiment.duckdb", ["subject", "session"])

class RawSignal(BaseVariable):
    schema_version = 1

RawSignal.save(np.array([1, 2, 3]), subject=1, session="A")
(raw,) = RawSignal.load(subject=1, session="A")  # generator; unpack for single result
all_versions = list(RawSignal.load(subject=1, session="A", version="all"))
```
## Logging

Every scidb operation logs through the shared [`scistacklog`](../scistacklog/README.md) facade. `configure_database()` points the file sink at `scidb.log` next to the database file and writes a run-context header (package versions, Python version, pid), so each log file is self-describing.

Two sinks with independent levels, both defaulting to INFO:

- **console** (stderr): the pipeline narrative — for_each banner, periodic progress, run summary with failure reasons, `[timing]` summaries.
- **file** (`scidb.log`): the same narrative, with date + millisecond timestamps and the originating layer (`[scidb]`, `[scifor]`, `[matlab]`, …) on every line; one record per line.

For debugging, raise a sink to DEBUG to capture the full execution trace (named internal steps with durations, per-iteration `[run]`/`[skip]` lines, per-phase `[timing]` tables, column/dtype dumps):

```python
from scidb import Log

Log.set_level("DEBUG", sink="file")   # full detail in scidb.log only
Log.set_level("DEBUG")                # both sinks
```

or set the `SCIDB_LOG_LEVEL` environment variable, or pass `-v` to the `scidb` CLI.
