Metadata-Version: 2.4
Name: dictionquery
Version: 0.1.0
Summary: Declarative aggregation queries over lists of dicts
License-Expression: MIT
License-File: LICENSE
Author: matthew
Author-email: beattyml1@gmail.com
Requires-Python: >=3.11
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Description-Content-Type: text/markdown

# dictionquery

[![CI](https://github.com/beattyml1/dictionquery/actions/workflows/ci.yml/badge.svg)](https://github.com/beattyml1/dictionquery/actions/workflows/ci.yml)

Declarative aggregation queries over lists of dicts — no dataframe, no schema, no dependencies.

```python
from dictionquery import count, fields, overall, run, values

data = [
    dict(a='x', b=1, c=None),
    dict(a='y', b=2, c=None),
    dict(a='z', b=2, c=None),
]

run(data, [overall('total', count()), fields(values(count()))])
# {'total': 3,
#  'a': {'x': 1, 'y': 1, 'z': 1},
#  'b': {1: 1, 2: 2},
#  'c': {None: 3}}
```

That query says: count the rows overall, and for every field, count how often each
distinct value appears. Point it at a JSON dump you have never seen before and it will
tell you what is in there.

## Install

```bash
pip install dictionquery      # or: poetry add dictionquery
```

Requires Python 3.11+. The runtime has no dependencies.

## Concepts

Three kinds of callable compose to make a query.

| Kind | Shape | Examples |
| --- | --- | --- |
| **Aggregate** | `Iterable[Any] -> Any` | `count()`, `values(count())` |
| **Query part** | `list[dict] -> dict` | `fields(...)`, `field(name, ...)`, `overall(name, ...)` |
| **Predicate** | `item -> bool` | `positive`, `not_(none)` |

A **query** is any iterable of query parts. `run` hands the whole dataset to each part
and merges the fragments they return:

```python
run(data, {overall('rows', count()), fields(count())})
```

Aggregates nest, which is where the expressiveness comes from — `values(count())` is
"group by distinct value, then count each group", and `fields(values(count()))` applies
that to every field.

### Filters take an item

Every `filter` argument is a predicate over a single item, returning whether that item
takes part. **An item is whatever the surrounding construct aggregates** — so the same
`count(...)` call filters different things depending on where you put it:

```python
sales = [
    dict(region='north', units=3, rep='ann'),
    dict(region='south', units=3, rep='bob'),
    dict(region='north', units=7, rep='cal'),
]

# under overall(), rows are the items
overall('north', count(lambda row: row['region'] == 'north'))(sales)
# {'north': 2}

# under fields(), one field's values are the items
fields(count(positive))(sales)
# {'region': 0, 'units': 3, 'rep': 0}
```

The `filter` on `fields` and `overall` themselves is the exception that proves the rule:
both choose which *records* the query part looks at, so both take a row. The two stack —
a row filter narrows the records, then a value filter narrows what is aggregated from
them:

```python
big = lambda value: isinstance(value, int) and value > 5
north = lambda row: row['region'] == 'north'

fields(count(big), north)(sales)   # {'region': 0, 'units': 1, 'rep': 0}
```

An aggregate used on its own filters whatever you hand it:

```python
count(positive)([10, 0, -5])   # 1
```

## Query parts

### `fields(aggregate, filter=None, default=NOT_SET)`

Applies `aggregate` to each field across the dataset, keyed by field name in first-seen
order. `filter` is a row predicate; rejected rows contribute no values and cannot
introduce a field. Filters nested inside `aggregate` test one field value at a time.

Rows that lack a field are skipped. Pass `default=` to substitute a value instead:

```python
data = [dict(a=1), dict(a=2, b=3)]

fields(count())(data)                  # {'a': 2, 'b': 1}   — b missing from row 1
fields(list, default=None)(data)       # {'a': [1, 2], 'b': [None, 3]}
```

### `field(name, aggregate, filter=None, default=NOT_SET)`

The single-field counterpart of `fields`, and its **override**. Put both in a query and
`name` is reported this way while every other field falls back to the general treatment:

```python
run(data, {fields(count()), field('b', values(count()))})
# {'a': 3, 'b': {1: 1, 2: 2}, 'c': 3}   — b broken down, everything else counted
```

`run` applies the override whichever order the query is in — parts that name what they
produce (`field`, `overall`) are applied after the general `fields` — so an unordered
query like a `set` still expresses an override. Result keys stay in the order the query
gave them; overriding a field's value does not move its key.

Because it sees one field, the aggregate can be type-specific in a way `fields` could not
risk:

```python
field('price', sum)(orders)      # safe; fields(sum) would hit a text field and raise
```

Its `filter` narrows only this part — the rest of the query still sees every row. A field
no row carries still gets an entry, from the aggregate over nothing:

```python
field('nope', count())(data)     # {'nope': 0}
```

### `overall(name, aggregate, filter=None)`

Applies `aggregate` to the **rows themselves** and publishes the result under `name`.

```python
overall('total', count())(data)                                   # {'total': 2}
overall('adults', count(), lambda row: row['age'] >= 18)(people)
```

### `run(data, query)`

Runs every part over the same dataset and merges the results. `data` may be any iterable
of rows (it is materialized once and never mutated); `query` may be a set, list, or tuple.
Every part sees the whole dataset, so a `filter` on one part never narrows another.

Parts that name what they produce (`field`, `overall`) are applied after the general
`fields`, which is what makes overriding work from an unordered query. If two *equally
specific* parts produce the same key the later one wins — and within a `set` "later" is
not predictable, so keep those names distinct.

## Aggregates

### `count(filter=None)`

Counts the items it is given, falsy ones included. With a `filter`, counts only matching
items — field values under `fields`, rows under `overall`.

```python
count()([None, 0, ''])                             # 3
fields(count(positive))(sales)                     # per field, positive values only
overall('big', count(lambda r: r['units'] > 5))    # rows with big orders
```

### `values(aggregate, filter=None)`

Groups items by distinct value (first-seen order, hashable values), then applies
`aggregate` to each group.

```python
values(count())(['x', 'y', 'y'])  # {'x': 1, 'y': 2}
values(list)(['x', 'y', 'y'])     # {'x': ['x'], 'y': ['y', 'y']}
```

Note the difference between filtering the group and filtering what is counted in it: a
`filter` on `values` drops items before grouping, so a value only they carried gets no
entry at all, while a `filter` on the inner `count` keeps the entry and shrinks its
number.

```python
fields(values(count(), lambda v: v != 'south'))(sales)['region']  # {'north': 2}
fields(values(count(big)))(sales)['units']                        # {3: 0, 7: 1}
```

### Bring your own

Any callable over an iterable is an aggregate, so `list`, `set`, `max`, `sum`, and
`statistics.mean` all drop straight in:

```python
fields(list)(people)                    # every value of every field
fields(max)(people)                     # per-field maximum
```

Under `fields` your aggregate gets that field's values; under `overall` it gets the rows.

Note that `fields` applies the aggregate to *every* field, so a type-specific aggregate
like `sum` will raise on a text field. Name the field instead:

```python
field('score', sum)(people)                              # {'score': 13}
run(people, {fields(count()), field('score', sum)})      # sum that one, count the rest
```

## Predicates

Filters are ordinary predicates, and `dictionquery` re-exports a vocabulary of them so
most filters need no lambda.

| Group | Names |
| --- | --- |
| Transformers | `not_(p)`, `and_(*ps)`, `or_(*ps)` |
| Constants | `always`, `never` |
| Emptiness | `none`, `not_none`, `empty`, `none_or_empty`, `none_or_empty_or_whitespace` |
| Numbers | `number`, `whole_number`, `positive`, `negative`, `zero`, `none_or_zero` |

These are plain functions, so pass them by name — `count(positive)`, not
`count(positive())`. Only the transformers are called.

```python
from dictionquery import and_, count, none_or_empty_or_whitespace, not_, number, negative

count(not_(none_or_empty_or_whitespace))(notes)   # notes with actual text
count(and_(number, not_(negative)))(scores)       # 0 and up, ignoring None and strings
```

They drop straight into a `fields` query, where the items are values. Where the item is a
row — an aggregate under `overall`, or the `filter` on `fields`/`overall` itself — pull
the field out with a lambda first:

```python
fields(count(not_(none_or_empty_or_whitespace)))(people)             # value-shaped
overall('scored', count(lambda row: number(row.get('score'))))(people)   # row-shaped
```

Details worth knowing:

- **Nothing raises on an off-type value.** `positive('nope')` is `False`, not a
  `TypeError` — the fields of a dict dataset are rarely uniform, and a filter that
  explodes on row 3 is useless.
- **`empty` is length-based.** `empty(0)` is `False`; `0` is not an empty container.
  That is the distinction a bare truthiness check gets wrong.
- **`bool` is not a number.** `positive(True)` is `False` despite `bool` subclassing
  `int`, because a `True` in a data field is a flag, not the quantity 1.
- **`whole_number` means "no fractional part"**, orthogonal to sign: `-7` and `3.0`
  qualify, `3.5` does not. Compose the other readings — `and_(whole_number, positive)`
  for the counting numbers, `and_(whole_number, not_(negative))` for the non-negative
  integers.
- **`NaN` and infinity** fail `positive`, `negative`, `zero`, and `whole_number`.

## `NOT_SET`

The sentinel meaning "this row has no such field", distinct from a stored `None`. It is
the default for `fields(..., default=)` and never appears in a result.

## Development

```bash
poetry install
poetry run pytest                              # test suite
poetry run pytest --doctest-modules dictionquery   # docstring examples
python examples.py
```

CI runs all three on Python 3.11, 3.12, and 3.13 for every push and pull request.

## Releasing

| Branch | Goes to | Version published |
| --- | --- | --- |
| `develop` | [TestPyPI](https://test.pypi.org/project/dictionquery/) | `<version>.dev<run number>` — a fresh build every push |
| `main` | [PyPI](https://pypi.org/project/dictionquery/) | exactly what `pyproject.toml` says |

Both publish only after the full test matrix passes. A push to `main` that does not bump
the version is a no-op rather than a failure (`skip-existing`), so shipping a release
means bumping the version and pushing:

```bash
poetry version patch     # or minor / major
git commit -am "Release $(poetry version --short)" && git push origin main
```

### One-time setup

Publishing uses [trusted publishing](https://docs.pypi.org/trusted-publishers/) — OIDC,
no API tokens stored in the repo. On **each** of PyPI and TestPyPI, add a pending
publisher under *Your projects → Publishing*:

| Field | Value |
| --- | --- |
| PyPI project name | `dictionquery` |
| Owner | `beattyml1` |
| Repository | `dictionquery` |
| Workflow | `ci.yml` |
| Environment | `pypi` on PyPI, `testpypi` on TestPyPI |

The environment names must match the `environment:` keys in the workflow. To use API
tokens instead, drop the `environment:` and `permissions: id-token` blocks and give the
publish step `password: ${{ secrets.PYPI_API_TOKEN }}`.

## License

MIT — see [LICENSE](LICENSE).

