Metadata-Version: 2.4
Name: vcti-nputils
Version: 2.0.0
Summary: NumPy structured array utilities — dtype construction, field views, joining, enum mapping, position arrays, byte/string conversion for C++ interop, and growable (std::vector-like) arrays
Author: Visual Collaboration Technologies Inc.
License-Expression: LicenseRef-Proprietary
Project-URL: Repository, https://github.com/vcollab/vcti-python-nputils
Project-URL: Changelog, https://github.com/vcollab/vcti-python-nputils/blob/main/CHANGELOG.md
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: <3.15,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=2.0
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Provides-Extra: lint
Requires-Dist: ruff; extra == "lint"
Provides-Extra: typecheck
Requires-Dist: mypy; extra == "typecheck"
Requires-Dist: numpy; extra == "typecheck"
Provides-Extra: bench
Requires-Dist: tabulate; extra == "bench"
Dynamic: license-file

# NumPy Utils

NumPy structured array utilities — dtype construction, field views, joining, enum mapping, position arrays, byte/string conversion for C++ interop, and growable (std::vector-like) arrays.

## Overview

`vcti-nputils` collects the low-level NumPy helpers shared across the vcti
stack. Most of it is stateless functions over structured arrays and
dtypes — building and reshaping dtypes, taking zero-copy field views,
joining arrays, mapping enum values, and converting byte fields for
pybind11/C++ interop. Alongside those it provides two small stateful
containers for building a numpy array of unknown final size:
`GrowableArray` (append-only — the common case) and `DynamicArray`
(the same growth model plus soft deletion). Both grow on demand and hand
off a plain contiguous array via `to_numpy()` for the read-heavy phase.

**Which to use:** append-only? `GrowableArray`. Need to remove elements
mid-build? `DynamicArray`.

## Installation

```bash
pip install "vcti-nputils>=2.0.0"
```

### In `pyproject.toml` dependencies

```toml
dependencies = [
    "vcti-nputils>=2.0.0",
]
```

---

## Quick Start

```python
import numpy as np
from vcti.nputils import (
    as_ndarray,
    check_overflow,
    drop_fields,
    DynamicArray,
    fields_view,
    flatten_dtype,
    get_string,
    GrowableArray,
    join_struct_arrays,
    merge_adjacent_fields,
    name_array,
    position_array,
    rename_fields,
    set_string,
    structured_dtype,
    with_field_metadata,
)

# Join structured arrays horizontally
dt1 = np.dtype([('id', 'i4'), ('value', 'f8')])
dt2 = np.dtype([('name', 'U10')])
arr1 = np.array([(1, 1.5), (2, 2.5)], dtype=dt1)
arr2 = np.array([('Alice',), ('Bob',)], dtype=dt2)
joined = join_struct_arrays([arr1, arr2])
# dtype: [('id', 'i4'), ('value', 'f8'), ('name', 'U10')]

# Create a zero-copy view with selected fields
view = fields_view(joined, ['id', 'name'])

# Drop fields from a structured array (zero-copy)
clean = drop_fields(joined, ['value'])

# Build a structured dtype from a scalar dtype + names
coord_dt = structured_dtype('f8', ['x', 'y', 'z'])
# dtype([('x', '<f8'), ('y', '<f8'), ('z', '<f8')])

# Rename fields in a dtype
new_dt = rename_fields(dt1, {'id': 'node_id', 'value': 'temperature'})

# Flatten array fields into individual columns (default naming)
dt = np.dtype([('id', 'i4'), ('coords', 'f8', (3,))])
_, cols = flatten_dtype(dt)
# cols: ['id', 'coord_0', 'coord_1', 'coord_2']

# Flatten with explicit per-field names
_, cols = flatten_dtype(dt, field_names={'coords': ['x', 'y', 'z']})
# cols: ['id', 'x', 'y', 'z']

# Flatten with a custom format string
_, cols = flatten_dtype(dt, fmt="{name}[{dim}]")
# cols: ['id', 'coord[0]', 'coord[1]', 'coord[2]']

# Merge adjacent 'S' fields into one (pure dtype view). Multiple merges
# can be specified at once; same-field overlap and name collisions are
# validated before anything is returned.
dt = np.dtype([
    ('first', 'S4'), ('last', 'S6'),
    ('city', 'S8'), ('state', 'S2'),
    ('age', 'i4'),
])
merged = merge_adjacent_fields(dt, {
    'name':    ['first', 'last'],
    'address': ['city', 'state'],
})
# dtype([('name', 'S10'), ('address', 'S10'), ('age', '<i4')])

# Map numeric enum values to names
enum_dict = {1: 'ACTIVE', 2: 'INACTIVE', 3: 'PENDING'}
names = name_array(np.array([1, 2, 1, 3]), enum_dict)

# Convert counts to cumulative offsets
offsets = position_array(np.array([3, 2, 4, 1]))
# array([0, 3, 5, 9, 10])

# Safely coerce inputs to ndarray
arr = as_ndarray([1, 2, 3], dtype=np.float64)
empty = as_ndarray(None)  # array([], dtype=float64)

# Strings in byte fields, for C++/pybind11 interop: declare once which
# column holds each field's length, then read and write single rows
dt = np.dtype([('id', 'u4'), ('name_str', 'S1024'), ('name_len', 'u8')])
dt = with_field_metadata(dt, {'name_str': {'length_field': 'name_len'}})
materials = np.zeros(2, dtype=dt)
set_string(materials, 0, 'name_str', 'Structural steel S355')  # also sets name_len
name = get_string(materials, 0, 'name_str')                    # decodes this row only
overflow = check_overflow(materials, 'name_str')               # vectorized, decodes nothing

# Build an array incrementally without knowing the final size (append-only)
ga = GrowableArray(np.dtype([('id', 'i4'), ('value', 'f8')]))
ga.append((1, 1.5))
ga.extend([(2, 2.5), (3, 3.5)])
result = ga.to_numpy()  # independent, exact-size array you own

# Need to remove elements mid-build? DynamicArray adds soft deletion
da = DynamicArray(np.dtype('i8'))
da.extend([10, 20, 30, 40])
da.delete(1)            # elements shift: da[1] is now 30
clean = da.to_numpy()   # array([10, 30, 40])
```

> These are typed, numpy-backed accumulators, not faster `list`s. For a
> method-by-method comparison, measured trade-offs, and guidance on when to
> use them versus a Python `list`, see
> [docs/design/comparison.md](docs/design/comparison.md); for usage recipes,
> [docs/patterns.md](docs/patterns.md). Reproduce the numbers with
> `python benchmarks/bench_growable_array.py`.

---

## Module layout

Each category lives in its own module. All public functions are re-exported from `vcti.nputils`.

| Module | Functions |
|--------|-----------|
| `dtype_utils` | `structured_dtype`, `flatten_dtype` (+ `flatten_record_dtype` alias), `merge_adjacent_fields`, `rename_fields` |
| `view_utils` | `fields_view`, `drop_fields` |
| `join_utils` | `join_struct_arrays` |
| `mapping_utils` | `name_array`, `name_array_sparse` |
| `offset_utils` | `position_array` |
| `coerce_utils` | `as_ndarray` |
| `growable_array` | `GrowableArray` |
| `dynamic_array` | `DynamicArray` |
| `byte_utils` | `get_string`, `set_string`, `check_overflow`, `check_terminator`, `byte_field_dtype`, `with_field_metadata`, `get_encoding`, `with_encoding`, `string_from_bytes`, `bytes_from_string`, `StringOverflowWarning`, `UnknownFieldMetadataWarning`, `ZERO_CHAR` |

---

## Functions

### Dtype construction & transformation
| Function | Purpose |
|----------|---------|
| `structured_dtype(dtype, names)` | Build a structured dtype from a scalar or subdtype plus field names |
| `flatten_dtype(dt, *, field_names, fmt, strip_plural)` | Expand array fields into scalars with flexible naming |
| `flatten_record_dtype(dt, ...)` | Legacy alias for `flatten_dtype` |
| `merge_adjacent_fields(dt, merges)` | Merge one or more groups of adjacent 'S' fields into a single field each (pure dtype view) |
| `rename_fields(dt, mapping)` | Return a new dtype with fields renamed |

### Zero-copy views
| Function | Purpose |
|----------|---------|
| `fields_view(sa, fields)` | View containing only the selected fields |
| `drop_fields(sa, exclude)` | View containing all fields except those excluded |

### Joining
| Function | Purpose |
|----------|---------|
| `join_struct_arrays(arrays)` | Join structured arrays horizontally by combining fields |

### Mapping, offsets, coercion
| Function | Purpose |
|----------|---------|
| `name_array(nparray, enum_dict, default, *, max_lookup_bytes)` | Map integer values to names through a dense lookup array (keys that mostly fill their range) |
| `name_array_sparse(nparray, enum_dict, default)` | Same result for keys scattered over a wide range; cost independent of the key range |
| `position_array(counts, dtype)` | Convert count array to cumulative offset array |
| `as_ndarray(value, dtype)` | Coerce None, list, or ndarray to ndarray |

### Containers
| Type | Purpose |
|------|---------|
| `GrowableArray(dtype, initial_capacity, growth_factor)` | Append-only growable numpy array: amortized O(1) `append`/`extend` (`append_get_index` returns the index), `reserve`/`shrink_to_fit`/`clear`, `full`/`resize` for sized fills, zero-copy `as_array()` view, and `to_numpy()` for an independent copy |
| `DynamicArray(dtype, initial_capacity, growth_factor)` | Same growth model **plus soft deletion**: lazy `delete` (shifting semantics), `compact`, `active_indices` |

### Byte / string conversion (pybind11 interop)
| Function | Purpose |
|----------|---------|
| `get_string(sa, row, field_name, *, length_field, encoding, on_overflow)` | Decode one row's string from the raw field bytes — exactly `length` bytes with a paired length column, else up to the first zero code unit (C semantics) |
| `set_string(sa, row, field_name, value, *, length_field, encoding, on_overflow)` | Encode one row C++-style (text in at most `itemsize - 1` bytes, rest zero-filled), set its length column, and return the original byte length |
| `check_overflow(sa, field_name, length_field)` | Rows whose original byte length exceeded the field's payload — vectorized |
| `check_terminator(sa, field_name)` | Rows whose terminator (last) byte is not `\x00` — vectorized |
| `byte_field_dtype(max_chars, encoding, bytes_per_char)` | Dtype of a field holding any string of `max_chars` characters (`S<4 * (max_chars + 1)>` by default, the C++ `char[4 * N]` rule), with its properties in the metadata |
| `with_field_metadata(dtype, attrs, *, on_unknown)` | Attach `encoding`, `bytes_per_char` and `length_field` to fields — e.g. of an array from C++, via `arr.view(...)` |
| `get_encoding(dtype, default)` / `with_encoding(dtype, encoding)` | Read / attach a scalar dtype's encoding |
| `string_from_bytes(value, encoding)` / `bytes_from_string(value, length, encoding)` | Scalar primitives: one value to a string; a string to `(field_bytes, original_length)` |
| `StringOverflowWarning`, `UnknownFieldMetadataWarning` | Warning categories for the `"warn"` policy |
| `ZERO_CHAR` | The null character (`"\x00"`) |

---

## Dependencies

- [numpy](https://numpy.org/) (>=2.0)

---

## Documentation

| If you want to… | Read |
|---|---|
| Get started using the package | Quick Start above |
| See practical, real-world usage | [docs/patterns.md](docs/patterns.md) |
| Understand the architecture and design decisions | [docs/design.md](docs/design.md) |
| Choose between a container and a Python `list` | [docs/design/comparison.md](docs/design/comparison.md) |
| Navigate and understand the source | [docs/source-guide.md](docs/source-guide.md) |
