Metadata-Version: 2.4
Name: nominal-iads
Version: 0.3.0
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: Microsoft :: Windows
Classifier: Typing :: Typed
Requires-Dist: nominal>=1.180.1,<2
Summary: Scan IADS archives and export data to Nominal (Python bindings).
License: Apache-2.0
Requires-Python: >=3.11
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# nominal-iads

Read an IADS archive from Python and upload its channel data to a Nominal dataset.
The package handles extraction, temporary Parquet files, and concurrent uploads.
You choose the archive, destination, and optional channel/time filters.

**Windows x64 only · Python 3.11+ (tested on 3.11–3.14) · Licensed IADS required**

[Install](#1-install) · [Inspect an archive](#2-check-your-archive) ·
[Upload to Nominal](#3-upload-to-nominal) · [Read windows and UTC](#read-windows-archive-irig-time-versus-real-utc-time) · [Options](#upload-options) ·
[Tables and metadata sources](#archive-metadata-event-logs-and-configuration-tables) · [Troubleshooting](#troubleshooting)

## 1. Install

Run in PowerShell with 64-bit Python 3.11 or newer:

```powershell
pip install nominal-iads
```

The package includes its worker and the Nominal SDK; install and license IADS separately.

IADS must be installed and its COM component registered. Use the DLL from your **64-bit IADS
installation**. If it has not already been registered, run this in an **elevated 64-bit PowerShell**
(Run as administrator), adjusting the DLL path to your installation:

```powershell
regsvr32 "C:\Program Files\IADS\Common\IadsDataInterfaces.dll"
```

Confirm registration succeeds. Repeat it if IADS moves or registration is damaged.

Verify activation from 64-bit PowerShell under the account that runs Python:

```powershell
New-Object -ComObject IadsDataInterfaces.iadsread
```

This checks IADS activation, not archive access. Local inspection needs no Nominal credentials.

## 2. Check your archive

Replace the placeholder path with your local IADS archive path:

```python
from nominal.iads import Archive

with Archive(r"C:\data\example-archive") as archive:
    info = archive.info()
    print("Archive time span:", info.start_time, "to", info.end_time)
    names = archive.channel_names()
    print(f"{len(names)} extractable channels")
    print("First channels:", names[:10])
```

Keep archive operations inside the `with` block. Leaving it releases the archive's
COM connection; using the archive outside that block raises `RuntimeError`.

For channel metadata, use `archive.channels()` (a dictionary keyed by channel name).
`archive.read_groups()` shows the rate-aligned groups used for extraction.

## 3. Upload to Nominal

You need a Nominal API token, your tenant's API base URL, and access to the destination
workspace. Find the API URL in Nominal's API-key settings; do not substitute the web-app URL.
See the [Nominal authentication guide](https://nominal-io.github.io/nominal-client/guides/authentication/)
for tenant and workspace setup.

### Configure a connection profile once

Replace the placeholders with your Nominal credentials and destination workspace:

```powershell
nom config profile add "<profile_name>" --token "<api_token>" --base-url "<url>" --workspace-rid "<workspace_rid>"
```

Reuse an existing profile or load the saved profile with
`NominalClient.from_profile("<profile_name>")`.

### Run your first upload

Save this as `upload.py`, replacing the archive path and profile name.
It creates a dataset in your Nominal workspace. Run it with `python upload.py`.

```python
from nominal.core import NominalClient
from nominal.iads import Archive, enable_logging

enable_logging("info")
client = NominalClient.from_profile("<profile_name>")
dataset = client.create_dataset("Example IADS archive")

with Archive(r"C:\data\example-archive") as archive:
    with archive.upload_to_nominal(client, dataset, raise_on_failure=True) as run:
        stats = run.join()

print(f"Uploaded {stats.files_uploaded} files and {stats.points_uploaded} channel values")
print("Cancelled:", stats.cancelled)
print("Dataset:", dataset.rid)
```

**Call `join()` inside the upload context.** It waits for extraction and upload, returns
final counts, and surfaces errors. Exiting the context without joining cancels the run
and waits for cleanup; cleanup alone does not report its errors.

Progress and `cancel()` remain available while another thread waits in `join()`.
After joining, you can still inspect progress and drain successful ingest-job handles.
Complete extraction/upload failure raises unless the run was cancelled; `raise_on_failure=True` also raises
on partial failure. Both retain the report in `NominalIadsError.stats`:

```python
from nominal.iads import NominalIadsError

try:
    stats = run.join()
except NominalIadsError as error:
    if error.stats is not None:
        print("Uploaded:", error.stats.files_uploaded)
        print("Extraction failures:", error.stats.extraction_failures)
        for failure in error.stats.failures:
            print(failure.path, failure.error)
    raise
```

Call `join()` only once; the snippet above replaces the quick-start call.
Startup errors have `stats=None` because no run has completed.

`join()` confirms that successful files were uploaded and ingest was requested. It does
**not** wait for Nominal's server-side ingest to finish. Open the destination dataset in
Nominal to check ingest status and view the channels.

To append to an existing dataset, replace `create_dataset(...)` with:

```python
import os

dataset = client.get_dataset(os.environ["NOMINAL_DATASET_RID"])
```

Rerunning `create_dataset(...)` creates another dataset. Uploading again to an existing
dataset appends data; this library does not deduplicate repeated uploads.

## Select channels and time windows

Options are keyword arguments; the defaults work for a first upload. Preview a channel
selection before uploading. This example uploads only names containing `accel` or `gyro`,
excluding names ending in `_raw`:

```python
from nominal.iads import Archive

# Reuse the authenticated client and destination dataset from the quick start.
with Archive(r"C:\data\example-archive") as archive:
    include = ["accel", "gyro"]
    exclude = ["_raw$"]
    names = archive.extract_channel_names(
        channel_whitelist=include,
        channel_blacklist=exclude,
    )
    print(f"Selected {len(names)} channels")
    if not names:
        raise ValueError("No channels matched; check the patterns before uploading")

    with archive.upload_to_nominal(
        client,
        dataset,
        channel_whitelist=include,
        channel_blacklist=exclude,
        raise_on_failure=True,
    ) as run:
        stats = run.join()
```

Patterns are regular expressions with substring matching; use `^` and `$` for exact
boundaries. An empty whitelist includes all extractable channels. `priority_channels`
accepts exact names, not regexes.

## Read windows: archive IRIG time versus real UTC time

**Use the timestamps recorded in the archive to select data. Use a separate UTC anchor
when you know when that data actually occurred.** These are two different decisions.

IADS archives can use a file-local time reference. Their IRIG start/end values and
`flight_date` metadata are not necessarily an accurate record of real-world time.
For example, an archive may start at `001:00:00:00.000` even though the recording was
made in July. Do not assume that this means the recording actually began on January 1.

| Value | What it controls |
| --- | --- |
| `archive.info().start_time` / `.end_time` | The archive's recorded IRIG coordinates; use these to understand its read range |
| `archive.info().flight_date` | Calendar metadata used for default date resolution; it may need independent verification |
| `between=(start, end)` | Which part of the archive to read, expressed in that archive's IRIG coordinates |
| `absolute_read_start_time` | The real UTC instant corresponding to the requested `start` in `between` |

IRIG strings use `DDD:HH:MM:SS.sss`. They carry a day number and time, but no year or
time zone. A read bound is a coordinate on the archive's clock, **not an elapsed duration**
from opening the file and not a UTC timestamp. Inspect the actual values first:

```python
from nominal.iads import Archive

with Archive(r"C:\data\example-archive") as archive:
    info = archive.info()
    print("Recorded IRIG range:", info.start_time, "to", info.end_time)
    print("Recorded flight date:", info.flight_date)
```

### Select a range on the archive's clock

Suppose an archive starts at `001:00:00:00.000`. To read from five minutes through
fifteen minutes into that recording, use:

```python
from nominal.iads import IrigTimestamp

start = IrigTimestamp.from_irig("001:00:05:00.000")
end = IrigTimestamp.from_irig("001:00:15:00.000")
# Pass between=(start, end) to archive.upload_to_nominal(...).
```

If the archive instead starts at `123:10:00:00.000`, the same elapsed window is
`123:10:05:00.000` through `123:10:15:00.000`. Copying the first example's day and clock
time would select the wrong range, potentially returning no data. Channel coverage
can further limit which samples exist within the requested window.

### Put that window at the correct time in Nominal

Continuing the first example, suppose an independent source tells you the recording
actually began at **2026-07-01 14:00:00 UTC**. The selected window begins five minutes
later, so its UTC anchor must be **14:05:00**, not 14:00:00:

```python
from datetime import datetime, timezone

from nominal.iads import Archive, IrigTimestamp

# Reuse client and dataset from the quick start.
# These are illustrative times: replace them with your archive's coordinates
# and an independently known UTC time for the START OF THE SELECTED WINDOW.
start = IrigTimestamp.from_irig("001:00:05:00.000")
end = IrigTimestamp.from_irig("001:00:15:00.000")
read_start_utc = datetime(2026, 7, 1, 14, 5, tzinfo=timezone.utc)

with Archive(r"C:\data\example-archive") as archive:
    with archive.upload_to_nominal(
        client,
        dataset,
        between=(start, end),
        absolute_read_start_time=read_start_utc,
        raise_on_failure=True,
    ) as run:
        stats = run.join()
```

This maps archive coordinate `001:00:05:00.000` to `2026-07-01 14:05:00 UTC`;
a sample at `001:00:06:00.000` maps to `14:06:00 UTC`. The mapping is:

```text
sample UTC = absolute_read_start_time + (sample IRIG - requested start IRIG)
```

The anchor changes the time assigned to samples in Nominal; it does not change which
archive samples are read. It anchors the requested start even if a particular channel's
first available sample occurs later. For a whole-archive read without `between`, the
anchor corresponds to the archive's recorded start instead.

### Defaults and date validation

If you omit `absolute_read_start_time`, the library derives an absolute timeline from
the archive's IRIG values and flight-date metadata. Use that default only when those
metadata describe the timeline you intend to see in Nominal.

`validate_flight_date=True` checks calendar consistency when resolving the archive span.
It does **not** prove that the metadata match the real recording time. Setting it to
`False` skips that consistency check; it does **not** correct a wrong clock, select a
different read window, or provide a real UTC anchor. The flight-date year is still used
for calendar validation of read bounds, even with an explicit UTC anchor.

When the absolute date is unreliable but you know the real time of the selected window,
provide both `between` and `absolute_read_start_time` as above. If you do not know that
real time, the library cannot infer it from an arbitrary file-local clock.

## Upload options

| Option | Default | Use |
| --- | --- | --- |
| `between` | Entire archive | Select an IRIG time window |
| `absolute_read_start_time` | From archive metadata | Supply a known UTC read-window start |
| `validate_flight_date` | `True` | Require the flight date within the archive's IRIG span |
| `priority_channels` | None | Read groups containing these exact channel names first |
| `tags` | None | Key/value tags on ingested files |
| `channel_prefix` | None | Prefix every ingested channel name |
| `files_in_flight` | `4` | Maximum concurrent file uploads |
| `num_workers` | `0` (auto) | Number of isolated extraction worker processes |
| `raise_on_failure` | `False` | Make `join()` raise on partial upload/extraction failures |

`read_batch_size` and `write_batch_size` count **channel values**, not bytes or rows.
Use `help(Archive.upload_to_nominal)` for the complete signature.

## Progress, cancellation, and partial failures

Reuse `client` and `dataset` from above. Poll at a modest interval:

```python
import time

from nominal.iads import Archive

with Archive(r"C:\data\example-archive") as archive:
    with archive.upload_to_nominal(client, dataset, raise_on_failure=False) as run:
        while not run.is_finished():
            progress = run.upload_progress()
            print(f"Uploaded {progress.files_uploaded} files; {progress.upload_mb_per_second:.1f} MiB/s")
            time.sleep(0.5)
            # To stop early: run.cancel()
        stats = run.join()

print("Cancelled:", stats.cancelled)
for failure in stats.failures:
    print("Upload failure:", failure)
for message in stats.extraction_failures:
    print("Extraction failure:", message)
```

With `raise_on_failure=False`, partial failures are returned in `stats.failures` and
`stats.extraction_failures`; inspect both. Fatal infrastructure errors and an all-failed
upload still raise. With `True` (used in the quick start), any recorded upload or
extraction failure raises. Cancellation can still produce partial successful counts;
check `stats.cancelled`. Cancelling does not undo files already uploaded to Nominal.

Call `join()` only once. After it, use the returned stats; `upload_progress()` and
`drain_results()` are no longer available. Before joining, `extract_progress()` gives
an extraction snapshot and `drain_results()` returns newly uploaded files.

Extraction uses temporary disk space; allow room for Parquet files awaiting upload.
Temporary files are cleaned up after extraction and upload have stopped.

## Archive metadata, event logs, and configuration tables

An archive contains more than sampled channels. You can read event markers, time
segments, test points, analysis logs, and configuration records without uploading
anything to Nominal. All of these calls belong inside `with Archive(...)`.

### Read event markers and time segments

```python
from nominal.iads import Archive

with Archive(r"C:\data\example-archive") as archive:
    for event in archive.event_markers():
        print(event.time, event.comment, event.group)
        print("Additional columns:", event.extras)

    for segment in archive.time_segments():
        print(segment.start, segment.end, segment.description)
```

These methods return lists of typed records with read-only properties. Times are
`IrigTimestamp` values on the **archive's clock**, not automatically corrected UTC
values. The upload's `absolute_read_start_time` does not transform records returned
by these methods; use the same explicit time mapping described in the read-window
section when relating annotations to the uploaded channels.

Reading these tables does not create Nominal events, annotations, or test points.
`upload_to_nominal()` uploads channel samples; transferring log/configuration records
to Nominal is a separate application step.

### Available tables

| `Archive` method | IADS table | Contents |
| --- | --- | --- |
| `event_markers()` | `EventMarkerLog` | Point-in-time comments, user and group information |
| `time_segments()` | `TimeSegmentLog` | Labeled start/end ranges |
| `test_points()` | `TestPointLog` | Recorded test points, maneuvers and time ranges |
| `actual_flutter_test_points()` | `ActualFlutterTestPointsLog` | Recorded flutter test points and conditions |
| `actual_loads_test_points()` | `ActualLoadsTestPointsLog` | Recorded loads test points and conditions |
| `analysis_log()` | `AnalysisLog` | Parameter analysis, frequency and damping results |
| `threshold_log()` | `ThresholdLog` | Parameter threshold criteria |
| `peaks_log()` | `PeaksLog` | Peak records |
| `data_edits()` | `DataEditLog` | Recorded data edits |
| `selections_log()` | `SelectionsLog` | Saved selections |
| `planned_test_points()` | `PlannedTestPoints` | Planned test points |
| `planned_flutter_test_points()` | `PlannedFlutterTestPoints` | Planned flutter test points |
| `planned_loads_test_points()` | `PlannedLoadsTestPoints` | Planned loads test points |
| `mission_attributes()` | `MissionAttributes` | Mission attributes |
| `constants()` | `Constants` | Configuration constants |
| `classifications()` | `Classifications` | Classification records |

Not every archive populates every table. Empty, absent, and unknown tables all return
`[]`: IADS does not distinguish these cases in the query result. A misspelled table name
can therefore look like an empty table. Genuine COM/query failures still raise errors.

Typed properties can be `None` when the source column is absent, empty, or cannot be
converted to the expected type. Unmapped columns, including names that are awkward
Python identifiers, are retained in `record.extras`. Malformed mapped values are also
retained there rather than silently discarded. Values in `extras` are `str`, `float`,
or `None`. Use `help(type(record))` to inspect the properties of a returned record.

### Read a table without a typed wrapper

Use `read_table(name)` for another table exposed by IADS, or when you want the original
column names and values instead of the typed view:

```python
from nominal.iads import Archive

with Archive(r"C:\data\example-archive") as archive:
    for record in archive.read_table("EventMarkerLog"):
        print(record.fields.get("Comment"))
        print("Raw Time:", record.fields.get("Time"))
        print("Parsed IRIG Time:", record.times.get("Time"))
```

Each `IadsRecord` exposes two dictionaries:

- `fields`: every returned column under its original name, with `str`, `float`, or `None` values.
- `times`: successfully parsed IRIG values for `Time`, `StartTime`, `StopTime`, and `EndTime`.
  Unparseable values remain in `fields`; the row is kept.

Pass a table name, not a SQL statement or a filename. This is a read-only table lookup,
not a general SQL API. Typed table reads are cached on the archive object; `read_table()`
queries IADS each time. Reopen the archive to refresh cached metadata after external changes.

### Where the metadata comes from: `pfconfig`, `iadsarchiveinfo.txt`, and IADS

Pass the **archive path** to `Archive`, not paths to `pfconfig` or
`iadsarchiveinfo.txt`. Metadata and samples come from the installed IADS component,
which resolves those files. Returned values reflect IADS's interpretation, not the
raw sidecar contents; this package supplies no separate file-precedence rules.

| Python view | How it is obtained |
| --- | --- |
| `info()` | IADS archive-info response: `StartTime` → `start_time`, `StopTime` → `end_time`, `Date` → `flight_date`, plus `DataDir`, `Flight`, `Test`, and `Tail` |
| `raw_channel_names()` | IADS parameter-name enumeration, including names that may not be extractable |
| `channels()` / `channel_names()` | Parameter metadata from `ParameterDefaults` plus per-parameter queries, then extraction eligibility and update-rate resolution |
| Typed log/config methods | IADS table queries, mapped to named Python properties |
| `read_table(name)` | IADS `select * from <table>` query, retained as generic records |
| Uploaded channel samples | IADS sample reads for the selected channels and IRIG windows |

`channels()` is an **extractable-channel view**, not every row from the configuration.
Channels without recorded time bounds or a resolvable sample rate are excluded;
rate-zero derived channels can inherit a rate from their dependencies. Compare
`raw_channel_names()` with `channel_names()` when investigating missing channels.
Use `read_table("ParameterDefaults")` to inspect the raw configuration rows exposed by IADS.

`info().flight_date` comes from the returned `Date` field (parsed as month/day/two-digit
year). `start_time` and `end_time` retain the returned IRIG coordinates. Reading metadata
does not establish that these values describe real UTC time—see the read-window section.

## Troubleshooting

Call `enable_logging("info")` before starting work for progress and summaries on `stderr`,
or use `"debug"` for diagnostics. The first successful configuration applies to all
workers; invalid filters raise `ValueError`.

### COM troubleshooting

Errors retain the COM operation, hexadecimal HRESULT, and original Windows message:

- **`CLSIDFromProgID` / `0x800401F3` ("Invalid class string")** means Windows could not resolve
  a valid class identifier for `IadsDataInterfaces.iadsread`; registration may be missing or invalid.
- **`CoCreateInstance` / `0x80040154` ("Class not registered")** indicates missing or invalid
  in-process registration, possibly for the wrong architecture. For either registration error,
  register the installed 64-bit DLL using the elevated 64-bit PowerShell command above and repeat
  the activation check.
- **Other `CoCreateInstance` errors** mean class lookup succeeded but activation failed. Use the
  original Windows error to check DLL dependencies, 64-bit compatibility, and file/access
  permissions. Re-registering is not a universal fix.
- **`IDispatch::Invoke` errors** occur after activation, during an IADS method call. Preserve the
  operation, HRESULT, original message, and query context when reporting the problem; do not
  assume that registration is the cause.

| Symptom | Check |
| --- | --- |
| No matching wheel / installation fails | Windows x64 and 64-bit Python 3.11+ |
| `Invalid class string` / `Class not registered` | See [COM troubleshooting](#com-troubleshooting) and the DLL registration steps above |
| `FileNotFoundError` opening an archive | The archive path exists and is accessible |
| `RuntimeError` about an archive not entered | Keep calls inside `with Archive(...)` |
| Missing bundled worker | Reinstall the wheel to restore the bundled extraction worker |
| Authentication / workspace error | Use the API base URL and a profile/token with access to the destination workspace |
| Flight-date / IRIG validation error | Check archive dates and the selected time range; supply a known UTC anchor when appropriate |
| Zero uploaded files | Check channel selection, the time window, and `stats.extraction_failures` |
| Uploads finished but data is not visible | Check server-side ingest status in Nominal |

Package-specific errors derive from `nominal.iads.NominalIadsError`, including `ArchiveError`,
`ConfigError`, `WorkerError`, and `ExtractionError`. Upload/client errors may use the base
class directly. Missing paths and invalid context-manager use raise Python's built-in
`FileNotFoundError` and `RuntimeError` respectively. Other filesystem and argument failures
can raise built-in exceptions such as `PermissionError`, `FileExistsError`, `OSError`, or `ValueError`.

## Links

- [Source and issue tracker](https://github.com/nominal-io/nominal-iads)
- [Nominal Python SDK](https://nominal-io.github.io/nominal-client/)
- [License: Apache-2.0](https://github.com/nominal-io/nominal-iads/blob/main/LICENSE)

