Metadata-Version: 2.5
Name: ipaapi
Version: 1.0.0
Summary: Command-line client for QIAGEN Ingenuity Pathway Analysis: map arbitrary column layouts, submit datasets in bulk, and resume cleanly when the analysis allowance runs out.
Author-email: Ken Jones <Ken.Jones@bioinformaticsolutions.com>
License: MIT License
        
        Copyright (c) 2026 Ken Jones
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: bioinformatics,ingenuity,ipa,pathway,qiagen
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: >=3.9
Requires-Dist: pandas>=1.5
Requires-Dist: requests-oauthlib>=1.3
Requires-Dist: requests>=2.28
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: progress
Requires-Dist: tqdm>=4.64; extra == 'progress'
Description-Content-Type: text/markdown

# ipaapi

A Python package and command-line tool for QIAGEN Ingenuity Pathway Analysis
(IPA). Upload datasets into an IPA project using an explicit column mapping,
submit them for analysis, and track the results — one file or several hundred.

Free software (MIT). Built on QIAGEN's `python-api-demo` example code — **not
an official QIAGEN product**, and not endorsed by QIAGEN.

```bash
ipaapi submit ~/data --ID 1:hugo --FC 4:logratio --skip-rows 1 \
    --reference-set ipkb --project MyStudy --pattern _DEG
```

---

## Contents

- [Why this exists](#why-this-exists)
- [Installation](#installation)
- [Quick start](#quick-start)
- [How the mapping works](#how-the-mapping-works)
- [Command-line reference](#command-line-reference)
- [Recipes](#recipes)
- [Working with IPA](#working-with-ipa) — the undocumented parts
- [Authentication](#authentication)
- [Python API](#python-api)
- [Troubleshooting](#troubleshooting)
- [How a submission is encoded](#how-a-submission-is-encoded)
- [Development](#development)
- [Contributing](#contributing)
- [Licence](#licence)

---

## Why this exists

QIAGEN's demo script works, but assumes a rigid file layout: the gene ID in
column 0, then `n_observations × n_measurements` value columns in strict
repeating order, every observation carrying the same measurement types in the
same positions. Real analysis output rarely looks like that.

This package replaces that assumption with a declaration. You name the
identifier column and describe each observation as a set of
`(column, measurement type)` pairs. Columns may be in any order, named
anything, and interleaved with columns the analysis should ignore.

It also fixes a number of things the demo got wrong or left out — see
[Differences from the demo](#differences-from-the-demo).

---

## Installation

```bash
git clone <this-repo> ipaapi && cd ipaapi
pip install -e .
```

Or build and install a wheel:

```bash
python3 -m pip wheel . --no-deps -w dist
python3 -m pip install dist/ipaapi-*.whl
```

Requires Python 3.9+, `requests`, `requests-oauthlib`, `pandas`.

Confirm what you're running — this reports the version, the install location,
and whether it's an editable checkout rather than a built wheel:

```bash
$ ipaapi --version
ipaapi 1.0.0
installed at /usr/lib/python3.11/site-packages/ipaapi
python 3.11.5 (/usr/bin/python3)
```

---

## Quick start

Say your file looks like this — a comment line, then a header, then data:

```
# generated by pipeline v3
Gene,Common_name,Control_mean,Treatment_mean,Fold_change,P-value,Q-value
ENSG00000229807,XIST,4.21,2.88,-1.33,0.001,0.02
```

Column positions are **0-based** and counted from the *header* row:

```
0 Gene   1 Common_name   2 Control_mean   3 Treatment_mean   4 Fold_change   5 P-value   6 Q-value
```

Check the mapping without contacting IPA:

```bash
ipaapi validate results.csv --ID 1:hugo --FC 4:logratio --skip-rows 1
```

```
results: 2,338 rows
gene id: 'Common_name' (hugo)
observations: 1
  results:
    'Fold_change' -> Log Ratio

     Common_name  Fold_change
0           XIST        -1.33
...
1 file valid. Nothing was uploaded.
```

When that looks right, submit:

```bash
ipaapi submit results.csv --ID 1:hugo --FC 4:logratio --skip-rows 1 \
    --reference-set ipkb --project MyStudy
```

```
submitted results: 43595871

Submitted 1 analysis.
Analyses are running in IPA. Check on them with:
  ipaapi status 43595871
  ipaapi report 43595871
Recorded in ~/.local/state/ipaapi/submissions.tsv -- see 'ipaapi history'.
```

---

## How the mapping works

Three ideas, and they mirror how IPA thinks about a dataset.

**Measurement** — one value column: which column, what kind of number it holds,
and an optional cutoff.

**Observation** — a named sample or contrast, and the measurement columns
belonging to it. One analysis is created per observation.

**ColumnMapping** — the identifier column, its type, and the observations.

```python
ColumnMapping(
    gene_id_column="Common_name",
    gene_id_type="hugo",
    observations=[
        Observation("drug A vs ctrl", [
            Measurement("A_log2fc", MeasurementType.LOG_RATIO),
            Measurement("A_padj",   MeasurementType.FALSE_DISCOVERY, cutoff=0.05),
        ]),
        Observation("drug B vs ctrl", [
            # declared in a different order on purpose -- this is fine
            Measurement("B_padj",   MeasurementType.FALSE_DISCOVERY, cutoff=0.05),
            Measurement("B_log2fc", MeasurementType.LOG_RATIO),
        ]),
    ],
)
```

**One constraint is imposed by IPA, not by this package.** The wire format
declares `expvaltype`, `expvaltype2`, … and `cutoff`, `cutoff2`, … *once for the
whole submission*, then supplies per-observation column names against those
slots. So every observation must contribute exactly one column per measurement
type, and a given type carries one cutoff throughout. Both are checked before
anything is uploaded, with an error that explains why.

Within those limits, order and naming are free — observations declared in
different column orders are normalised automatically.

Everything is validated against the actual data before upload: columns exist,
none is claimed twice, and values fall in the range IPA expects for their type.
That last check matters more than it looks — see
[measurement types](#measurement-types).

---

## Command-line reference

```
ipaapi validate   check a mapping against file(s) without uploading
ipaapi submit     upload into a project and start analyses
ipaapi status     check the state of existing analyses
ipaapi report     print IPA Interpret links
ipaapi history    list analyses submitted through this tool
```

### Mapping arguments

Used by `validate` and `submit`.

| Flag | Form | Meaning |
| --- | --- | --- |
| `PATH` | positional | a data file, or a directory to search |
| `--ID` | `COLUMN:TYPE` | 0-based identifier column and its IPA gene ID type. May be given twice — see [two identifier columns](#two-identifier-columns) |
| `--FC` | `COLUMN:TYPE[:CUTOFF]` | 0-based value column, [measurement type](#measurement-types), optional cutoff |
| `--skip-rows` | `N` | discard N lines above the header row |
| `--sep` | `CHAR` | field delimiter (sniffed from the header line by default) |
| `--pattern` | `TEXT` | when PATH is a directory: substring or glob selecting files |
| `--recursive` | flag | search subdirectories too |
| `--observation` | `NAME` | observation name in IPA (default: the filename). Single file only |
| `--no-range-check` | flag | skip the value-range validation |
| `--list-id-types` | flag | print all 33 gene ID types and exit |

### `submit`

| Flag | Default | Meaning |
| --- | --- | --- |
| `--project` | *required* | destination IPA project. **Created if it doesn't exist**, so a typo silently makes a new one |
| `--reference-set` | `omit` | `ipkb`, `dataset`, or `omit`. See [the reference set](#the-reference-set) |
| `--wait` | off | poll until analyses finish and print report links |
| `--interval` / `--timeout` | 30s / 3600s | polling, only with `--wait` |
| `--dry-run` | off | validate and stop before login |
| `--analysis-name` / `--dataset-name` | filename | single file only |
| `--log-file` | `~/.local/state/ipaapi/submissions.tsv` | submission log |

### Authentication arguments

Used by every command that contacts IPA.

| Flag | Meaning |
| --- | --- |
| `--no-cache` | ignore any cached token |
| `--token-file` | token cache path (default `~/.cache/ipaapi/token.json`) |
| `--application-name` | `applicationname` IPA scopes the session to (default `PythonAPI`) |
| `--browser` | browser to launch for login, e.g. `firefox` |

### `history`

| Flag | Meaning |
| --- | --- |
| `--project` / `--since` / `--limit` | filters |
| `--status` | look up each analysis's current state (requires login) |
| `--log-file` | read a different log |

### Environment variables

| Variable | Purpose |
| --- | --- |
| `IPAAPI_TOKEN_FILE` | token cache location — set this if `$HOME` isn't writable |
| `IPAAPI_LOG_FILE` | submission log location |

---

## Recipes

### Many files, one analysis each

```bash
ipaapi submit ~/data --pattern _DEG --ID 1:hugo --FC 4:logratio \
    --skip-rows 1 --reference-set ipkb --project Study1
```

`--pattern` takes plain text or a glob. Text with no `*`, `?` or `[` matches as
a **substring**, so `--pattern SampleA` finds `SampleA_DEG.txt` and
`SampleA_raw.tsv`. With no `--pattern`, `*.txt`/`*.tsv`/`*.csv` are searched.
Hidden files are skipped and results sorted, so run order is predictable.

Every matched file must fit the same `--ID`/`--FC` positions.

### Files are filed as they're processed

When `PATH` is a directory, each file moves as its outcome becomes known:

| Outcome | Destination |
| --- | --- |
| IPA accepted it | `submitted/` |
| The file is at fault | `failed/`, with a `.error.txt` note beside it |
| Allowance exhausted, or IPA declined | left in place for the next run |

```
submitted SampleA_DEG: 43595001
submitted SampleB_DEG: 43595002

Allowance exhausted while submitting SampleC_DEG:
REJECTED: the analysis allowance appears to be exhausted.
IPA said: 'Unable to run analysis: Analysis limit exceeded'

2 file(s) moved to submitted/
2 file(s) left in place for the next run
Re-run the same command later; the files left in place are exactly the ones
still to do.
```

The source directory shrinks to exactly the work outstanding, and re-running the
identical command resumes. `submitted/` and `failed/` are excluded from
discovery, so a run can't re-ingest its own output.

Nothing is moved when the *command* is at fault — a bad `--ID` type or a mapping
that fails every file leaves the directory untouched, because that's a mistake
to fix rather than data to quarantine. Single-file submits are never moved.

### Draining a backlog against a daily allowance

Because a stopped run resumes cleanly, this is safe to leave unattended:

```cron
0 6 * * * cd ~/data && ipaapi submit ./ --pattern _DEG --ID 1:hugo \
    --FC 4:logratio --skip-rows 1 --reference-set ipkb --project Study1 \
    >> ~/ipaapi-cron.log 2>&1
```

It submits until the allowance runs out, files what succeeded, leaves the rest.
Check the log after the first few runs — a cron job whose *refresh* token has
expired fails into that file rather than prompting anyone.

### Finding analysis IDs later

IPA's API cannot list the analyses on an account, so the package keeps its own
log — every submission appends a timestamped row.

```bash
ipaapi history
ipaapi history --project Study1 --since 2026-08-01
ipaapi history --status
```

```
2026-08-05T08:35:53-06:00  43595039  Study1  SampleA_DEG
2026-08-05T08:35:53-06:00  43595041  Study1  SampleB_DEG

2 submission(s). Report links: ipaapi report 43595039 43595041
```

Plain TSV — grep it, open it in a spreadsheet. It only covers submissions made
through this tool; anything submitted from the IPA client won't appear.

### Comment lines above the header

```
# generated by pipeline v3, 2026-08-05
EnsemblID	log2FC	pval
```

`--skip-rows 1` discards the preamble. Column numbers count from the header, so
they don't change when you add it.

Skipping also fixes delimiter detection: the delimiter is sniffed from the
header line, and a comment line is a bad thing to sniff — the one above has
commas but no tabs, so without `--skip-rows` the file would be read as CSV and
collapse into nonsense. Rather than let that through, a header that looks like a
comment is rejected with a message pointing at this flag.

### Two identifier columns

`--ID` may be given twice. The first is the primary; the second fills rows where
the primary is blank (`.`, `NA`, empty, and similar are all treated as missing).

```bash
ipaapi submit data.csv --ID 0:ensembl --ID 1:hugo --FC 4:logratio --project S1
```

> **Read this before relying on it.** IPA accepts one `geneidtype` per
> submission. Rows filled from the second column are still uploaded under the
> *primary's* type, so they may fail to map. The fill count is always reported:
>
> ```
> Warning: 344 of 2,338 rows took their identifier from the fallback column
> 'Common_name' (hugo). IPA is told a single gene ID type for the submission --
> 'ensembl' -- so those rows are uploaded under that declaration and may not map.
> ```
>
> If a large fraction is being filled, using the fallback column as the *only*
> identifier is usually better than mixing.

---

## Working with IPA

Most of this is either undocumented or documented somewhere hard to find. It's
recorded here because getting it wrong is expensive — analyses consume a
metered allowance.

### Gene ID types

`--ID COLUMN:TYPE` takes any value from IPA's `geneidtype` list (Integration
Module §3.1). `ipaapi submit --list-id-types` prints all 33.

Common ones: `ensembl`, `hugo`, `entrezgene`, `refseq`, `swissprot`,
`affymetrix`, `illumina`, `agilent`.

Two things are not guessable:

- **Human gene symbols are `hugo`.** Not `genesymbol`, not `hgnc`, and not the
  desktop client's own label `Gene Symbol` — all three are rejected outright.
- **Species rides on the identifier type.** There is no species parameter:
  `hugo` human, `mousesymeg` mouse, `ratsymeg` rat.

A type outside the documented list produces a warning with a near-match
suggestion but is still sent, since IPA is the authority and the list will age.
An unrecognised value fails before anything is uploaded, and IPA names it.

### The reference set

The background enrichment is scored against — the denominator of the Fisher's
exact test behind every p-value.

| Value | Background |
| --- | --- |
| `ipkb` | Ingenuity Knowledge Base (Genes Only, or + Endogenous Chemicals if chemicals are present) |
| `dataset` | the genes you uploaded |
| `omit` (default) | IPA chooses |

Which to use depends on **what you uploaded**:

- Uploading a **complete measured transcriptome** with a cutoff? `dataset` is
  the better science — the background is what your assay could actually detect,
  which controls for detection bias.
- Uploading a **pre-filtered hit list**? `dataset` makes the background nearly
  identical to the foreground. Use `ipkb`.

§4.1.3.1 states that with the parameter omitted IPA picks by size — `ipkb` below
2000 identifiers, `dataset` at 2000 or more. **In practice this has not been
observed to hold**: files of 1,804–6,245 rows all came back as
`Ingenuity Knowledge Base (Genes Only)`. Since the behaviour is unpredictable,
set it explicitly for anything you intend to compare against itself.

Verify after the fact — the setting is recorded in every IPA export:

```bash
grep -h "^Reference set" *_IPA_output.txt | sort | uniq -c
```

Array platforms can also be named as reference sets, paired with a
`referencesettype`. Not exposed here; see §4.1.3.

### Measurement types

| Value | Meaning | Valid range |
| --- | --- | --- |
| `ratio` | Ratio | `[0, +∞)` |
| `foldchange` | Fold Change | `(-∞, -1]` and `[1, +∞)` |
| `logratio` | Log Ratio | `(-∞, +∞)` |
| `pvalue` | p-value | `[0, 1]` |
| `falsediscovery` | FDR / q-value | `[0, 100]` |
| `intensity` | Intensity | `[0, +∞)` |
| `other` | Other (normalised around zero) | `(-∞, +∞)` |
| `gain_loss` | Variant Gain/Loss | `-2, -1, 0, 1, 2` |
| `classification` | Variant ACMG Classification | `-2, -1, 0, 1, 2` |

> **Out-of-range values are silently discarded by IPA.** §3.1: "analysis will
> still proceed without errors or warning diagnostics" — offending entries are
> simply dropped. This is why the range check exists and why it refuses rather
> than warns. Declaring log2 fold changes as `foldchange`, for instance, would
> quietly discard every gene between −1 and 1, which in a typical scRNA-seq
> table is most of them.

The package helps in both directions:

- Values declared `foldchange` that cluster inside (−1, 1) → suggests `logratio`.
- A column declared `logratio` with *no* values in (−1, 1) → warns that it looks
  like signed fold change, since a real log ratio is centred on zero.

A column called `Fold_change` may hold either. Check the data, not the name.

### What the API cannot do

- **List your projects.** `--project` creates one if the name doesn't exist, so
  a typo silently makes a new project rather than erroring.
- **List your analyses.** Every endpoint needs an ID you already hold — hence
  the local submission log.
- **Tell you your remaining allowance.** You discover the limit by hitting it.

### Errors IPA actually returns

IPA answers a rejected submission with an **HTML error page**, not plain text.
The reason is at the *end*, after support boilerplate. This package strips the
boilerplate and the page footer, and classifies what's left:

| IPA's message | Class | What the tool does |
| --- | --- | --- |
| `Unknown GeneId Type (X)` | `MalformedRequestError` | stops; names the flag; moves nothing |
| `Unable to run analysis: Analysis limit exceeded` | `QuotaExceededError` | stops; leaves remaining files for the next run |
| `Unable to run analysis: …` (other) | `AnalysisRefusedError` | as above — reached the analysis logic, so not a parameter fault |
| anything else | `SubmissionError` | files that one under `failed/` |

Quota matching is deliberately broad (`ipaapi.client.QUOTA_PATTERNS` plus HTTP
429): a false positive only leaves a file for the next run, while a false
negative would quarantine a retryable submission. The raw response is always
printed, so a misclassification is visible.

### Interpret links

`ipaapi report <id>` fetches the IPA Interpret URL for a finished analysis. It
checks status first, so an unfinished analysis says so rather than surfacing a
bare HTTP 500.

**These have been observed to return HTTP 500 even for succeeded analyses.**
The cause is unconfirmed — possibly the commercial add-on licence, possibly a
stale endpoint path inherited from the demo. `examples/probe_interpret.py`
prints the raw response for diagnosis. Analyses open fine in IPA itself.

---

## Authentication

Browser-based OAuth 2.0 with PKCE. Your password never reaches this package.

1. A short-lived HTTP server binds `127.0.0.1:8000`.
2. Your browser opens QIAGEN's authorization page; you log in there.
3. QIAGEN redirects back to `localhost:8000` with a one-time code. The `state`
   parameter is verified, then the code plus the PKCE verifier is exchanged for
   a token.
4. The token is used as `Authorization: Bearer …` and the server shuts down.

Whichever account you log in as owns the datasets and projects.

The client ID is the public one any IPA user may use — it is not a secret.

### Token caching and refresh

Tokens are cached at `~/.cache/ipaapi/token.json`, owner-only (0600). **Access
tokens are short-lived**, but a refresh token comes with them and is spent
automatically: an expired cache is renewed over HTTP with no browser and no
prompt. A browser login is only needed when the refresh token itself is
rejected.

Deleting the cache is effectively logging out. `--no-cache` forces a fresh
login. Be aware the token is plaintext on disk — anyone who can read your home
directory can use it until it expires.

### Headless servers

Because refresh is automatic, a token copied from a machine with a browser keeps
renewing itself indefinitely:

```bash
# once, on a machine with a browser
ipaapi submit ... # or any command that logs in

scp ~/.cache/ipaapi/token.json server:~/.cache/ipaapi/token.json
ssh server chmod 600 ~/.cache/ipaapi/token.json
```

**If `$HOME` isn't writable**, the cache can't be saved and every run needs a
fresh login — crippling on a headless box. Point it somewhere writable:

```bash
export IPAAPI_TOKEN_FILE=$HOME/ipaapi-token.json
export IPAAPI_LOG_FILE=$HOME/ipaapi-submissions.tsv
```

Both failures are reported loudly rather than swallowed, because a cache that
never writes looks exactly like a token that expires instantly.

When an interactive login is genuinely needed, X forwarding is the cleanest
route — the server-side browser renders locally *and* `localhost:8000` resolves
server-side where the callback listens, so no port forwarding is required:

```bash
ssh -X you@server        # ssh -Y from macOS, with XQuartz running
```

Failing that, forward the callback port and use your own browser:

```bash
ssh -L 8000:localhost:8000 you@server
```

The error message distinguishes `DISPLAY` unset from no browser found.

> The redirect URI is pinned to `http://localhost:8000` by the OAuth client
> registration, so the port is not configurable in practice.

### Using a token obtained elsewhere

```python
import os
from ipaapi import Credentials, IPAClient

client = IPAClient(Credentials.from_token(os.environ["IPA_TOKEN"]))
```

---

## Python API

```python
from ipaapi import (
    ColumnMapping, Dataset, IPAClient, Measurement, MeasurementType,
    Observation, ReferenceSet, TokenCache,
)

mapping = ColumnMapping(
    gene_id_column="Common_name",
    gene_id_type="hugo",
    observations=[
        Observation("HIV vs NEG", [
            Measurement("Fold_change", MeasurementType.LOG_RATIO),
        ]),
    ],
)

dataset = Dataset.from_file("results.csv", mapping, skip_rows=1)
print(dataset.describe())          # confirm before uploading

client = IPAClient.login(cache=TokenCache())
ids = client.submit(dataset, project="MyStudy", reference_set=ReferenceSet.IPKB)

for analysis_id, status in client.wait_for(ids).items():
    if status.succeeded:
        print(client.report_url(analysis_id))
```

Key objects:

| Object | Purpose |
| --- | --- |
| `ColumnMapping`, `Observation`, `Measurement` | describe the file |
| `Dataset.from_file` / `.from_frame` | load and validate |
| `IPAClient.login()` | OAuth, with caching and refresh |
| `.submit()` `.status()` `.wait_for()` `.results()` `.report_url()` | the API |
| `GENE_ID_TYPES` | all 33 identifier types and what they mean |
| `ipaapi.history` | the submission log |
| `ipaapi.errors` | everything derives from `IPAError` |

### Results

```python
results = client.results(analysis_id)
print(results.canonical_pathways.head())
cp, ur, df = results                  # unpacks like the demo's ipa_results()
```

> Programmatic result retrieval is a **commercial IPA add-on**. Without it these
> calls raise `ResultsUnavailableError`. Submission, status polling and report
> links are unaffected.

---

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| `REJECTED: IPA does not recognise the gene ID type 'X'` | not in IPA's vocabulary | `--list-id-types`; human symbols are `hugo` |
| `declared 'foldchange' but holds N out-of-range value(s)` | log2 values declared as linear fold change | `--FC N:logratio` |
| `Could not find a header row … looks like a comment` | preamble above the header | `--skip-rows N` |
| `--FC refers to column N, but the file has only M column(s)` | 1-based counting, or wrong `--skip-rows` | positions are 0-based, from the header |
| `Every row is missing an identifier` | wrong column, or no header | check with `head -1 file \| tr '\t' '\n' \| nl -v0` |
| `the analysis allowance appears to be exhausted` | daily/period limit | re-run later; files left in place resume |
| Login prompt on every run | token cache not writable | `export IPAAPI_TOKEN_FILE=...`; check for a root-owned cache |
| `Could not open a browser automatically` | headless | `ssh -X`, or copy a token across |
| `report` returns HTTP 500 on a succeeded analysis | unconfirmed; possibly add-on licence | open the analysis in IPA; see `examples/probe_interpret.py` |
| Analyses have z-scores but no p-values | reference set equals the gene list | `--reference-set ipkb` |
| Half of all pathways significant | list too large for the background | apply a cutoff, or upload unfiltered data with a cutoff |

Useful first move for any column problem:

```bash
head -1 yourfile.csv | tr ',\t' '\n' | nl -v0
```

---

## How a submission is encoded

Worth knowing when debugging. `--ID 1:hugo` becomes three separate things:

| From `--ID` | Wire parameter | Sent |
| --- | --- | --- |
| the type | `geneidtype=hugo` | once |
| the column, resolved from position to header name | `genecolname=Common_name` | once |
| that column's values | `geneid=XIST`, `geneid=UTY`, … | once per row |

The column *number* never leaves your machine.

The whole dataset travels in one `application/x-www-form-urlencoded` POST to
`/pa/api/v2/multiobsanalysis`, which both creates the dataset in the project and
starts one analysis per observation. Parameter naming is positional and
irregular — for measurement slot *k* and observation *i*, both zero-based:

| Parameter | Meaning |
| --- | --- |
| `expvaltype`, `expvaltypeK+1` | measurement type for slot *k* (global) |
| `cutoff`, `cutoffK+1` | cutoff for slot *k* (global, optional) |
| `obsI+1name` | observation name |
| `expvalname`, `expvalK+1name` | column label, first observation |
| `obsI+1expvalname`, `obsI+1expvalK+1name` | column label, later observations |
| `geneid` | one per data row |
| `expvalue`, `expvalK+1` | one per slot per observation, per row |

Per-row value parameters carry no observation prefix — they cycle through the
slots of observation 1, then observation 2, and so on. Order is load-bearing.

The body is properly percent-encoded. The demo concatenated it by hand, so any
value containing a space, `&`, `=`, `+` or `%` corrupted the request — including
the `Group Max Intensity` column in the demo's own sample dataset.

---

## Development

```
src/ipaapi/
  __init__.py    public API and the version (single source of truth)
  models.py      MeasurementType, AnalysisStatus, ReferenceSet, GENE_ID_TYPES
  mapping.py     Measurement, Observation, ColumnMapping
  dataset.py     Dataset, load_table
  _payload.py    multiobsanalysis body construction
  auth.py        OAuth 2.0 + PKCE, Credentials, TokenCache, refresh
  client.py      IPAClient, error classification
  history.py     the submission log
  triage.py      submitted/ and failed/ filing
  cli.py         the ipaapi console script
  errors.py      exception hierarchy
tests/           offline; no network required
examples/        runnable scripts and diagnostics
```

```bash
pip install -e ".[dev]"
pytest
```

The suite is fully offline — mapping validation, the exact parameter layout of
the submission body, encoding of hostile characters, error classification,
triage behaviour, token cache and refresh logic.

**Versioning.** The version lives only in `src/ipaapi/__init__.py`;
`pyproject.toml` reads it at build time. Bump it there and nowhere else, and add
a `CHANGELOG.md` entry. `ipaapi --version` reports the install path too, which
is what actually answers "am I running the wheel I think I am".

### Differences from the demo

- Column mapping by name in any order, validated before upload.
- Request bodies are percent-encoded.
- OAuth: no CPU-spinning wait loop, `state` is verified, logins time out, the
  callback server is shut down, error redirects are handled, tokens are cached
  and refreshed.
- Submissions are never retried automatically — a retried POST could create a
  duplicate analysis. GETs retry with backoff.
- Typed exceptions; access tokens excluded from `repr()`.
- No `install_dependencies()` shelling out to `pip3`.

---

## Contributing

Issues and pull requests are welcome. The most useful contributions are
**corrections to the [Working with IPA](#working-with-ipa) section** — much of
it was established by trial against a live account, and a few points have
already had to be corrected more than once. If IPA behaves differently for you,
that is worth reporting even without a code change.

```bash
pip install -e ".[dev]"
pytest
```

Tests are fully offline; none of them contact IPA.

---

## Status

**1.0** — stable and in production use against live IPA. The command line and
the Python API are settled; breaking changes from here mean a major version
bump. See `CHANGELOG.md`.

Known open questions, none of which affect submission:

- Interpret links (`ipaapi report`) have returned HTTP 500 for analyses that
  succeeded. Cause unconfirmed; possibly the commercial add-on licence.
- Programmatic result retrieval (`client.results()`) requires that same add-on
  and is largely untested here.
- The documented reference-set size rule does not match observed behaviour;
  set `--reference-set` explicitly.

---

## Licence

MIT — see [LICENSE](LICENSE). Free to use, modify and redistribute.

Not affiliated with, endorsed by, or supported by QIAGEN. IPA is QIAGEN's
product; this is an independent client for its public API, built on the
`python-api-demo` example code QIAGEN publishes. For questions about the API
itself, QIAGEN's contact is `AdvancedGenomicsSupport@qiagen.com` — please don't
send them bug reports about this package.
