Metadata-Version: 2.4
Name: python-fedci
Version: 0.2.1
Summary: A small package for federated independence tests
Author-email: Maximilian Hahn <max.hahn@gmx.de>
License-Expression: AGPL-3.0-or-later
Project-URL: Homepage, https://github.com/maxhahn/fedci
Project-URL: Repository, https://github.com/maxhahn/fedci
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: matplotlib>=3.11.0
Requires-Dist: numpy>=2.4.6
Requires-Dist: polars>=1.41.2
Requires-Dist: scipy>=1.17.1
Requires-Dist: tqdm>=4.68.3
Requires-Dist: pyzmq>=27.1.0
Provides-Extra: dev
Requires-Dist: pytest>=9.1.0; extra == "dev"
Dynamic: license-file

# fedci

Federated conditional independence (CI) testing via likelihood ratio tests. Data never leaves each client — only aggregated sufficient statistics are exchanged with the central server.

---

## Installation

```bash
uv add python-fedci        # or: pip install python-fedci
```

Requires Python 3.10+. ZMQ networking requires `pyzmq`.

---

## Core concepts

| Class | Role |
|---|---|
| `Client` | Holds a local dataset; answers statistical queries from the server |
| `Server` | Orchestrates tests across all clients; never sees raw data |
| `TestResult` | Result of one CI test (p-value, Bayes factors) |
| `RepeatedTestResult` | Aggregated statistics over many bootstrap runs |

---

## Basic workflow

```python
import polars as pl
from fedci import Client, Server

# Each site wraps its own data
df1 = pl.read_parquet("site1.parquet")
df2 = pl.read_parquet("site2.parquet")

client1 = Client("site1", df1)
client2 = Client("site2", df2)

server = Server([client1, client2])
```

### Test a single conditional independence

```python
# Is X independent of Y given {Z, W}?
result = server.test("X", "Y", {"Z", "W"})

result.p_value          # float
result.log10_bic_bf     # log10 Bayes factor (BIC approximation)
result.log10_bff        # log10 Bayes factor (BFF method)
result.pH0_bic          # P(H0) = 1 / (1 + BF)  via BIC
result.pH0_bff          # P(H0) = 1 / (1 + BF)  via BFF
result.n_samples        # total observations used
```

### Run all pairwise tests

```python
# All pairs up to conditioning set size 2, 4 parallel workers
results = server.run(max_cond_size=2, workers=4, progress_bar=True)

# results is Dict[(x, y, frozenset(s)), TestResult]
for (x, y, s), r in results.items():
    print(f"{x} ⊥ {y} | {s}  p={r.p_value:.4f}  log10BF={r.log10_bic_bf:.2f}")
```

`workers` parallelises at the test level — each CI test runs in its own thread. Numpy releases the GIL during computation so multiple cores are used effectively.

### Repeated tests with subsampling

Bootstrap stability analysis: run the same test many times on random subsamples to assess reliability.

```python
rtr = server.test_repeatedly(
    "X", "Y", {"Z"},
    num_runs=100,
    sample_fraction=0.80,
    workers=4,           # parallel runs (local clients only; see Networking)
    progress_bar=True,
)

rtr.n_runs              # 100
rtr.mean_p              # mean p-value across runs
rtr.mean_log10_bic_bf   # mean log10 BF
rtr.std_log10_bic_bf    # spread — lower means more stable
rtr.rate_dependence_bic # fraction of runs where BF > 1 (evidence for dependence)
rtr.mean_pH0_bic        # mean P(H0) across runs
```

Run the full test suite repeatedly:

```python
repeated = server.run_repeatedly(
    num_runs=50,
    sample_fraction=0.80,
    max_cond_size=2,
    workers=4,           # parallel tests within each run
    progress_bar=True,
)
# Dict[(x, y, frozenset(s)), RepeatedTestResult]
```

---

## Interpreting results

Two Bayes factor estimates are available alongside the p-value.

### BIC Bayes factor

```
log10(BF) ≈ ½ (2·ΔLL − Δk·log n) / log(10)
```

Closed-form, conservative, low variance across subsamples.

### BFF — Bayes Factor Function

Maximises a chi² Bayes factor over a prior width grid.

### Decision guide

| log10(BF) | Interpretation |
|---|---|
| > 1.0 | Strong evidence for dependence |
| 0 to 1.0 | Weak / inconclusive |
| < 0 | Evidence for independence |

`pH0 = 1 / (1 + BF)` converts a Bayes factor directly to a probability of the null.

For stability assessment across repeated runs, prefer `rate_dependence_bff` (fraction of runs where BF > 1) and `std_log10_bff_bf` (spread on log scale) over a single p-value threshold or its `_bic` counterparts.

---

## Model configuration

### Default — GLM

Logistic regression for binary/categorical responses; linear regression for continuous. No extra configuration required.

```python
server = Server([client1, client2])
```

### GAM — nonlinear continuous relationships

Cubic B-spline basis expansion for each continuous predictor. Custom knot placement is supported, though preferably, data is normalised to **[0, 1]** before being passed to clients so that a fixed knot grid is valid across all sites.

```python
import numpy as np
from fedci import Server, Client, GAMConfiguration, ModelType

# Knot arithmetic:
#   num_knots=8, num_degrees=3  →  n_basis = num_knots + num_degrees - 2 = 9
#   knot vector length = num_knots + 2*num_degrees - 1 = 13  (5 interior knots)
N_KNOTS  = 8
DEGREE   = 3
interior = list(np.linspace(0.0, 1.0, N_KNOTS - DEGREE + 2)[1:-1])
knot_vec = [0.0] * (DEGREE + 1) + interior + [1.0] * (DEGREE + 1)

# Normalise to [0, 1] on the combined dataset before splitting into clients
mins = df_full.min()
maxs = df_full.max()
df_norm = df_full.with_columns([
    ((pl.col(c) - mins[c][0]) / (maxs[c][0] - mins[c][0])).alias(c)
    for c in df_full.columns
])

gam_config = GAMConfiguration(
    type=ModelType.GAM,
    num_knots=N_KNOTS,
    num_degrees=DEGREE,
    knots={v: knot_vec for v in df_norm.columns},  # include all variables
)

server = Server([client1, client2], model_configuration=gam_config)
```

Including a variable in `knots` is safe even when it appears as the response in some tests — the spline basis is only applied to predictors, never to the response.

### Site heterogeneity

When datasets across sites differ systematically (e.g. batch effects or recruitment differences), enable random effects:

```python
from fedci import HeterogenietyType

# Random intercepts only (recommended — far fewer parameters)
server = Server(
    [client1, client2],
    heterogeniety=HeterogenietyType.GLOBAL,
    random_intercept_only=True,              # one shift per site per model
    local_ridge_coefficient=1.0,             # starting prior strength; adapted via EM
)

# Full random effects on all coefficients (intercept + slopes)
server = Server(
    [client1, client2],
    heterogeniety=HeterogenietyType.GLOBAL,
    random_intercept_only=False,             # site-specific shift on every coefficient
    local_ridge_coefficient=1.0,
)
```

`random_intercept_only=False` is heavily overparameterised for most datasets — a warning is raised when used with GAM, where the spline basis already has many coefficients. Prefer `True` unless you have a strong reason for full random slopes.

| `HeterogenietyType` | Behaviour |
|---|---|
| `NONE` (default) | Pooled model, no site effects |
| `GLOBAL` | Random effects with shared variance across sites, estimated by EM; calculates effective DoF |
| `LOCAL` | Independent random effects per site, not shared; uses regular DoF |

---

## Additive masking

Additive masking protects intermediate aggregates: each client adds a per-pair random mask to its output; masks cancel exactly in the sum, so the server only ever sees the correct aggregate — never any individual client's contribution.

### Local masking

For multiple clients running in the same process:

```python
server = Server([client1, client2], additive_masking=True)
results = server.run(max_cond_size=1)
# Numerically identical to unmasked results
```

### Network masking

When clients run on separate machines, `additive_masking=True` triggers a peer-to-peer seed exchange: the server passes each client's address to all others, peers connect directly to agree on shared random seeds, and thereafter each masks its own output independently. The server never participates in seed exchange.

```python
from fedci import connect_client, Server

nc1 = connect_client("192.168.1.10", 5555)
nc2 = connect_client("192.168.1.11", 5555)

server = Server([nc1, nc2], additive_masking=True)
results = server.run(max_cond_size=1)
```

---

## Networking

Each client runs `serve_client` on its own machine. The orchestrating server uses `connect_client` and receives a `NetworkClient` that has the same interface as a local `Client`.

### On each client machine

```python
import polars as pl
from fedci import Client, serve_client

df = pl.read_parquet("local_data.parquet")
client = Client("site1", df)

serve_client(client, port=5555)   # blocks; run as a separate process or service
```

### On the server machine

```python
from fedci import Server, connect_client

nc1 = connect_client("192.168.1.10", 5555)
nc2 = connect_client("192.168.1.11", 5555)

server = Server([nc1, nc2])
results = server.run(max_cond_size=2, workers=4)
```

### How it works

Communication uses **ZeroMQ** (ROUTER/REQ pattern) with pickle serialisation, which handles numpy arrays, dataclasses, and sets without extra schema definitions. Each calling thread gets its own ZMQ socket, so multiple concurrent tests can call the same remote client without conflicts. The remote `serve_client` loop processes requests sequentially.

### Parallelism with network clients

`workers` in `server.run()` and `server.test_repeatedly()` parallelises across threads and works transparently with both local and network clients.

Each parallel run in `test_repeatedly` registers its subsample on every client under a unique run-ID, then passes that ID with every compute call. The remote `serve_client` loop processes requests sequentially but each request carries its own run-ID, so concurrent runs never interfere — they simply reference different subsample entries in the client's registry. The registry is cleaned up automatically when each run completes.

---

## Variable types

`Client` infers variable types from the Polars schema automatically:

| Polars dtype | Treated as | Model |
|---|---|---|
| `Float32` / `Float64` | Continuous | Gaussian GLM / GAM |
| `Boolean` | Binary | Logistic regression |
| `Utf8` / `Categorical` | Categorical | Multinomial logistic |
| `Int*` | Ordinal | Proportional-odds |

When sites have different category levels for the same variable, the server takes the union across all clients automatically.
