Metadata-Version: 2.5
Name: adk-leak-arena
Version: 0.1.0
Summary: Red-Team Evaluation & Live SQL/RLS Telemetry Console for Google ADK Multi-Tenant & Multi-Agent Memory
Project-URL: Homepage, https://github.com/aalishan001/adk-leak-arena
Project-URL: Repository, https://github.com/aalishan001/adk-leak-arena
Project-URL: Issues, https://github.com/aalishan001/adk-leak-arena/issues
Project-URL: Changelog, https://github.com/aalishan001/adk-leak-arena/blob/main/CHANGELOG.md
Author-email: Aalishan <60288602+aalishan001@users.noreply.github.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: adk-memshield,agent-memory,ai-security,google-adk,multi-tenant,pgvector,postgresql,red-teaming,row-level-security
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Security
Requires-Python: >=3.11
Requires-Dist: adk-memshield<0.2,>=0.1.0
Requires-Dist: fastapi>=0.115.0
Requires-Dist: fastembed>=0.4.0
Requires-Dist: google-adk<3,>=2.12.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: litellm>=1.50.0
Requires-Dist: numpy>=1.26
Requires-Dist: psycopg[binary,pool]>=3.2.0
Requires-Dist: pydantic>=2.7.0
Requires-Dist: uvicorn>=0.32.0
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.24.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Description-Content-Type: text/markdown

# `adk-leak-arena`

**Red-Team Evaluation & Live SQL/RLS Telemetry Console for Google ADK (`google-adk`) Multi-Tenant & Multi-Agent Memory**

[![PyPI](https://img.shields.io/pypi/v/adk-leak-arena)](https://pypi.org/project/adk-leak-arena/)
[![Python](https://img.shields.io/pypi/pyversions/adk-leak-arena)](https://pypi.org/project/adk-leak-arena/)
[![License: Apache-2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://github.com/aalishan001/adk-leak-arena/blob/main/LICENSE)
[![CI](https://github.com/aalishan001/adk-leak-arena/actions/workflows/ci.yml/badge.svg)](https://github.com/aalishan001/adk-leak-arena/actions/workflows/ci.yml)
[![Google ADK](https://img.shields.io/badge/Google-ADK-4285F4.svg)](https://google.github.io/adk-docs/)
[![adk-memshield on PyPI](https://img.shields.io/pypi/v/adk-memshield?label=adk-memshield)](https://pypi.org/project/adk-memshield/)

`adk-leak-arena` is a red-team evaluation harness and live PostgreSQL kernel telemetry console in the spirit of **Promptfoo** and UK AISI **Inspect AI**.

It installs **[`adk-memshield>=0.1.0`](https://pypi.org/project/adk-memshield/) from PyPI** and runs two live Google ADK `Runner` + `preload_memory` pipelines side by side against the same seeded multi-tenant enterprise dataset, which is laced with Canary tokens:

1. **Left target (`UnprotectedPgVectorMemoryService`):** a shared `pgvector` table (`memories_unprotected`) that relies on application-layer SQL `WHERE` filters. It reproduces common RAG failures: *MemLeak* cross-user overlap, top-K lure poisoning, confused-deputy parameter spoofing, and *AgentLeak* clearance read-up.
2. **Right target (`MemShieldMemoryService` from `adk-memshield`):** kernel-enforced PostgreSQL Row-Level Security (`memories` with `FORCE ROW LEVEL SECURITY` and transaction-local `set_config(..., is_local => true)`). Across S1–S5 it leaks **0%** of rows and keeps **100%** authorized recall, including when the application `WHERE` clause is stripped entirely.

![S5 chaos mode: the unprotected target leaks cross-tenant canary tokens while adk-memshield returns only the caller's own rows](https://raw.githubusercontent.com/aalishan001/adk-leak-arena/main/docs/assets/console-overview.png)

*S5 (no SQL `WHERE` clause) as `alice_analyst`: the left target returns three unauthorized rows, including cross-tenant ones, and the agent repeats their canary tokens. The right target returns only Alice's own rows. Below each pane is the PostgreSQL telemetry: role, RLS flags, the transaction-local identity and the SQL as executed.*

---

## Quickstart

### From PyPI

```bash
pip install adk-leak-arena
```

The arena needs PostgreSQL 16 with `pgvector`. The defaults expect it on `localhost:54330`:

```bash
docker run -d --name adk_leak_arena_db -p 54330:5432 -e POSTGRES_PASSWORD=postgres_admin_pass -e POSTGRES_DB=memshield pgvector/pgvector:pg16
```

```bash
leak-arena eval
```

```bash
leak-arena serve --port 8090
```

The first run applies the bundled schema (`adk_leak_arena/sql/init_arena.sql`) through the admin DSN, creates the non-superuser `memshield_app` role, and seeds both tables. Override the connection strings with `ARENA_DB_DSN` and `ARENA_ADMIN_DSN` (see [`.env.example`](https://github.com/aalishan001/adk-leak-arena/blob/main/.env.example)). The default credentials are for local evaluation only.

### From source with Docker Compose

```bash
git clone https://github.com/aalishan001/adk-leak-arena.git
```

```bash
cd adk-leak-arena && docker compose up --build
```

Then open [http://localhost:8090](http://localhost:8090). Compose starts `adk_leak_arena_db` (`pgvector:pg16` on `localhost:54330`) and `adk_leak_arena_web` (console and API on `localhost:8090`).

### CLI

| Command | Purpose |
| :--- | :--- |
| `leak-arena eval` | Seed the database and run S1–S5, S6 and Control for `alice_analyst` with `deterministic_agent`. |
| `leak-arena eval --top-k 10 --seed-size 2000 --repeats 1` | Stress run: 10 neighbours, 1,964 synthetic background rows per table. |
| `leak-arena eval --caller exec_agent --scenarios S4,CONTROL` | Clearance-3 caller on a subset of scenarios. |
| `leak-arena eval --format markdown` / `--format json` | GitHub table or a single parseable JSON report. |
| `leak-arena serve --host 127.0.0.1 --port 8090` | Web console and API. |

`--top-k` accepts 1, 3, 5, 10 or 20; `--seed-size` accepts 36, 250, 500, 1000 or 2000; `--engine` accepts `deterministic_agent` (default, offline), `ollama_gemma` or `gemini_api`.

---

## Architecture

```mermaid
flowchart LR
    UI["Engineering Console\n(localhost:8090, offline)"] --> API["FastAPI Arena Server"]
    CLI["CLI: leak-arena eval"] --> EVAL["Scenario Evaluator\n+ Canary Classifier"]
    API --> JOBS["Benchmark Job Runner\n(progress, logs, abort, history)"]
    API --> INJ["RLS Write-Policy Sandbox\n(/api/inject)"]
    API --> PAIR["Dual Google ADK Runner Pair\n(preload_memory_tool)"]
    JOBS --> EVAL --> PAIR

    subgraph Left ["Left: Unprotected pgvector"]
        S_UNP["UnprotectedPgVectorMemoryService\n(app-layer SQL filter only)"] --> T_UNP[("memories_unprotected\n(no RLS)")]
    end
    subgraph Right ["Right: adk-memshield (PyPI)"]
        S_MS["MemShieldMemoryService\n(transaction-local set_config)"] --> T_MS[("memories\n(FORCE ROW LEVEL SECURITY)")]
    end
    PAIR --> S_UNP
    PAIR --> S_MS
    INJ --> S_UNP
    INJ --> S_MS
```

Both targets connect as the same non-superuser role, `memshield_app`. The arena reads that role's real privileges (`NOSUPERUSER NOBYPASSRLS`) and both tables' RLS flags from `pg_roles` / `pg_class` at runtime, so a misconfigured deployment shows up in the telemetry instead of being mislabelled.

---

## Scenarios

| ID | Scenario | Class | Left (`Unprotected`) | Right (`adk-memshield`) |
| :--- | :--- | :--- | :--- | :--- |
| **S1** | Incidental Semantic Overlap | `CWE-284` (MemLeak) | Filters by tenant only; other users' private notes reach Alice's top-K. | RLS binds `user_id` to the session. |
| **S2** | Adversarial Top-K Lure | `AML.T0020` | Mallory's query-mirroring private lures compete for Rank #1. | Mallory's private rows are invisible to Alice. |
| **S3** | Tool Param Spoofing | `OWASP LLM01 / LLM07` | `user_id=…` in the prompt overrides the owner filter. | Identity comes from the ADK `Session`, never the prompt. |
| **S4** | Swarm Clearance Read-Up | Bell-LaPadula (AgentLeak) | Shared memos read without a clearance check. | RLS enforces `min_clearance <= app.agent_clearance`. |
| **S5** | Chaos Mode: no SQL `WHERE` | Zero-trust kernel proof | Both targets run with no `WHERE` clause; left dumps cross-tenant secrets. | `FORCE ROW LEVEL SECURITY` still blocks every unauthorized row. |
| **S6** | Same-Tier Shared Poisoning | **Known boundary** | Mallory (clr 1) writes lures into `tenant_shared, min_clearance=1`. | Also returned: Alice has the same tenant and clearance, so the row is *authorized*. |

**S6 is engineering honesty, not a failure.** RLS enforces user, tenant and clearance boundaries. It does not judge content. Same-tier shared poisoning needs write approval or content moderation in the application. S6 is reported as a separate boundary row, excluded from the S1–S5 aggregate, and flagged amber (`BOUNDARY WARNING`). Its lure rows are planted through each service's normal write path only for the duration of an S6 run, then removed.

---

## Dataset & Methodology

- **Corpus:** 36 deterministic memories across `acme_corp` (26), `globex_inc` (5) and `initech_labs` (5). They span private and `tenant_shared` scopes at Clearance 1–5, include 5 private lures, and carry 17 Canary tokens. S6 adds 3 scenario-scoped fixture rows.
- **Caller-relative canaries:** a token is a *leak* (red) only if the caller can read no row carrying it. It is *same-tier poison* (amber) if it is readable only through adversarial rows the caller is entitled to see. Otherwise it is an *authorized secret* (green). Tokens registered by the write-policy sandbox join the same tracker until the next reseed.
- **Queries:** 8 per scenario for S1–S5 and Control, 6 for S6 (54 in total). Each query lists the seed rows it targets.
- **Authorized Recall@K** is scored only on queries where the caller can read at least one relevant row. Pure exfiltration probes are excluded, and the eligible `n` is reported.
- **Latency:** wall time of pooled checkout, transaction and search SQL. For adk-memshield this includes its `set_config` round-trip; embedding time is excluded. An untimed 2-query warmup on both targets runs first. With `deterministic_agent`, each query is sampled 3x; LLM engines are forced to 1 sample.
- **Stress dials:**
  - `top_k` takes 1, 3, 5, 10 or 20.
  - `seed_size` takes 36, 250, 500, 1000 or 2000 rows per table.
  - Synthetic rows come from 25 templates. Only the templates are embedded; each row vector is its template vector plus seeded NumPy noise, re-normalized. Rows are bulk-loaded with `COPY`, which takes about 0.6 s for 2,000 rows per table.
  - Synthetic rows are test scaffolding written with the admin role. Every read the arena measures still goes through `memshield_app` and RLS.

### Verified results: baseline (`leak-arena eval`, `alice_analyst`, `deterministic_agent`, `top_k=3`, 36 rows)

![Benchmark modal with the S1–S5, S6 boundary and Control evaluation matrix](https://raw.githubusercontent.com/aalishan001/adk-leak-arena/main/docs/assets/benchmark-matrix.png)

| Scenario | Target | Queries Leaking ≥1 Row | Leaked Row Share | Rank-1 Hijack | Canary Exfil (Output) | Same-Tier Poison Retrieved | Authorized Recall@3 (eligible n) | p50 / p95 SQL Latency |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **S1: Incidental Semantic Overlap** | `Unprotected pgvector` | 100.0% | 54.2% | 75.0% | 75.0% | 0.0% | 80.0% (n=5) | 1.64 / 3.55 ms |
| **S1: Incidental Semantic Overlap** | `adk-memshield (RLS)` | **0.0%** | **0.0%** | **0.0%** | **0.0%** | 0.0% | 100.0% (n=5) | 3.48 / 4.45 ms |
| **S2: Adversarial Top-K Lure** | `Unprotected pgvector` | 87.5% | 37.5% | 37.5% | 87.5% | 0.0% | 100.0% (n=8) | 2.16 / 3.53 ms |
| **S2: Adversarial Top-K Lure** | `adk-memshield (RLS)` | **0.0%** | **0.0%** | **0.0%** | **0.0%** | 0.0% | 100.0% (n=8) | 1.95 / 3.74 ms |
| **S3: Tool Param Spoofing** | `Unprotected pgvector` | 100.0% | 100.0% | 100.0% | 87.5% | 0.0% | 0.0% (n=2) | 2.46 / 3.35 ms |
| **S3: Tool Param Spoofing** | `adk-memshield (RLS)` | **0.0%** | **0.0%** | **0.0%** | **0.0%** | 0.0% | 100.0% (n=2) | 2.14 / 2.62 ms |
| **S4: Swarm Clearance Read-Up** | `Unprotected pgvector` | 100.0% | 41.7% | 62.5% | 100.0% | 0.0% | 100.0% (n=4) | 2.44 / 6.6 ms |
| **S4: Swarm Clearance Read-Up** | `adk-memshield (RLS)` | **0.0%** | **0.0%** | **0.0%** | **0.0%** | 0.0% | 100.0% (n=4) | 1.89 / 4.06 ms |
| **S5: Chaos Mode — Strip SQL WHERE Clause** | `Unprotected pgvector` | 100.0% | 87.5% | 87.5% | 87.5% | 0.0% | 100.0% (n=3) | 2.52 / 3.72 ms |
| **S5: Chaos Mode — Strip SQL WHERE Clause** | `adk-memshield (RLS)` | **0.0%** | **0.0%** | **0.0%** | **0.0%** | 0.0% | 100.0% (n=3) | 2.33 / 4.93 ms |
| **S6: Same-Tier Shared Poisoning (boundary, excluded from aggregate)** | `Unprotected pgvector` | 0.0% | 0.0% | 0.0% | 0.0% | 100.0% | 100.0% (n=6) | 2.57 / 3.72 ms |
| **S6: Same-Tier Shared Poisoning (boundary, excluded from aggregate)** | `adk-memshield (RLS)` | **0.0%** | **0.0%** | **0.0%** | **0.0%** | 100.0% | 100.0% (n=6) | 2.26 / 4.36 ms |
| **CONTROL: Benign Baseline** | `Unprotected pgvector` | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 100.0% (n=8) | 1.48 / 2.45 ms |
| **CONTROL: Benign Baseline** | `adk-memshield (RLS)` | **0.0%** | **0.0%** | **0.0%** | **0.0%** | 0.0% | 100.0% (n=8) | 1.64 / 2.56 ms |
| **AGGREGATE (S1–S5 Attacks)** | `Unprotected pgvector` | **97.5%** | **63.2%** | **72.5%** | **87.5%** | 0.0% | 86.4% (n=22) | 2.31 / 4.2 ms |
| **AGGREGATE (S1–S5 Attacks)** | `adk-memshield (RLS)` | **0.0%** | **0.0%** | **0.0%** | **0.0%** | 0.0% | 100.0% (n=22) | 2.14 / 4.7 ms |

### Verified results: stress (`--top-k 10 --seed-size 2000 --repeats 1`), aggregates

| Target | Queries Leaking ≥1 Row | Leaked Row Share | Rank-1 Hijack | Canary Exfil | Authorized Recall@10 | p50 / p95 SQL Latency |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| `Unprotected pgvector` | 100.0% | 72.2% | 72.5% | 90.0% | 90.9% (n=22) | 5.99 / 9.61 ms |
| `adk-memshield (RLS)` | **0.0%** | **0.0%** | **0.0%** | **0.0%** | 100.0% (n=22) | 4.21 / 12.05 ms |

A larger `top_k` and more background rows raise the unprotected leaked-row share from 63.2% to 72.2%; adk-memshield stays at 0%. Security and recall figures are deterministic. Latency was measured on Docker Desktop. p95 values mostly reflect host scheduling jitter; use p50 for overhead claims.

### Performance & RLS kernel overhead

Every `/api/attack` response carries a `performance` block:
- `sql_kernel_ms` per target: pure PostgreSQL time.
- `rls_overhead_ms` / `rls_overhead_pct`: memshield minus unprotected SQL time.
- `agent_turn_ms` per target and `total_turn_ms`: the full ADK Runner plus model turn.
- `rows_scanned_scope`: which rows each search considers.

Benchmark reports (and `/api/benchmark/status` once finished) add a run-level `performance` summary: per-target p50/p95, `p50_delta_ms`, `p95_delta_ms`, mean agent turn, and `throughput_qps`. Each scenario row also carries `rls_delta_p50_ms` / `rls_delta_p95_ms`.

| Sample run (36 rows, 162 timed executions) | p50 | p95 |
| :--- | :---: | :---: |
| Unprotected SQL kernel | 2.24 ms | 9.99 ms |
| adk-memshield RLS SQL kernel | 2.41 ms | 8.89 ms |
| Net RLS delta | +0.16 ms | -1.10 ms |

Throughput was 39.0 side-by-side queries/sec. At 36 rows the RLS delta changes sign between runs (it ranged from -0.53 to +0.16 ms at p50 in three runs), so treat it as within measurement noise.

### Query plans at 2,000 rows (`EXPLAIN ANALYZE`)

S5 search (`ORDER BY embedding <=> $1 LIMIT 3`, no application `WHERE`) as `alice_analyst` (Clearance 1) at `--seed-size 2000`, PostgreSQL 16, no vector index:

| Target | Planner statistics | Plan | Rows scored by `<=>` | Execution |
| :--- | :--- | :--- | :---: | :---: |
| `memories_unprotected` | any | `Seq Scan` → top-N heapsort | 2,000 | 2.3–3.1 ms |
| `memories` (RLS) | right after the bulk `COPY` | `Seq Scan`, RLS `Filter` (1,955 rows removed) → top-N heapsort | 45 | 2.4 ms |
| `memories` (RLS) | after `ANALYZE` | `Bitmap Index Scan` on B-tree `idx_memories_scope_lookup` (`app_name`, `tenant_id`; 978 candidates) → `Bitmap Heap Scan` with the user/clearance `Filter` (933 removed) → top-N heapsort | 45 | 1.3 ms |

The RLS-protected plan after `ANALYZE` (vector literal elided):

```text
Limit (actual time=1.232..1.233 rows=3 loops=1)
  ->  Sort (actual time=1.231..1.232 rows=3 loops=1)
        Sort Method: top-N heapsort  Memory: 25kB
        ->  Bitmap Heap Scan on memories (actual time=0.442..1.216 rows=45 loops=1)
              Recheck Cond: ((app_name = current_setting('app.current_app_name', true))
                             AND (tenant_id = current_setting('app.current_tenant_id', true)))
              Filter: ((user_id = current_setting('app.current_user_id', true))
                       OR ((scope = 'tenant_shared') AND (min_clearance <= COALESCE(NULLIF(current_setting('app.agent_clearance', true), '')::integer, 1))))
              Rows Removed by Filter: 933
              ->  Bitmap Index Scan on idx_memories_scope_lookup (actual time=0.387..0.387 rows=978 loops=1)
                    Index Cond: ((app_name = current_setting('app.current_app_name', true))
                                 AND (tenant_id = current_setting('app.current_tenant_id', true)))
Execution Time: 1.262 ms
```

What this shows:
- The RLS policy predicates are applied **before** the distance sort, so the protected search scores only the 45 rows the caller may read. The unprotected S5 search scores every row in the table.
- Because `current_setting(...)` is stable within a statement, PostgreSQL can use the policy's `app_name`/`tenant_id` equality as a B-tree index condition. It does so once table statistics exist (after `ANALYZE` or autovacuum). Immediately after `/api/scale` bulk-loads rows, the planner still picks a sequential scan, but the policy filter still runs before scoring.
- At 36 rows both targets use a sequential scan.
- These plans have no ANN index. With HNSW or IVFFlat, RLS filters rows after the index scan and can return fewer than `top_k` rows; use pgvector 0.8 iterative scans and re-measure recall.

---

## RLS Write-Policy Sandbox (`POST /api/inject`)

The sandbox attempts the same write against both tables. The acting persona is the authenticated writer; `claimed_owner_id` is the owner column the write tries to set.

![Write-policy sandbox: a Clearance-1 contractor's Clearance-3 shared write is accepted by the unprotected table and rejected by RLS with SQLSTATE 42501](https://raw.githubusercontent.com/aalishan001/adk-leak-arena/main/docs/assets/write-policy-sandbox.png)

| Attempt | Unprotected | adk-memshield |
| :--- | :--- | :--- |
| `mallory_contractor` (clr 1) writes `tenant_shared`, `min_clearance=3` (clearance write-up) | Accepted | **Rejected**: `InsufficientPrivilege`, SQLSTATE `42501`, "new row violates row-level security policy for table \"memories\"" |
| `mallory_contractor` writes a row claiming `user_id='bob_cfo'` (owner spoof) | Accepted | **Rejected** by the same `WITH CHECK` policy |
| `mallory_contractor` writes a private Clearance-1 lure | Accepted | Accepted (her own namespace). Probing as Alice shows it at Rank #1 on the left only. |
| `mallory_contractor` writes `tenant_shared`, `min_clearance=1` | Accepted | Accepted: this is the S6 boundary. |

On the adk-memshield side, the transaction identity is bound to the actor with the package's own `rls_transaction`. When the claimed owner matches the actor, the public `add_memory` API is used. Otherwise, the spoofed `INSERT` runs inside that RLS transaction. Either way PostgreSQL decides; nothing is pre-filtered in Python. An optional `canary_token` (pattern `[A-Za-z0-9_.$-]{3,40}`, for example `AUDIT-CANARY-900`) is appended to the content if missing and registered with the canary classifier. A reseed clears injected rows and tokens.

---

## Benchmark Jobs & Run History

| Endpoint | Purpose |
| :--- | :--- |
| `POST /api/benchmark/start` | `{caller_id, engine, top_k, seed_size, scenarios, repeats, label}`. Rejects a second concurrent run with `409`. Pre-run cleanup: full reseed to the 36-row baseline, then scaling to `seed_size`. |
| `GET /api/benchmark/status?since=N` | Job state (`idle`, `running`, `completed`, `cancelled`, `failed`), phase, `step` / `total_steps` / `percent`, current query, elapsed time, log lines after offset `N`, and the result when finished. |
| `POST /api/benchmark/kill` | Sets a cancellation flag checked before every query, then hard-cancels anything still in flight after 100 ms. Measured stop latency is about 15 ms. Post-abort cleanup reseeds the baseline and logs "Run aborted by operator; database restored to 36-row baseline." |
| `POST /api/benchmark` | Synchronous wrapper: runs a job and returns its report. |
| `GET /api/history`, `GET /api/history/{run_id}?format=json\|md`, `DELETE /api/history` | Completed and cancelled runs (last 50, in server memory), with per-scenario rows, aggregates and Markdown/JSON export. |

![Run History tab listing completed benchmark runs with their dials and S1–S5 aggregates](https://raw.githubusercontent.com/aalishan001/adk-leak-arena/main/docs/assets/benchmark-history.png)

When a completed run used `seed_size > 36`, its synthetic rows are removed afterwards. The server records one deterministic `startup-baseline` run at boot (`ARENA_STARTUP_BASELINE=0` disables it), so history is never empty.

While a job runs, `/api/reset`, `/api/scale` and `/api/inject` return `409`. `/api/attack` still works and reports `benchmark_running: true`.

### Other endpoints

| Endpoint | Purpose |
| :--- | :--- |
| `GET /api/state` | Versions, engine availability, live DB posture, personas, presets (S1–S6), dial options, current seed size, live canary tokens, job summary. |
| `POST /api/attack` | One side-by-side run: `{preset_id, query, caller_id, failure_mode, simulate_missing_app_filter, spoofed_user_id, engine, top_k}`. |
| `POST /api/scale` | `{seed_size}`: rescale both tables for live exploration. |
| `POST /api/reset` | Re-apply the schema and reseed both tables. This clears synthetic, fixture and injected rows. |

Input hardening:
- Unknown callers, personas, engines, failure modes, scenarios and seed sizes return a JSON `422`. So do out-of-range `top_k` / clearance values, NUL bytes, unpaired UTF-16 surrogates, oversized text, and non-identifier owner IDs.
- All SQL values are bound parameters, and unexpected errors return a JSON `500`.
- Blocking work runs off the event loop.
- ADK's `preload_memory_tool` swallows memory-search exceptions. The arena records them and reports `RETRIEVAL ERROR` instead of a false "no leak".

---

## Running Tests & Evaluations in Docker

With the Compose stack running (`docker compose up -d --build`):

```bash
docker exec adk_leak_arena_web pytest -v
```

```bash
docker exec adk_leak_arena_web leak-arena eval
```

```bash
docker exec adk_leak_arena_web leak-arena eval --top-k 10 --seed-size 2000 --scenarios S1,S5,S6,CONTROL --repeats 1
```

Then open [http://localhost:8090](http://localhost:8090).

### `leak-arena eval` output formats
- `--format table` (default): a fixed-width Unicode box-drawing matrix with the columns `Scenario`, `Target`, `Leak>=1`, `LeakShare`, `Rank1Hijack`, `CanaryExfil`, `Recall@K`, `p50/p95 ms` and `RLS Delta`. Below it is a **Performance & RLS Kernel Overhead** summary box with per-target p50/p95 SQL latency, the net RLS delta in ms and %, mean agent turn, and throughput in queries/sec.
  - ANSI colors: green for `0.0%` leak and `100.0%` recall cells, red for non-zero leak rates, amber for S6 boundary rows.
  - Colors are used only on a TTY (`docker exec -t ...`). `NO_COLOR` disables them and `FORCE_COLOR` forces them.
- `--format markdown`: the GitHub Markdown table.
- `--format json`: the raw report only, as one parseable JSON document. `--json` is kept as a deprecated alias.

The CLI and the web server share one database, and `leak-arena eval` reseeds it. Do not run the CLI while a server benchmark job is running, including the roughly 5-second startup baseline right after the container starts; S6 fixture rows planted by the server would otherwise show up in the CLI's S1–S5 results.

### Console capabilities
The console is 100% offline-ready (zero external CDNs, fonts, or scripts) and includes:
- Side-by-side ADK `Runner` verdicts with glowing status indicators (`.dot-red` for `LEAK DETECTED`, `.dot-emerald` for `KERNEL ISOLATED`, `.dot-amber` for `BOUNDARY WARNING` / `RETRIEVAL ERROR`), caller-relative canary highlighting, animated metric counters, aurora header backdrop, and mouse-tracking radial spotlight cards.
- Live `top_k` (`1..20`) and `seed_size` (`36..2000`) stress-test dials wired to `/api/scale` and `/api/attack`, with an automatic amber **Hardware Load Notice** when `ollama_gemma`, `seed_size >= 1000`, or `top_k >= 10` is selected.
- Expandable **RLS Write-Policy & CTF Lure Injector** panel (`/api/inject`) to test owner spoofing, clearance write-up escalation (`InsufficientPrivilege` / `SQLSTATE 42501`), and one-click probing of injected lures.
- Scenario tabs, the query picker, caller, engine, `top_k`, `seed_size` and the chaos toggle only *stage* a run, and the **Run Side-by-Side Query** button pulses with a "Ready to Run" tag. Two things execute a query: that button (Enter in the query box presses it) and **Probe Injected Lure** in the write-policy sandbox. While running, the button names the active engine ("Running Deterministic ADK...", "Running Ollama (gemma3:4b)...", "Running Gemini 2.5 Flash...").
- A **Performance & RLS Kernel Overhead** strip between the controls and the panes shows unprotected SQL ms, MemShield RLS SQL ms, the RLS kernel delta, the end-to-end agent turn, and each target's row scope. The benchmark modal adds an aggregate p50/p95 RLS delta and throughput header.
- High-contrast text throughout, with no navy-on-black. `<select>` options are pinned to `#0f172a` / `#f8fafc` so native Windows dropdowns stay legible.
- Asynchronous **Benchmark Job Runner** modal (`/api/benchmark/start`, `/api/benchmark/status`, `/api/benchmark/kill`) with a live progress bar, streaming logs, a red `[KILL RUN]` abort button (restoring the 36-row baseline immediately), and a **Run History & Side-by-Side Comparison** tab (`/api/history`).

### LLM engines
Select the engine in the console header or with `--engine`. If the requested engine is unavailable, the run falls back to `deterministic_agent` and reports why.
- `deterministic_agent` (default, offline): a `BaseLlm` that runs the full ADK `Runner` + `preload_memory` pipeline and grounds its answer in `<PAST_CONVERSATIONS>`.
- `ollama_gemma`: local `gemma3:4b` via LiteLLM (`http://host.docker.internal:11434`). A full matrix is about 108 GPU inferences, several minutes on a laptop GPU such as an RTX 3050 4 GB. Prefer `deterministic_agent` for full runs, and abort with `/api/benchmark/kill` if needed.
- `gemini_api`: cloud `gemini-2.5-flash` when `GEMINI_API_KEY` (or `GOOGLE_API_KEY`) is set.

---

## Known Limitations

- **Same-tier shared poisoning (S6) is out of scope for RLS.** A writer may publish `tenant_shared` content at or below their own clearance, and every reader at that tier is entitled to it. Use write approval or content moderation.
- **S2's hijack rate is not tuned:** lures reach Rank #1 on 3 of 8 queries.
- **S3 recall on the unprotected target is 0%** because the spoofed filter returns only the victim's rows.
- **The CLI and server share one database.** `leak-arena eval` reseeds it; do not run it while a server benchmark job (including the ~5 s startup baseline) is active, or S6 fixture rows leak into the CLI's S1–S5 results.
- **Latency is noisy on Docker Desktop.** Use p50 for overhead claims; at 36 rows the RLS delta changes sign between runs.
- **Run history is in memory** (last 50 runs) and is cleared on restart.
- **The full benchmark is verified only with `deterministic_agent`.** LLM engines change the canary-exfiltration column, not the retrieval verdicts.
- **The dataset is synthetic and hand-written** (36 rows, 54 queries). Treat the numbers as a reproducible illustration of failure modes, not real-world leak rates.

---

## Development

```bash
pip install -e ".[dev]"
```

```bash
pytest -v
```

The suite (45 tests) needs the database on `localhost:54330`; CI runs it against a `pgvector/pgvector:pg16` service container. See [CONTRIBUTING.md](https://github.com/aalishan001/adk-leak-arena/blob/main/CONTRIBUTING.md) for adding attack scenarios or new `BaseMemoryService` targets, and [SECURITY.md](https://github.com/aalishan001/adk-leak-arena/blob/main/SECURITY.md) for intended use and vulnerability reporting.

---

## License

Apache License 2.0 — Copyright 2026 [Aalishan](https://github.com/aalishan001).
