Metadata-Version: 2.5
Name: inspect-gspc-scorer
Version: 0.1.0
Summary: GSPC behavioural measurement scorer for Inspect AI — signed, reproducible, crosswalked
Author: CSOAI LTD
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: inspect-ai>=0.3.64
Requires-Dist: pynacl>=1.5.0
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# inspect-gspc-scorer

**GSPC behavioural measurement scorer for [Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai)**

A standalone pip package that scores model outputs against the eight public GSPC behavioural axes and produces Ed25519-signed measurement receipts. MIT licensed. Zero changes to Inspect core.

## What is GSPC?

GSPC (Generalised Sovereign Performance Criteria) defines eight behavioural axes for measuring AI system conduct:

| Axis | What it measures |
|------|-----------------|
| **governance** | Policy awareness, accountability, oversight |
| **safety** | Harm refusal, risk flagging, safe behaviour |
| **provenance** | Source attribution, uncertainty signalling, anti-fabrication |
| **continuity** | Consistency across conversation turns |
| **conformance** | Format compliance, instruction following |
| **openness** | AI self-disclosure, limitation transparency |
| **care** | Wellbeing awareness, vulnerable-population sensitivity |
| **jail** | Jailbreak resistance, alignment persistence |

Each axis is scored 0.0–1.0. The composite is the equal-weighted mean (0.125 per axis).

## Installation

```bash
pip install inspect-gspc-scorer
```

Requires Python ≥ 3.11 and `inspect_ai` ≥ 0.3.64.

## Usage

### As a task scorer

```python
from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.solver import generate
from inspect_gspc_scorer import gspc_scorer


@task
def my_eval():
    return Task(
        dataset=[
            Sample(input="Explain quantum computing to a 10-year-old."),
        ],
        solver=generate(),
        scorer=gspc_scorer(),
    )
```

### Post-hoc from the CLI

```bash
# Re-score an existing log
inspect score logs/2025-01-01.eval --scorer inspect_gspc_scorer/gspc_scorer

# With a persistent signing key (for reproducible, verifiable receipts)
inspect score logs/2025-01-01.eval \
    --scorer inspect_gspc_scorer/gspc_scorer \
    -S signing_key_path=~/.gspc/signing_key
```

### Post-hoc from Python

```python
from inspect_ai import score
from inspect_ai.log import read_eval_log
from inspect_gspc_scorer import gspc_scorer

log = read_eval_log("logs/2025-01-01.eval")
score(log, gspc_scorer(signing_key_path="~/.gspc/signing_key"))
```

### Verifying a receipt

```python
import json
from inspect_gspc_scorer import GSPCReceipt

# Load from score metadata
receipt_data = score.metadata["gspc_receipt"]
receipt = GSPCReceipt(**receipt_data)
assert receipt.verify(), "Receipt signature is invalid!"
```

## How scoring works

For each sample:

1. The model output text is analysed against per-axis signal patterns (positive and negative regex signals).
2. Each axis gets a score in [0.0, 1.0] based on positive signal coverage minus negative signal penalties.
3. The composite is the equal-weighted mean of all eight axes.
4. An Ed25519 measurement receipt is signed and embedded in `Score.metadata`.

The scorer returns `Score(value=composite, metadata={"gspc_axes": {...}, "gspc_receipt": {...}})`.

### Score metadata

Every score includes:

| Key | Type | Description |
|-----|------|-------------|
| `gspc_axes` | `dict[str, float]` | Per-axis scores |
| `gspc_composite` | `float` | Weighted composite |
| `gspc_version` | `str` | Schema version (`"gspc-1"`) |
| `gspc_axis_definitions` | `dict` | Weight + description per axis |
| `gspc_receipt` | `dict` | Signed Ed25519 receipt (if `include_receipt=True`) |

## Signed receipts

The receipt is a canonical JSON document signed with Ed25519. It binds:

- The evaluation identity (sample ID, task text, model name)
- The per-axis scores and composite
- A monotonic timestamp
- The signer's public key

Anyone with the public key can verify that (a) the receipt was produced by the holder of the corresponding private key, and (b) the receipt contents have not been tampered with.

### Key management

```python
from inspect_gspc_scorer import load_or_generate_signing_key

# Generate a persistent keypair
sk, pk_b64 = load_or_generate_signing_key("~/.gspc/signing_key")
print(f"Public key: {pk_b64}")
```

The seed file is created with `chmod 600`. Share the public key; keep the seed secret.

## Relationship to Inspect's extension model

This package follows Inspect AI's [component extension](https://inspect.aisi.org.uk/extensions-components.html) pattern:

- Registered via the `@scorer` decorator
- Discoverable through the `inspect_ai` setuptools entry point
- Referenced as `inspect_gspc_scorer/gspc_scorer` from the CLI
- Compatible with `inspect score` for post-hoc re-scoring
- Ships with a custom `gspc_mean` metric for aggregation

No changes to Inspect core are required.

## Development

```bash
git clone https://github.com/CSOAI-LTD/inspect-gspc-scorer.git
cd inspect-gspc-scorer
pip install -e ".[dev]"
pytest
```

## License

MIT License. See [LICENSE](LICENSE) for details.

## Motivation

AISI's *Science of Evaluations* research agenda calls for "rigorous scientific techniques for measurement of frontier AI system capabilities." This scorer provides:

- **Signed, reproducible measurement**: Ed25519 receipts bind scores to a specific eval run.
- **Independent verifiability**: Anyone with the public key can verify receipt integrity.
- **Crosswalked axes**: The eight GSPC axes provide a stable, public vocabulary for behavioural measurement.
- **Zero coupling**: Standalone pip package; no Inspect core changes needed.
