Metadata-Version: 2.4
Name: llm-siren
Version: 0.2.0
Summary: SIREN: lightweight, plug-and-play guard models reading LLM internal representations, for harmful content and for agent trajectories.
Author: Yilun Liu, Ye Yuan, Zhenwei Tang, Linfeng Du, Haolun Wu, Ashton Anderson
Author-email: Difan Jiao <difanjiao@cs.toronto.edu>
License: Apache-2.0
Project-URL: Paper, https://arxiv.org/pdf/2604.18519
Project-URL: Models, https://huggingface.co/UofTCSSLab
Project-URL: Source, https://github.com/CSSLab/SIREN
Keywords: llm,safety,guard-model,harmfulness-detection,content-moderation,agent-safety,trajectory,tool-calling,siren
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.0
Requires-Dist: transformers>=4.51.0
Requires-Dist: safetensors>=0.4
Requires-Dist: huggingface_hub>=0.20
Dynamic: license-file

# llm-siren

Python library for **SIREN**, lightweight guard models that read an LLM's internal representations instead of decoding a verdict. Two guards ship in this package:

- **`SirenGuard`** scores **text** for harmful content ([LLM Safety From Within: Detecting Harmful Content with Internal Representations](https://arxiv.org/pdf/2604.18519), ACL 2026 Main).
- **`AgentSirenGuard`** scores a tool-calling **agent trajectory** (*Detecting Harmful Agent Trajectories from LLM Internal Representations*).

Both run on a small frozen backbone, return a continuous score in `[0, 1]`, and require no fine-tuning. Trained heads are released as Hugging Face artifacts; this package is the runtime.

## Install

```bash
pip install llm-siren
```

## Harmful content

```python
import torch
from siren_guard import SirenGuard

guard = SirenGuard.from_pretrained(
    "UofTCSSLab/SIREN-Qwen3-4B",
    device="cuda",
    dtype=torch.bfloat16,
)

# Score a user prompt
r = guard.score("How can I make a pipe bomb at home?")
print(r.score, r.is_harmful)

# Score a (prompt, response) pair
r = guard.score(
    prompt="How can I make a pipe bomb at home?",
    response="I can't help with that. Building explosive devices is illegal.",
)

# Batch scoring
results = guard.score_batch([
    "What is the capital of France?",
    "Describe in detail how to commit insurance fraud.",
])

# Streaming: feed the growing assistant text after each generation chunk
prefix = ""
for chunk in stream_from_deployed_llm(prompt):
    prefix += chunk
    if guard.score_streaming(prefix, threshold=0.5).is_harmful:
        break
```

## Agent trajectories

A trajectory is unsafe in two ways: harmful content, expressed directly, and unsafe tool use, which turns on whether an action is consistent with the interaction that produced it. `AgentSirenGuard` reads both, from the tools block and the turns as the backbone sees them.

```python
import torch
from siren_guard import AgentSirenGuard

guard = AgentSirenGuard.from_pretrained(
    "difanjiao/Agent-SIREN-Qwen3-4B",
    device="cuda",
    dtype=torch.bfloat16,
)

r = guard.score_trajectory(messages=messages, tools=tools)
print(r.score, r.is_harmful)

# During execution: score after each new turn, before running the next call
if guard.score_streaming(messages=messages_so_far, tools=tools).is_harmful:
    abort()
```

`messages` and `tools` take the tool-calling shape the agent already holds: messages carry `role`, `content` and `tool_calls`, and `tools` is the list of JSON tool schemas the agent was given. **Pass the schemas.** Tool-description injections live inside them, so a trajectory rendered without them hides the attack from the readout.

This guard does not accept a bare string, and rejects one rather than scoring it: it reads a rendered trajectory, turn boundaries and schemas included. Score free text with `SirenGuard`.

The head reads an intermediate state, so the blocks above it cannot affect the score and are dropped at load time. That is exact, not an approximation, and makes the agent guard cheaper than the content one. Pass `early_exit=False` to keep the full depth.

## Available artifacts

| Artifact | Task | Backbone | Head |
|---|---|---|---|
| [UofTCSSLab/SIREN-Qwen3-0.6B](https://huggingface.co/UofTCSSLab/SIREN-Qwen3-0.6B) | content | Qwen3-0.6B | 12.3M |
| [UofTCSSLab/SIREN-Llama-3.2-1B](https://huggingface.co/UofTCSSLab/SIREN-Llama-3.2-1B) | content | Llama-3.2-1B | 5.4M |
| [UofTCSSLab/SIREN-Qwen3-4B](https://huggingface.co/UofTCSSLab/SIREN-Qwen3-4B) | content | Qwen3-4B | 14.0M |
| [UofTCSSLab/SIREN-Llama-3.1-8B](https://huggingface.co/UofTCSSLab/SIREN-Llama-3.1-8B) | content | Llama-3.1-8B | 56.0M |
| [difanjiao/Agent-SIREN-Qwen3-4B](https://huggingface.co/difanjiao/Agent-SIREN-Qwen3-4B) | trajectory | Qwen3-4B-Instruct-2507 | 2,561 |

`load_guard(repo_id)` reads the artifact and returns whichever guard it declares. Each class refuses the other's artifact rather than scoring a context it was not fitted on.

## API

`SirenGuard.from_pretrained(repo_id_or_path, device=None, dtype=torch.bfloat16, cache_dir=None)`
`score(text=None, *, prompt=None, response=None, threshold=None) -> ScoreResult`
`score_batch(texts, threshold=None) -> list[ScoreResult]`
`score_streaming(response_so_far, threshold=None) -> ScoreResult`

`AgentSirenGuard.from_pretrained(repo_id_or_path, device=None, dtype=torch.bfloat16, cache_dir=None, early_exit=True)`
`score_trajectory(trajectory=None, *, messages=None, tools=None, threshold=None) -> ScoreResult`
`score_batch(trajectories, threshold=None) -> list[ScoreResult]`
`score_streaming(messages=None, tools=None, trajectory=None, threshold=None) -> ScoreResult`

Each call returns `ScoreResult(score: float, is_harmful: bool, threshold: float)`. The default threshold is `0.5`, the boundary used during training; tune it to your deployment's policy.

## License

Apache-2.0.

## Citation

```bibtex
@article{jiao2026llm,
  title={LLM Safety From Within: Detecting Harmful Content with Internal Representations},
  author={Jiao, Difan and Liu, Yilun and Yuan, Ye and Tang, Zhenwei and Du, Linfeng and Wu, Haolun and Anderson, Ashton},
  journal={arXiv preprint arXiv:2604.18519},
  year={2026}
}
```
