Metadata-Version: 2.4
Name: harness-eval
Version: 6.0.0
Summary: Evaluate and compare AI agent setups through experiments, inspections, and rubric scoring.
License-Expression: Apache-2.0
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: anthropic>=0.40
Requires-Dist: click>=8.0
Requires-Dist: google-genai>=1.0
Requires-Dist: pydantic>=2.0
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: scikit-learn>=1.0
Provides-Extra: bash-ast
Requires-Dist: bashlex>=0.18; extra == 'bash-ast'
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pre-commit>=4.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: tiktoken
Requires-Dist: tiktoken>=0.7; extra == 'tiktoken'
Provides-Extra: watch
Requires-Dist: watchfiles>=1.0; extra == 'watch'
Provides-Extra: yara
Requires-Dist: yara-python>=4.0; extra == 'yara'
Description-Content-Type: text/markdown

# harness-eval

[![CI](https://github.com/redhat-community-ai-tools/harness-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/redhat-community-ai-tools/harness-eval/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/harness-eval)](https://pypi.org/project/harness-eval/)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org/downloads/)
[![Rules](https://img.shields.io/badge/rules-69-blue)](https://github.com/redhat-community-ai-tools/harness-eval#inspection-rules-69)
[![License: Apache 2.0](https://img.shields.io/badge/license-Apache%202.0-green)](LICENSE)

Evaluate AI code agent setups for best practices, redundancy, security, and cross-component issues.

Available as a **CLI tool**, a **GitHub Action**, a **Claude Code plugin**, and **Cursor commands**.

Supports Claude Code, Cursor, Copilot, Gemini CLI, and OpenCode projects. Auto-detects which tool(s) a project uses. Also discovers third-party modules installed via package managers.

## What it does

Most tools test whether a skill produces correct output. This tool checks the setup itself: CLAUDE.md, GEMINI.md, AGENTS.md, skills, commands, hooks, MCP configs, agents, `.cursor/rules/*.mdc`, `.cursorrules`, `.github/prompts/`, `.opencode/`.

### Security

Cross-component analysis is the core differentiator. Most linters check files in isolation; harness-eval builds a component graph that traces data flows across skills, agents, hooks, and MCP servers. This catches threats like credential exfiltration chains (one skill reads secrets, another has network access) and confused deputy attacks (an agent disallows a tool but references a skill that uses the equivalent capability).

Beyond cross-component analysis, the security engine includes Python taint tracking, bash taint analysis, YARA signature scanning, CVE lookups via OSV.dev, and optional LLM-based semantic review.

Five commands, same engine:

| Command | What it does | LLM in CLI | LLM in Claude Code / Cursor |
|---------|-------------|-----------|----------------------------|
| `lint` | 69 deterministic rules + system analysis (token budget, trigger overlaps, dependencies). Fast, CI-suitable. Supports `--format sarif` for GitHub code scanning. | No | No |
| `review` | Per-component rubric review with 0-3 scoring per dimension, 21 cross-type checks. KEEP/REVIEW/REMOVE verdicts. | Yes (API key) | Yes (in-session) |
| `security` | All security rules + YARA + CVE lookups + semantic review. SAFE/CAUTION/UNSAFE. | Scan: no. Semantic review: `--review` flag | Yes (in-session) |
| `skill` | Deep-evaluate one skill individually and in context of the full setup. | Lint: no. Rubric: `--rubric` flag | Yes (in-session) |
| `rules` | List all available rules with ID, severity, target type, and description. Filter by `--category` or `--target`. | No | No |

## Supported AI Assistants

Auto-detects which tool(s) a project uses and evaluates all discovered components.

| Assistant | What it discovers |
|-----------|------------------|
| Claude Code | `CLAUDE.md`, `skills/`, `commands/`, `.claude/agents/`, `.claude/settings.json`, `.mcp.json` |
| Cursor | `.cursor/rules/*.mdc`, `.cursorrules`, `.cursor/commands/`, `.cursor/skills/`, `.cursor/hooks.json`, `.cursor/mcp.json` |
| Copilot | `.github/skills/`, `.github/prompts/`, `.github/agents/` |
| Gemini CLI | `GEMINI.md`, `.gemini/commands/` |
| OpenCode | `AGENTS.md`, `.opencode/commands/`, `.opencode/agents/` |
| Third-party modules | `.lola/modules/` (skills, commands, agents installed via package managers) |

Multi-tool projects are fully supported. When a project contains files for multiple assistants, all are discovered, deduplicated, and evaluated together.

## Install

See [`INSTALL.md`](INSTALL.md) for all installation options, CI integration, and configuration (Available as a **CLI tool**, **GitHub Action**, **Claude Code plugin**, and **Cursor commands**)

## Inspection Rules (69)

| Category | Rules | What they check |
|----------|-------|-----------------|
| Structural | 1 | SKILL.md exists |
| Frontmatter | 3 | Description required/quality, format valid |
| Content | 8 | Duplicate detection (TF-IDF), broken references, circular references, token budget, orphan skills, MCP-skill alignment, total context budget, permission escalation |
| Quality | 8 | Imprecise instructions, redundant guidance, unfinished content, example gap, stale references, negative-only prohibitions, scope overreach, trigger manipulation |
| Security | 13 | Credential access, prompt injection, data exfiltration, obfuscation, reverse shells, AST analysis, Python taint tracking, bash taint tracking, MCP least-privilege, tool poisoning, coercive override, stealth persistence, prompt exfiltration |
| Security (opt-in) | 2 | YARA signatures, CVE lookups via OSV.dev |
| Cross-component | 1 | Cross-component exfiltration chains, confused deputy attacks, phantom MCP tool references |
| Commands | 11 | Description, script exists, duplicates, credentials, injection, exfiltration, obfuscation, reverse shells, skill overlap, shadows built-in, references nonexistent skill |
| CLAUDE.md | 3 | Exists, skill duplication, generic advice detection |
| MCP | 4 | Configuration structure, duplicate servers, suspicious endpoints (localhost/private IPs), wildcard tool exposure |
| Hooks | 5 | Structure validation, script boundary, dangerous commands, env variable leakage, network access |
| Agents | 10 | Description, model specified, skills exist, tool format, constraint matching, credentials, injection, exfiltration, obfuscation, reverse shells |

Four presets: `recommended` (default), `strict`, `security`, `pre-workflow`.

### Frameworks

Rules are mapped to industry security frameworks where applicable:

- **OWASP LLM Top 10** (2025): coverage across LLM01 (prompt injection), LLM02 (sensitive data), LLM06 (excessive agency), and others
- **OWASP Agentic Security**: AG04 (data exfiltration), AG05 (credential access), and related controls
- **MITRE ATLAS**: AML.T0054 (LLM prompt injection) and related techniques

## Security

For a full overview of how this tool protects your code, your credentials, and your supply chain, see [`how-can-you-know-its-safe-to-use-this-tool.md`](how-can-you-know-its-safe-to-use-this-tool.md).

## Privacy and Data Handling

`harness-eval` reads files from your project directory to analyze your AI agent setup. Here is what happens with your data in each mode:

| Command | Sends data externally? | What is sent | Where |
|---------|----------------------|--------------|-------|
| `lint` | No | Nothing. Fully offline. | N/A |
| `review` | Yes (CLI only) | Code snippets from your setup files | Gemini or Anthropic API (your choice via `--provider`) |
| `security` | Scan: No. `--review`: Yes (CLI only) | Code snippets from flagged files | Gemini or Anthropic API |
| `skill` | Lint: No. `--rubric`: Yes (CLI only) | The skill content being evaluated | Gemini or Anthropic API |

When used as a **Claude Code plugin**, review/security/skill commands use the existing Claude session. No additional API calls are made.

When used as **Cursor commands**, the evaluation happens in the Cursor session. No additional API calls are made.

**File access:** The tool only reads files within the project directory you point it at. Path traversal protections prevent reading files outside the project boundary.

## Contributing

See [`CONTRIBUTING.md`](CONTRIBUTING.md) for adding rules and submitting PRs.

## Changelog

See [`CHANGELOG.md`](CHANGELOG.md) for release history.

## Roadmap

See [open issues](https://github.com/redhat-community-ai-tools/harness-eval/issues) for planned improvements and feature requests.
