Metadata-Version: 2.4
Name: nikhail
Version: 0.2.0
Summary: Lint agent capabilities: Agent Skills (SKILL.md) and MCP tool definitions
Project-URL: Repository, https://github.com/prove-ai/Nikhail
Project-URL: Issues, https://github.com/prove-ai/Nikhail/issues
License: MIT
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: pyyaml>=6.0
Provides-Extra: advisory
Requires-Dist: anthropic>=0.40; extra == 'advisory'
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

# Nikhail

**Lint agent capabilities.** Nikhail statically analyzes the two artifact types that
determine how well AI agents discover and use capabilities:

- **Agent Skills** — `SKILL.md` packages (frontmatter, instructions, bundled scripts and references)
- **MCP tools** — tool definitions as returned by an MCP server's `tools/list`

It is designed to run locally as a CLI and in CI as a GitHub Action, with rules grounded
in published research: the skill-smells study ([arXiv:2607.01456](https://arxiv.org/abs/2607.01456))
and the MCP description-quality study ([arXiv:2602.14878](https://arxiv.org/abs/2602.14878)).

---

## The pieces (8)

| # | Piece | Where | What it's for |
|---|-------|-------|---------------|
| 1 | **CLI + engine** | [cli.py](src/nikhail/cli.py), [engine.py](src/nikhail/engine.py), [artifacts.py](src/nikhail/artifacts.py), [output.py](src/nikhail/output.py) | Everything interesting lives here: artifact discovery/parsing, the rule registry, and text/JSON/SARIF output with CI-friendly exit codes. The Action is only a wrapper around this. |
| 2 | **Rules** | [rules/skills.py](src/nikhail/rules/skills.py) (S001–S019), [rules/mcp.py](src/nikhail/rules/mcp.py) (M001–M008), [rules/skills_advisory.py](src/nikhail/rules/skills_advisory.py) (A001–A014, opt-in LLM tier backed by [advisory.py](src/nikhail/advisory.py)) | The deterministic rule catalog plus the v2 advisory tier. Each rule is a small registered function — adding a rule is writing one function, which is also the plugin story going forward. |
| 3 | **Fixtures** | [fixtures/](fixtures/) | Known-good and known-bad artifacts. `good-skill` and `tools-good.json` must lint clean; `smelly-skill`, `broken-skill`, and `tools-smelly.json` must trigger specific rules. [fixtures/smells/](fixtures/smells/) holds one fixture per smell in the arXiv:2607.01456 taxonomy (26 total): the 12 deterministically-detectable ones must trigger their mapped rule, and the 14 semantic ones must lint clean deterministically (the advisory tier flags them). [fixtures/smells-clean/](fixtures/smells-clean/) holds a clean counterpart per semantic smell for the precision side of the live eval. They are test data, demo material, and the CI contract all at once. |
| 4 | **Tests** | [tests/](tests/) | `test_rules.py` asserts which rules fire on which fixtures; `test_cli.py` asserts exit codes and SARIF shape — i.e., exactly the behavior CI depends on. |
| 5 | **The Action** | [action.yml](action.yml) | A composite GitHub Action: sets up Python, `pip install`s this repo, runs the CLI. Deliberately too thin to have bugs of its own. Emits a SARIF file you can chain into `github/codeql-action/upload-sarif`. |
| 6 | **CI workflow** | [.github/workflows/ci.yml](.github/workflows/ci.yml) | Three jobs: `tests` (pytest), `fixture-contract` (clean fixtures exit 0, smelly fixtures exit 1), and `action-selftest` (runs the Action itself via `uses: ./`). The repo dogfoods its own Action on every push. |
| 7 | **act config** | [.actrc](.actrc) | Lets `act` run the CI workflow locally in Docker with a GitHub-like runner image, so you can test workflow wiring without pushing. |
| 8 | **Packaging** | [pyproject.toml](pyproject.toml) | Standard Python packaging (hatchling). Installable with `pip`/`uv`/`pipx`/`uvx`; the `nikhail` console script comes from here. |

---

## Commands

### Setup (once)

```bash
cd Nikhail
uv venv                      # create .venv (or: python3 -m venv .venv)
uv pip install -e ".[dev]"   # editable install + pytest
source .venv/bin/activate    # or prefix commands with .venv/bin/
```

### Run the linter

```bash
nikhail fixtures/skills/smelly-skill        # lint one skill (findings, exit 0 — warnings only)
nikhail fixtures/mcp/tools-smelly.json      # lint MCP tool definitions (errors, exit 1)
nikhail fixtures/                            # walk a tree: finds every SKILL.md + tools JSON
nikhail ~/.claude/skills                     # try it on your real skills
nikhail . --fail-on warning                  # stricter gate: warnings also fail the run
nikhail . --format json                      # machine-readable output
nikhail . --sarif nikhail.sarif            # also write SARIF (what the Action does)
nikhail . --html report.html                 # self-contained interactive HTML report
                                               #   skills: findings + source excerpts
                                               #   MCP: per-tool cards pairing each finding
                                               #   with the description the model reads
                                               #   (embeds source excerpts — treat like source)
nikhail . --handoff handoff.md               # self-contained fix pack: findings + full
                                               #   source per flagged artifact, absolute
                                               #   paths — enough context for a fixing
                                               #   agent in a fresh session
                                               #   (embeds sources — treat like source)
nikhail --list-rules                         # show the rule catalog with citations
nikhail . --disable S007                     # skip a rule, by id or name; repeatable,
                                               #   comma lists OK (--disable S007,S018)
nikhail . --kind mcp                         # lint only MCP tool JSON (or --kind skill)
nikhail . --advisory                         # + LLM-assisted advisory checks (see below)
nikhail . --advisory --fail-on advisory      # opt in to gating on advisory findings
```

Exit codes: `0` clean (below the `--fail-on` threshold), `1` findings at/above threshold, `2` usage error.
Lint an MCP server you run by dumping its `tools/list` response to JSON and passing that file.

### Run the tests

```bash
pytest -q          # full suite: rule behavior on fixtures + CLI exit codes + SARIF shape
```

### Test the GitHub Action locally (before GitHub ever sees it)

The testing ladder, cheapest first:

```bash
# 1. The CLI itself — this IS 95% of the Action (constant, instant)
nikhail fixtures/skills/good-skill --fail-on warning && echo PASS

# 2. The workflow wiring — act runs .github/workflows/ci.yml in Docker (occasional)
brew install act           # one-time; needs Docker running
act push -j tests          # pytest job locally
act push -j fixture-contract  # exit-code contract job locally
act push -j action-selftest   # composite action via 'uses: ./' — see act note below

# 3. Server-side rendering (PR annotations, code-scanning upload) — only on GitHub (rare)
#    Push to a scratch repo and reference the action by branch:
#    uses: prove-ai/Nikhail@main
```

`act` notes: the `.actrc` here selects `catthehacker/ubuntu:act-latest` (closer to GitHub's
real runner than act's slim default) and forces `linux/amd64` (needed on Apple Silicon).
The `tests` and `fixture-contract` jobs deliberately use the runner's system Python in a venv
instead of `actions/setup-python`, because setup-python's post-job cache step hits a known act
bug (`node` not found — nektos/act#107) that fails the job cosmetically. `action-selftest`
still uses setup-python (inside the composite action, as real consumers will), so under act
its main steps succeed but the post step may report that same cosmetic failure — its true
end-to-end check is the scratch-repo step below.
What act cannot emulate: PR annotations, the Checks UI, and SARIF upload to code scanning —
that's what the scratch repo is for. Validate SARIF locally first at
https://sarifweb.azurewebsites.net/Validation .

### Use the Action in another repo (once pushed to GitHub)

```yaml
- uses: actions/checkout@v4
- uses: prove-ai/Nikhail@v1
  with:
    path: .claude/skills
    fail-on: warning
```

---

## v1 rule catalog

**Skills (SKILL.md)** — S008 parseable frontmatter · S001 has description · S002 description
long enough to support activation · S003 description under the 1024-char limit · S004
kebab-case name matching the directory · S005 body within the progressive-disclosure budget
(500 lines) · S006 referenced files exist · S007 no orphan bundled files · S009 description
states *when* to use the skill · S010 name under the 64-char limit · S011 no XML/HTML tags
in the description · S012 forward-slash paths only · S013 body under 5,000 words · S014
description in third person · S015 name conveys the capability · S016 no rigid multi-command
sequences · S017 long workflows decomposed into steps · S018 substantial skills include an
example · S019 no time-anchored statements.

### Coverage of the 26 skill smells (arXiv:2607.01456, Table III)

The paper classifies only **5 of its 26 smells as statically detectable**; the other 21
required an LLM-based detector even for the authors. Nikhail v1 is deterministic, so:

- **Fully covered — the paper's entire static set (5/5):** Lengthy Skill Description → S003,
  Lengthy Skill Body → S013 (+ stricter S005 line budget), Lengthy Skill Name → S010,
  XML Included Description → S011, Backslash Path → S012.
- **Covered by conservative deterministic heuristics (7):** Confusing Skill Description →
  S001+S002+S009, Non Third Person Description → S014, Unclear Skill Name → S015 (+S004),
  Series of Commands → S016, Stepless Workflow → S017, Missing Example → S018, Time
  Sensitive Skill → S019.
- **Covered by the opt-in LLM advisory tier (14) — A001–A014:** Option Buffet, Missing
  Utility Script, Missing Decision Tree, No Validation Step, Execute Without a Plan,
  Never Asks Human, Rationalization Loophole, No Progress Tracking, Undelegated Detail,
  No Guardrails, Buried Gotchas, Missing Usage Rules, Missing Caveats, Missing Template.
  These are absence-based semantic judgments no deterministic rule can make; per the
  paper's own prevalence data they occur in 60–94% of real skills, so any
  keyword-absence approximation would be pure CI noise.

The full mapping also lives in the [skills.py](src/nikhail/rules/skills.py) module docstring.

### The advisory tier (v2, `--advisory`)

The 14 semantic smells are detected the way the paper's own SSD detector works: one
binary LLM classification per smell per file, each prompt carrying that smell's
definition plus a positive and a negative example. Because an LLM verdict is a
judgment, not a fact, the tier is engineered to be safe next to the deterministic rules:

- **Opt-in and non-gating.** Nothing runs without `--advisory`; findings carry the
  `advisory` severity, which ranks below `info` and never affects the exit code unless
  you explicitly pass `--fail-on advisory`. In SARIF they map to `note`.
- **Reproducible by construction.** Every verdict is cached keyed by (file content hash,
  smell, model, prompt version) under `~/.cache/nikhail/advisory` (`--cache-dir` /
  `NIKHAIL_CACHE_DIR`). Unchanged files are never re-evaluated, so CI re-runs are
  byte-identical cache hits; nondeterminism can only enter when a file actually changes.
- **Evidence-anchored.** When the model claims a smell is present because of specific
  text, it must quote it — and the quote is verified against the file. A hallucinated
  quote kills the finding mechanically. Absence-based verdicts (you can't quote an
  omission) are labeled `unanchored` in the message.
- **Cheap deterministic pre-gates.** Some classifiers only run when the question applies
  (e.g. *No Progress Tracking* requires an actual multi-step workflow).
- **Parallel.** The per-smell calls for a file run concurrently in batches of 10, so a
  cold file classifies in a few seconds; cached files are instant.
- **Model:** the latest Haiku (`claude-haiku-4-5` alias — tracks new Haiku snapshots;
  the resolved model is part of the cache key, so upgrades invalidate stale verdicts).
  Override with `--model` / `NIKHAIL_MODEL`. Credentials resolve however the Anthropic
  SDK normally does (`ANTHROPIC_API_KEY`, `ant auth login` profile, …).
- **Install:** `pip install 'nikhail[advisory]'`. **Eval:** `pytest -m live` runs both
  sides against the real API (skips cleanly without credentials): **recall** — every
  `fixtures/smells/` positive must be flagged by its mapped A-rule — and **precision** —
  every `fixtures/smells-clean/` counterpart (a skill containing the *mitigation* for
  its smell) must NOT be flagged by that rule. The eval re-earns all verdicts in a
  throwaway cache each run by design; that cost is eval-only, not CLI usage.

**MCP tools** — M001 has description · M002 description ≥ 10 words · M003 snake_case name ·
M004 has inputSchema · M005 parameters described · M006 parameters typed · M007 no
near-duplicate descriptions across tools (Jaccard overlap) · M008 description ≤ 300 words ·
M009 states when to use the tool · M010 states limitations/caveats · M011 not example-only
content · M012 more than one sentence · M013 states what the call returns or changes.

### Coverage of the six MCP description smells (arXiv:2602.14878)

The paper derives six smells from a six-component rubric (component score < 3 = smelly):
Unclear Purpose → M001 + M013 · Missing Usage Guidelines → M009 · Unstated Limitation →
M010 · Opaque Parameters → M005 + M006 · Exemplar Issues (examples-vs-description
balance) → M011 · Underspecified or Incomplete → M002 + M012. M009/M010/M012/M013 are
conservative cue heuristics at `info` severity (the paper's own scanner is FM-based);
M011 flags the confident degenerate case at `warning`.

All v1 rules are deterministic: no LLM calls, no network, stable output — safe to gate CI on.

## License

Versions 0.1.x are released under the [MIT License](LICENSE). Future versions
may be released under a different license.

## Roadmap status

v1 (deterministic rules) and the first slice of v2 (LLM-assisted advisory tier for the
14 semantic skill smells) are implemented. v3 (does fixing findings measurably help?)
has its first results: on a real 53-skill repo (garrytan/gbrain), applying description-level
findings improved skill-routing accuracy consistently across two router models, with the
gain concentrated on requests that don't use the skill's own vocabulary — see
[bench/README.md](bench/README.md) for the numbers and caveats. Still deliberately out of scope: security
scanning (delegate to Snyk agent-scan / Cisco skill-scanner instead of competing),
declared-vs-observed script consistency (v2), cross-skill composition analysis (v2),
cross-model selection benchmarking (v3), automatic rewriting with A/B measurement (v4).
