Metadata-Version: 2.3
Name: semlint
Version: 0.1.0
Summary: Find semantic redundancy in agent instruction corpora.
Author: Karlis Vagalis
Author-email: Karlis Vagalis <karlis.vagalis@gmail.com>
License: MIT
Requires-Dist: markdown-it-py[plugins]>=4.2.0
Requires-Dist: pydantic>=2.13.5
Requires-Dist: rich>=15.0.0
Requires-Dist: semhash>=0.4.1
Requires-Dist: typer>=0.27.2
Requires-Python: >=3.13
Description-Content-Type: text/markdown

# semlint

`semlint` is a small, local Python CLI for finding redundant information in agent instruction corpora: `SKILL.md`, `AGENTS.md`, prompts, rules, and other Markdown/text files.

It answers: **how many tokens are consuming context without adding unique information, and where are they?**

## Quick start

```bash
uv sync
uv run semlint ./skills
uv run semlint ./skills ./rules/AGENTS.md
uv run semlint './skills/**/*.md' './prompts/*.txt'
uv run semlint ./skills --threshold 0.90 --min-block-tokens 10
uv run semlint ./skills --split markdown
uv run semlint ./skills --output json > report.json
uv run semlint ./skills --limit 10
uv run semlint ./skills -s asc similarity
uv run semlint ./skills --fail-above 0.10
uv run semlint self list
```

The command exits with `1` when `--fail-above` is exceeded and `2` for invalid input/errors.

### Stronger semantic matching

SemHash with local Model2Vec embeddings is the default semantic engine. It loads its small CPU model on first run (and caches it locally). For an explicit alternate Model2Vec model, use:

```bash
uv run semlint ./skills --model minishlab/potion-base-8M --threshold 0.90
```

SemHash and Model2Vec run locally on CPU. No API, LLM, vector database, server, or GPU is needed.

## Agent capability discovery

`semlint self list` prints one full command synopsis and help description per line. Optional arguments and options are bracketed inline. The listing is generated by inspecting the registered Typer command definitions, so it stays aligned with the CLI.

## What it scans

- Accepts one or more file paths, directories, or glob patterns; recursively scans `.md` and `.txt` files and de-duplicates overlapping matches.
- By default (`--split auto`), uses Markdown block boundaries for `.md` files and blank-line paragraph splitting for other supported text files. Markdown blocks include paragraphs (including list and blockquote paragraphs), fenced/indented code, HTML blocks, and tables. Headings and front matter are not treated as instruction blocks.
- Use `--split paragraph` to force blank-line splitting, or `--split markdown` to force Markdown parsing.
- All modes preserve original source text and line ranges; none infer sections or rule IDs.
- Ignores blocks below `--min-block-tokens`.
- Clusters likely duplicates without modifying source files.

Similarity is intentionally configurable because generic phrases can be false positives. Review clusters before replacing instructions.

## Output formats

Text is the default and uses Rich-formatted summary metrics, cluster separators, cluster summaries, then a blank line followed by blank-line-separated `path:start-end` locations and numbered source lines. By default, clusters sort by estimated redundant tokens (largest first); `-s|--sort <asc|desc> <occurrences|similarity|estimated-savings>` selects a different order. `--limit N` displays only the first N clusters while keeping corpus-wide metrics unchanged. Linters commonly use this compact location notation with a source code frame; Rich provides readable terminal rendering without introducing a separate diagnostic framework.

Choose JSON with `--output json`. Top-level fields include `files_scanned`, `blocks_scanned`, `total_tokens`, `exact_duplicate_percentage`, `semantic_redundancy_percentage`, `estimated_unique_tokens`, `estimated_redundant_tokens`, `estimated_context_token_savings`, and `clusters`. Clusters include similarity, all source blocks, and token/savings estimates—no canonical or action suggestions.

The savings estimate assumes retaining the shortest occurrence in each cluster; review the source blocks yourself before deciding what to consolidate. `context_redundancy` is token-weighted:

```text
estimated_redundant_tokens / total_tokens
```

This is an estimate, not a tokenizer-specific bill. The initial version counts word tokens for transparent, consistent reporting.

## Development

`just build` uses the installed `doxxer` binary to inject the next Git-derived version into `pyproject.toml`, runs `uv build`, then restores the `0.0.0` source placeholder. Install `doxxer` before building.

```bash
just build
```

Install the project and development tools with `uv sync`, then run static checks:

```bash
uv sync
uv run ruff check src
uv run ty check src
```

The project uses Python 3.13+, Pydantic `BaseModel`s for domain and report models, and a modular `src/semlint` package (parser/scanner, SemHash clustering, metrics, reporting, and Typer CLI). Structured SemHash inputs and JSON report output are validated with Pydantic. It does not automatically edit files.
