Metadata-Version: 2.4
Name: agent-lexicon
Version: 0.6.0
Summary: A lightweight terminology memory layer for AI agents.
Project-URL: Homepage, https://github.com/SkeinRank/agent-lexicon
Project-URL: Repository, https://github.com/SkeinRank/agent-lexicon
Project-URL: Issues, https://github.com/SkeinRank/agent-lexicon/issues
Author: SkeinRank
License:                               Apache License
                                Version 2.0, January 2004
                             http://www.apache.org/licenses/
        
        TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
        
        1. Definitions.
        
           "License" shall mean the terms and conditions for use, reproduction,
           and distribution as defined by Sections 1 through 9 of this document.
        
           "Licensor" shall mean the copyright owner or entity authorized by
           the copyright owner that is granting the License.
        
           "Legal Entity" shall mean the union of the acting entity and all
           other entities that control, are controlled by, or are under common
           control with that entity. For the purposes of this definition,
           "control" means (i) the power, direct or indirect, to cause the
           direction or management of such entity, whether by contract or
           otherwise, or (ii) ownership of fifty percent (50%) or more of the
           outstanding shares, or (iii) beneficial ownership of such entity.
        
           "You" (or "Your") shall mean an individual or Legal Entity
           exercising permissions granted by this License.
        
           "Source" form shall mean the preferred form for making modifications,
           including but not limited to software source code, documentation
           source, and configuration files.
        
           "Object" form shall mean any form resulting from mechanical
           transformation or translation of a Source form, including but
           not limited to compiled object code, generated documentation,
           and conversions to other media types.
        
           "Work" shall mean the work of authorship, whether in Source or
           Object form, made available under the License, as indicated by a
           copyright notice that is included in or attached to the work
           (an example is provided in the Appendix below).
        
           "Derivative Works" shall mean any work, whether in Source or Object
           form, that is based on (or derived from) the Work and for which the
           editorial revisions, annotations, elaborations, or other modifications
           represent, as a whole, an original work of authorship. For the purposes
           of this License, Derivative Works shall not include works that remain
           separable from, or merely link (or bind by name) to the interfaces of,
           the Work and Derivative Works thereof.
        
           "Contribution" shall mean any work of authorship, including
           the original version of the Work and any modifications or additions
           to that Work or Derivative Works thereof, that is intentionally
           submitted to Licensor for inclusion in the Work by the copyright owner
           or by an individual or Legal Entity authorized to submit on behalf of
           the copyright owner. For the purposes of this definition, "submitted"
           means any form of electronic, verbal, or written communication sent
           to the Licensor or its representatives, including but not limited to
           communication on electronic mailing lists, source code control systems,
           and issue tracking systems that are managed by, or on behalf of, the
           Licensor for the purpose of discussing and improving the Work, but
           excluding communication that is conspicuously marked or otherwise
           designated in writing by the copyright owner as "Not a Contribution."
        
           "Contributor" shall mean Licensor and any individual or Legal Entity
           on behalf of whom a Contribution has been received by Licensor and
           subsequently incorporated within the Work.
        
        2. Grant of Copyright License. Subject to the terms and conditions of
           this License, each Contributor hereby grants to You a perpetual,
           worldwide, non-exclusive, no-charge, royalty-free, irrevocable
           copyright license to reproduce, prepare Derivative Works of,
           publicly display, publicly perform, sublicense, and distribute the
           Work and such Derivative Works in Source or Object form.
        
        3. Grant of Patent License. Subject to the terms and conditions of
           this License, each Contributor hereby grants to You a perpetual,
           worldwide, non-exclusive, no-charge, royalty-free, irrevocable
           (except as stated in this section) patent license to make, have made,
           use, offer to sell, sell, import, and otherwise transfer the Work,
           where such license applies only to those patent claims licensable
           by such Contributor that are necessarily infringed by their
           Contribution(s) alone or by combination of their Contribution(s)
           with the Work to which such Contribution(s) was submitted. If You
           institute patent litigation against any entity (including a
           cross-claim or counterclaim in a lawsuit) alleging that the Work
           or a Contribution incorporated within the Work constitutes direct
           or contributory patent infringement, then any patent licenses
           granted to You under this License for that Work shall terminate
           as of the date such litigation is filed.
        
        4. Redistribution. You may reproduce and distribute copies of the
           Work or Derivative Works thereof in any medium, with or without
           modifications, and in Source or Object form, provided that You
           meet the following conditions:
        
           (a) You must give any other recipients of the Work or
               Derivative Works a copy of this License; and
        
           (b) You must cause any modified files to carry prominent notices
               stating that You changed the files; and
        
           (c) You must retain, in the Source form of any Derivative Works
               that You distribute, all copyright, patent, trademark, and
               attribution notices from the Source form of the Work,
               excluding those notices that do not pertain to any part of
               the Derivative Works; and
        
           (d) If the Work includes a "NOTICE" text file as part of its
               distribution, then any Derivative Works that You distribute must
               include a readable copy of the attribution notices contained
               within such NOTICE file, excluding those notices that do not
               pertain to any part of the Derivative Works, in at least one
               of the following places: within a NOTICE text file distributed
               as part of the Derivative Works; within the Source form or
               documentation, if provided along with the Derivative Works; or,
               within a display generated by the Derivative Works, if and
               wherever such third-party notices normally appear. The contents
               of the NOTICE file are for informational purposes only and
               do not modify the License. You may add Your own attribution
               notices within Derivative Works that You distribute, alongside
               or as an addendum to the NOTICE text from the Work, provided
               that such additional attribution notices cannot be construed
               as modifying the License.
        
           You may add Your own copyright statement to Your modifications and
           may provide additional or different license terms and conditions
           for use, reproduction, or distribution of Your modifications, or
           for any such Derivative Works as a whole, provided Your use,
           reproduction, and distribution of the Work otherwise complies with
           the conditions stated in this License.
        
        5. Submission of Contributions. Unless You explicitly state otherwise,
           any Contribution intentionally submitted for inclusion in the Work
           by You to the Licensor shall be under the terms and conditions of
           this License, without any additional terms or conditions.
           Notwithstanding the above, nothing herein shall supersede or modify
           the terms of any separate license agreement you may have executed
           with Licensor regarding such Contributions.
        
        6. Trademarks. This License does not grant permission to use the trade
           names, trademarks, service marks, or product names of the Licensor,
           except as required for reasonable and customary use in describing the
           origin of the Work and reproducing the content of the NOTICE file.
        
        7. Disclaimer of Warranty. Unless required by applicable law or
           agreed to in writing, Licensor provides the Work (and each
           Contributor provides its Contributions) on an "AS IS" BASIS,
           WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
           implied, including, without limitation, any warranties or conditions
           of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
           PARTICULAR PURPOSE. You are solely responsible for determining the
           appropriateness of using or redistributing the Work and assume any
           risks associated with Your exercise of permissions under this License.
        
        8. Limitation of Liability. In no event and under no legal theory,
           whether in tort (including negligence), contract, or otherwise,
           unless required by applicable law (such as deliberate and grossly
           negligent acts) or agreed to in writing, shall any Contributor be
           liable to You for damages, including any direct, indirect, special,
           incidental, or consequential damages of any character arising as a
           result of this License or out of the use or inability to use the
           Work (including but not limited to damages for loss of goodwill,
           work stoppage, computer failure or malfunction, or any and all
           other commercial damages or losses), even if such Contributor
           has been advised of the possibility of such damages.
        
        9. Accepting Warranty or Additional Liability. While redistributing
           the Work or Derivative Works thereof, You may choose to offer,
           and charge a fee for, acceptance of support, warranty, indemnity,
           or other liability obligations and/or rights consistent with this
           License. However, in accepting such obligations, You may act only
           on Your own behalf and on Your sole responsibility, not on behalf
           of any other Contributor, and only if You agree to indemnify,
           defend, and hold each Contributor harmless for any liability
           incurred by, or claims asserted against, such Contributor by reason
           of your accepting any such warranty or additional liability.
        
        END OF TERMS AND CONDITIONS
        
        APPENDIX: How to apply the Apache License to your work.
        
           To apply the Apache License to your work, attach the following
           boilerplate notice, with the fields enclosed by brackets "[]"
           replaced with your own identifying information. (Don't include
           the brackets!)  The text should be enclosed in the appropriate
           comment syntax for the file format. We also recommend that a
           file or class name and description of purpose be included on the
           same "printed page" as the copyright notice for easier
           identification within third-party archives.
        
        Copyright [yyyy] [name of copyright owner]
        
        Licensed under the Apache License, Version 2.0 (the "License");
        you may not use this file except in compliance with the License.
        You may obtain a copy of the License at
        
            http://www.apache.org/licenses/LICENSE-2.0
        
        Unless required by applicable law or agreed to in writing, software
        distributed under the License is distributed on an "AS IS" BASIS,
        WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        See the License for the specific language governing permissions and
        limitations under the License.
License-File: LICENSE
Keywords: ai-agents,governance,lexicon,rag,terminology,tool-use
Classifier: Development Status :: 1 - Planning
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: oov
Requires-Dist: huggingface-hub<1.0,>=0.20; extra == 'oov'
Requires-Dist: tokenizers<1.0,>=0.15; extra == 'oov'
Provides-Extra: semantic
Requires-Dist: sentence-transformers<4.0,>=2.6; extra == 'semantic'
Description-Content-Type: text/markdown

# Agent Lexicon

Shared terminology memory for AI agents.

Agent Lexicon is a lightweight Python package for giving agents a small,
reviewable terminology layer before RAG, tool calls, and workflow automation.

## Install

```bash
pip install agent-lexicon
```

Optional tokenizer-backed OOV scoring for Scout quality ranking can be installed
when you want to compare candidate surfaces against a real tokenizer:

```bash
pip install "agent-lexicon[oov]"
```

## Quick start

Use the product-facing commands for the common local loop:

```bash
agent-lexicon init
agent-lexicon scan README.md docs src
agent-lexicon scan README.md docs src --quality-report
agent-lexicon scan README.md docs src --oov-tokenizer auto
agent-lexicon analyze --priority important --quality-report
agent-lexicon review
agent-lexicon publish
agent-lexicon mcp serve --root . --lexicon lexicon/lexicon.yaml
```

`scan` reads local project files, runs prompt-safety checks, discovers candidate
terms, builds evidence packs, computes candidate-quality signals, and saves the
result to `.agent-lexicon/`. `--quality-report` shows how many candidates were
classified as Important/Later, how much evidence coverage exists, and which
signals drove the prioritization. `analyze` summarizes the highest-priority
candidates so reviewers can start with the important terminology first.

## Quick check

```bash
agent-lexicon --version
python -m agent_lexicon --version
agent-lexicon init --root /tmp/agent-lexicon-demo
agent-lexicon scan examples/customer_limits/docs --root examples/customer_limits --max-candidates 5
agent-lexicon analyze --root examples/customer_limits
agent-lexicon publish --root examples/customer_limits
agent-lexicon validate examples/customer_limits/lexicon.yaml
agent-lexicon lint examples/customer_limits/lexicon.yaml
agent-lexicon validate examples/customer_limits/lexicon.yaml --lint
agent-lexicon match examples/customer_limits/lexicon.yaml "The customer cap and rate limit changed." --longest-only
agent-lexicon resolve examples/customer_limits/lexicon.yaml "increase the limit"
agent-lexicon guard examples/customer_limits/lexicon.yaml "increase the limit" --tool api.update_rate_limit
agent-lexicon guard examples/customer_limits/lexicon.yaml "increase the limit" --tool api.update_rate_limit --allow-risky-unicode
agent-lexicon validate-queries examples/customer_limits/queries.jsonl
agent-lexicon check examples/customer_limits/lexicon.yaml examples/customer_limits/queries.jsonl
agent-lexicon ingest README.md src examples/customer_limits/docs --root .
agent-lexicon discover-candidates examples/customer_limits/docs --root examples/customer_limits --quality-report
agent-lexicon build-evidence examples/customer_limits/docs --root examples/customer_limits
agent-lexicon safety scan examples/customer_limits/docs --root examples/customer_limits
agent-lexicon policy status --root examples/customer_limits
agent-lexicon workspace init --root examples/customer_limits
agent-lexicon workspace sync examples/customer_limits/docs --root examples/customer_limits --max-candidates 5
agent-lexicon workspace status --root examples/customer_limits
agent-lexicon review --root examples/customer_limits
agent-lexicon workspace export-review-events --root examples/customer_limits
agent-lexicon discover-migrations examples/customer_limits/lexicon.yaml
agent-lexicon dictionary init --root .
agent-lexicon dictionary validate --root .
agent-lexicon dictionary validate --root . --lint
agent-lexicon dictionary diff lexicon/lexicon.yaml lexicon-next.yaml
agent-lexicon dictionary merge lexicon-base.yaml lexicon-ours.yaml lexicon-theirs.yaml --output lexicon-merged.json
agent-lexicon check-merge --root . --base main --head HEAD
agent-lexicon check-merge --root . --base main --head HEAD --fail-on-review
agent-lexicon dictionary pr-check --root .
agent-lexicon review-agent assess --root examples/customer_limits
agent-lexicon review-agent consensus --root examples/customer_limits
agent-lexicon review-agent prompt --root examples/customer_limits
agent-lexicon review-agent dataset --root examples/customer_limits
agent-lexicon mcp tools
agent-lexicon mcp serve --root . --lexicon lexicon/lexicon.yaml
```

## Simple local workflow

The short commands are wrappers around the lower-level SDK and CLI surfaces.
They keep the first-run workflow small without hiding the underlying building
blocks.

```bash
agent-lexicon init
```

Creates the git-tracked `lexicon/` layout, the local SQLite workspace under
`.agent-lexicon/`, and a local policy file.

```bash
agent-lexicon scan README.md docs src
```

Reads local docs and source files, filters existing lexicon surfaces, scores
terminology candidates, builds positive/negative evidence, runs prompt-safety
checks, adds OOV-proxy and clustering metadata, and stores the result in the
local workspace. Add `--quality-report` to see candidate reduction, clusters,
OOV/code-style counts, evidence coverage, and top priority reasons.

```bash
agent-lexicon analyze --review-agent --consensus
```

Shows important candidates first and can include deterministic Review Agent
recommendations for quick triage. Use `--consensus` to show the consensus and
abstention wrapper used for safer auto-triage decisions. Use `--priority
important` to focus the inbox on surfaces that look risky, internal, clustered,
or likely to affect agent behavior. Add `--quality-report` to print the same
Scout metrics from the stored workspace.

```bash
agent-lexicon publish
```

Publishes accepted review decisions into a lexicon-compatible local snapshot.

## Candidate quality

Local Scout attaches quality signals to each discovered candidate. By default,
the quality layer is dependency-free. For projects that want a tokenizer-backed
OOV signal, install `agent-lexicon[oov]` and run `scan` or `discover-candidates`
with `--oov-tokenizer auto`. `auto` uses `BAAI/bge-small-en-v1.5`; a specific
Hugging Face tokenizer id can also be passed explicitly. Tokenizer scoring is
only used during offline Scout analysis, not during runtime resolve or tool
guard calls.

- `oov_proxy_score` estimates tokenizer pain from code-style shape, separators,
  camel case, acronyms, and digits.
- `oov_score` is the effective OOV score used for priority ranking. It is the
  proxy score by default and can include the optional tokenizer signal.
- `oov_source` shows whether the score came from `proxy`, `tokenizer`, or
  `proxy_fallback`.
- `token_fragmentation_score` highlights surfaces that are likely to split into
  many tokens.
- `surface_risk_score` combines shape, OOV proxy, cluster size, and negative
  evidence signals.
- `cluster_key` groups variants such as `PaymentCore`, `payment-core`, and
  `payments_core` before review.
- `priority` separates candidates into `important` and `later` review buckets.

This keeps the web inbox and `agent-lexicon analyze` focused on the candidates
that are most likely to matter for agent behavior and retrieval quality. The
quality report makes this visible as product metrics: Important/Later counts,
review reduction, cluster coverage, high-OOV candidates, code-style candidates,
negative-evidence coverage, and top priority reasons.

```bash
agent-lexicon scan README.md docs src --quality-report
agent-lexicon scan README.md docs src --oov-tokenizer auto --quality-report
agent-lexicon analyze --priority important --quality-report
```

## Core schema

The package includes dependency-free core models for terminology workflows:

- `Scope` — a project, team, domain, or workflow boundary where a term has a specific meaning.
- `Term` — a canonical domain term with aliases, scopes, tags, evidence, and metadata.
- `Alias` — a surface form that points to a canonical term.
- `EvidenceSpan` — a source-backed snippet with file path, line range, and evidence kind.
- `ProposalCandidate` — a reviewable terminology change suggested by local analysis or an agent.
- `Lexicon` — a validated terminology document containing scopes, terms, proposals, and metadata.

Example:

```python
from agent_lexicon import Alias, EvidenceSpan, ProposalCandidate, ProposalKind, Term

term = Term(
    id="billing.credit_limit",
    canonical="credit limit",
    aliases=(Alias(surface="customer cap", term_id="billing.credit_limit"),),
    scopes=("billing",),
)

evidence = EvidenceSpan(
    source_path="docs/billing.md",
    start_line=42,
    snippet="Customer cap is the credit limit for an account.",
)

proposal = ProposalCandidate(
    id="proposal.customer-cap.alias",
    kind=ProposalKind.ALIAS_CANDIDATE,
    surface="customer cap",
    candidate_term_id="billing.credit_limit",
    confidence=0.78,
    evidence=(evidence,),
)
```

## Lexicon documents

Agent Lexicon can load and validate JSON or YAML terminology documents. The
current document format is intentionally small and local-first:

```yaml
version: 1
scopes:
  - id: billing
    label: Billing
terms:
  - id: billing.credit_limit
    canonical: credit limit
    scopes: [billing]
    tools: [billing.update_credit_limit]
    aliases:
      - surface: customer cap
        scopes: [billing]
    evidence:
      - source_path: docs/billing.md
        start_line: 12
        snippet: Customer cap is the credit limit for an account.
        kind: positive
```

Validate a document from the command line:

```bash
agent-lexicon validate examples/customer_limits/lexicon.yaml
agent-lexicon lint examples/customer_limits/lexicon.yaml
agent-lexicon validate examples/customer_limits/lexicon.yaml --lint
agent-lexicon match examples/customer_limits/lexicon.yaml "The customer cap and rate limit changed." --longest-only
agent-lexicon resolve examples/customer_limits/lexicon.yaml "increase the limit"
agent-lexicon guard examples/customer_limits/lexicon.yaml "increase the limit" --tool api.update_rate_limit
```

Load the same document from Python:

```python
from agent_lexicon import Lexicon, load_lexicon

lexicon = load_lexicon("examples/customer_limits/lexicon.yaml")
assert lexicon.get_term("billing.credit_limit") is not None

lexicon_again = Lexicon.from_file("examples/customer_limits/lexicon.json")
```

The loader validates duplicate ids, unknown scope references, alias collisions,
and proposal references before returning a `Lexicon` object.

## Lexicon linting

Structural validation answers whether a document can be loaded. Lexicon linting
answers whether a valid document is likely to behave well in team usage, guard
routing, and agent-generated code reviews. It reports warnings for surfaces that
are technically valid but risky in real project text.

```bash
agent-lexicon lint examples/customer_limits/lexicon.yaml
agent-lexicon lint examples/customer_limits/lexicon.yaml --strict
agent-lexicon validate examples/customer_limits/lexicon.yaml --lint
agent-lexicon dictionary validate --root . --lint
```

The linter checks for:

- missing explicit `version: 1` in the source document,
- broad one-word surfaces such as `token`, `key`, `id`, `url`, `api`, `data`,
  `system`, `error`, and `limit`,
- broad deprecated aliases that can keep matching active terminology,
- broad surfaces on terms that route tools through `ToolGuard`,
- runtime-normalized surface collisions after Unicode normalization and
  generated code-style variants.

By default lint warnings do not fail the command, which makes the linter useful
for local review. Use `--strict` or `--strict-lint` when warnings should fail CI.
For automation, add `--json` to `agent-lexicon lint`.


## Surface matching

Agent Lexicon can scan text for canonical terms and aliases from a loaded
lexicon. The matcher is dependency-free and uses a trie with Aho-Corasick
failure links, so it can be used by runtime agents before retrieval, tool calls,
or local review workflows.

```python
from agent_lexicon import build_surface_matcher, load_lexicon

lexicon = load_lexicon("examples/customer_limits/lexicon.yaml")
matcher = build_surface_matcher(lexicon)

matches = matcher.match(
    "The customer cap and rate limit changed.",
    longest_only=True,
)

for match in matches:
    print(match.term_id, match.kind.value, match.matched_text)
```

Command line usage:

```bash
agent-lexicon match examples/customer_limits/lexicon.yaml "The customer cap and rate limit changed."
```

For code-heavy projects, natural surfaces also cover common identifier shapes:

```python
from agent_lexicon import Lexicon, Term, resolve_text

lexicon = Lexicon(
    terms=(Term(id="auth.access_token", canonical="access token"),)
)

decision = resolve_text(lexicon, "rotate self.access_token before using accessToken")
print(decision.primary_term_id)  # auth.access_token
```

The matcher supports scope filtering, case-sensitive aliases, deprecated surface
filtering, generated code-identifier variants, Unicode-normalized input, and
longest non-overlapping output for downstream resolver logic. A lexicon surface
such as `access token` can also match common code forms such as `accessToken`,
`access_token`, and `ACCESS_TOKEN` without adding those aliases manually.
Explicit aliases remain reserved, so case-sensitive surfaces such as `API_KEY`
keep their intended behavior.

## Unicode normalization policy

Runtime matching normalizes common Unicode forms before resolver and guard
checks. This handles text copied from chats, PDFs, web pages, and tool output
without putting a tokenizer or model on the hot path.

The policy is conservative:

- fullwidth and compatibility forms are normalized with per-character NFKC;
- non-ASCII spaces and zero-width separators are treated as plain spaces;
- bidi-control characters are removed and reported as high-risk findings;
- accents and real letters are not folded to ASCII.

Resolution decisions include `unicode_*` metadata when input normalization
changed the text. Tool guard blocks by default when bidi-control characters are
found, because visually reordered text can make the guard apply rules to a
different surface than the user sees. Use `--allow-risky-unicode` only when the
caller has already accepted that input risk.

```bash
agent-lexicon resolve examples/customer_limits/lexicon.yaml $'increase the customer\u00a0cap'
agent-lexicon guard examples/customer_limits/lexicon.yaml $'increase the customer\u202ecap' --tool billing.update_credit_limit
agent-lexicon guard examples/customer_limits/lexicon.yaml $'increase the customer\u202ecap' --tool billing.update_credit_limit --allow-risky-unicode
```


## Runtime resolution

The resolver turns surface matches into a deterministic runtime decision. It
prefers longer non-overlapping surfaces, preserves same-span ambiguity, handles
code identifiers such as `accessToken` and `partition_key`, and returns one of
three statuses: `resolved`, `ambiguous`, or `unknown`. The overlap selector uses
a bitmap-backed linear pass over matched spans instead of repeatedly scanning
all accepted intervals, keeping large-file resolution predictable for local
agent workflows.

```python
from agent_lexicon import load_lexicon, resolve_text

lexicon = load_lexicon("examples/customer_limits/lexicon.yaml")

decision = resolve_text(lexicon, "increase the limit")
print(decision.status.value)  # ambiguous
print(decision.action.value)  # ask_clarification

billing_decision = resolve_text(
    lexicon,
    "increase the limit",
    scopes=("billing",),
)
print(billing_decision.primary_term_id)  # billing.credit_limit
```

Command line usage:

```bash
agent-lexicon resolve examples/customer_limits/lexicon.yaml "increase the limit"
agent-lexicon guard examples/customer_limits/lexicon.yaml "increase the limit" --tool api.update_rate_limit
agent-lexicon resolve examples/customer_limits/lexicon.yaml "increase the limit" --scope billing
```

This gives agents a local way to stop before unsafe assumptions: if the same
surface can mean multiple canonical terms, the recommended action is
`ask_clarification`.


## Runtime cache and scale behavior

Long-running agent processes can reuse compiled runtime objects instead of
rebuilding the matcher for every request. The default helpers keep a small
thread-safe in-process cache keyed by a stable lexicon fingerprint and the
`include_deprecated` option. This is most useful for MCP servers, local web
review processes, and agent runners that resolve many texts against the same
lexicon snapshot.

```python
from agent_lexicon import (
    clear_runtime_cache,
    fingerprint_lexicon,
    get_cached_resolver,
    load_cached_lexicon,
    runtime_cache_stats,
)

lexicon = load_cached_lexicon("lexicon/lexicon.yaml")
print(fingerprint_lexicon(lexicon).value)

resolver = get_cached_resolver(lexicon)
print(resolver.resolve("increase the customer cap").primary_term_id)

print(runtime_cache_stats().to_dict())
clear_runtime_cache()
```

`resolve_text(...)` and `guard_tool_call(...)` use the process-wide cache by
default. Direct constructors such as `LexiconResolver.from_lexicon(...)` remain
available when a caller wants an isolated resolver instance. File-based cache
entries include path, size, and modification time, so `load_cached_lexicon(...)`
reloads after the lexicon file changes.

## Near-miss review hints

When a code-style identifier is unknown, the resolver can attach deterministic
near-miss metadata for review. This does not change the runtime decision from
`unknown`; it only helps reviewers decide whether the unknown surface should
be proposed as a new alias for an existing canonical term.

```python
from agent_lexicon import Lexicon, Term, resolve_text

lexicon = Lexicon(
    terms=(Term(id="auth.access_token", canonical="access token"),)
)

decision = resolve_text(lexicon, "rotate authToken")
print(decision.status.value)  # unknown
print(decision.metadata["near_miss_suggestions"][0]["suggestions"][0]["target_term_id"])
# auth.access_token
```

Near-miss scoring is dependency-free and uses fragments, edit similarity,
identifier shape, shared suffixes or prefixes, and a small deterministic set of
related domain fragments. It is designed for merge and review workflows where
agent-generated branches may introduce surfaces such as `authToken` or
`sessionKey` that look close to known terms but should still require a human
review decision.

The scorer is intentionally conservative with weak single-fragment bridges.
Broad fragments such as `key`, `id`, `token`, or `access` are not enough on
their own when the rest of the identifier is not a close typo and no related
fragment bridge exists. This keeps review output focused: `authToken` can still
point to `access token` through the deterministic `auth` → `access` relation,
while unrelated surfaces such as `sessionKey` are not pulled toward
`partition key` just because both share `key`. Precision dampening is recorded
in suggestion metadata for audit-friendly review.

The lower-level API is also available when a workflow already has a candidate
surface:

```python
from agent_lexicon import suggest_near_misses

report = suggest_near_misses(lexicon, "authToken", max_suggestions=3)
for suggestion in report.suggestions:
    print(suggestion.target_term_id, suggestion.confidence, suggestion.reasons)
```



## Semantic escalation interface

Near-miss hints stay deterministic by default. Each heuristic suggestion now
records `metadata["suggestion_source"] == "heuristic"` and a
`metadata["semantic_escalation"]` object that marks whether the item is a good
candidate for an optional semantic reranker. The built-in backend is a no-op,
so no model weights are loaded and runtime `resolve` / `guard` decisions remain
unchanged.

This gives Scout workflows a cheap-first cascade:

1. deterministic fragments, edit distance, and code-shape scoring run first;
2. confident items can be reviewed directly;
3. gray-zone or weak-bridge items are marked for an optional semantic backend.

```python
from agent_lexicon import Lexicon, Term, suggest_near_misses

lexicon = Lexicon(
    terms=(Term(id="auth.access_token", canonical="access token"),)
)

report = suggest_near_misses(lexicon, "authToken")
suggestion = report.suggestions[0]
print(suggestion.metadata["suggestion_source"])
# heuristic
print(suggestion.metadata["semantic_escalation"]["recommended"])
# True
```

Use this metadata when a review pipeline wants to route only uncertain
near-miss items to a heavier semantic scorer. The default package does not ship
a semantic model and does not call external services. Semantic backends implement
the `SemanticNearMissBackend` protocol and keep their output clearly labeled as
`semantic`, separate from the deterministic heuristic source.

## Optional BGE semantic near-miss

Scout workflows can opt into a BGE reranker for the gray zone where lexical
near-miss signals are useful but not decisive. This is an offline review aid:
`resolve` still returns `unknown`, guard decisions remain unchanged, and the
default package stays dependency-free.

Install the semantic extra in the environment where semantic review runs:

```bash
pip install "agent-lexicon[semantic]"
```

For local repository development with Poetry:

```bash
poetry install -E semantic
```

Command line usage:

```bash
agent-lexicon resolve lexicon/lexicon.yaml "rotate authToken" --semantic-near-miss
agent-lexicon check-merge --root . --base main --head HEAD --semantic-near-miss
agent-lexicon check-merge --root . --base main --head HEAD --semantic-near-miss --semantic-threshold 0.45
```

The default model is `BAAI/bge-base-en-v1.5`. Use `--semantic-model` when a
local cache or another compatible Sentence Transformers model should be used.
The backend calls the model only for near-miss candidates already marked by the
cheap deterministic gate, then reranks or filters those review hints. Suggestions
that pass through this mode include audit metadata such as:

```text
suggestion_source=semantic
semantic_backend=bge-base
semantic_score=0.812
```

Python usage:

```python
from agent_lexicon import BgeSemanticNearMissBackend, Lexicon, Term, suggest_near_misses

lexicon = Lexicon(terms=(Term(id="auth.access_token", canonical="access token"),))
backend = BgeSemanticNearMissBackend()

report = suggest_near_misses(
    lexicon,
    "authToken",
    semantic_backend=backend,
)
```

Keep this mode out of hot-path guard enforcement. It is best used in Scout,
merge review, and proposal preparation where semantic quality is worth the
extra model cost and the output is reviewed by a person or policy workflow.

## Git merge terminology check

Agent-generated branches often introduce new code identifiers before reviewers
notice the terminology drift. The merge terminology check scans added git diff
lines between two refs, reports known terminology as safe, and separates unknown
code-style identifiers that may need reviewer attention.

```bash
agent-lexicon check-merge --root . --base main --head HEAD
agent-lexicon check-merge --root . --base main --head HEAD --include 'src/**'
agent-lexicon check-merge --root . --base main --head HEAD --json
agent-lexicon check-merge --root . --base main --head HEAD --include-unresolved-unknowns
agent-lexicon check-merge --root . --base main --head HEAD --semantic-near-miss
agent-lexicon check-merge --root . --base main --head HEAD --fail-on-review
```

The command defaults to `lexicon/lexicon.yaml` under `--root`. Use `--lexicon`
when the dictionary lives elsewhere. The comparison uses the pull-request style
`base...head` git range, so the result focuses on what the branch adds relative
to the merge base.

Example output:

```text
Git merge terminology check: 1 files, 4 added lines
Range: main...HEAD
Summary: known=1, likely_alias=1, likely_new_term=1, unresolved_unknown=0, hidden_unresolved=1
Known terminology:
- src/auth.py:12 'accessToken' -> auth.access_token (access token)
Needs review:
Likely aliases:
- src/auth.py:13 'authToken' unknown; near miss: auth.access_token (access token) confidence=0.623 via 'AccessToken' semantic_escalation=related_fragment_bridge
New terminology candidates:
- src/auth.py:14 'credentialBlob' unknown; possible new term
Hidden unresolved identifiers: 1. Use --include-unresolved-unknowns to inspect low-signal identifiers.
```

Known occurrences are useful for confirming that different branch naming styles
still resolve to the same canonical term. Unknown identifiers are now separated
into two review classes. `likely_alias` items have deterministic near-miss hints
that point to an existing canonical term. `likely_new_term` items have no strong
canonical neighbor, but still look terminology-relevant enough to review as a
possible new lexicon term. Both classes keep the runtime decision unchanged: the
identifier remains unknown until a reviewer adds an alias or a new term.

Low-signal local identifiers such as helper function names, one-letter suffixes,
and temporary variables stay out of the default report. The default output still
shows `hidden_unresolved` and a short note when those items were suppressed, so a
quiet report does not hide that a full-audit basket exists. Use
`--include-unresolved-unknowns` when a workflow needs the full unknown identifier
list, including low-signal items. Use `--fail-on-review` in CI when alias or
new-term review items should block the workflow.

Python usage:

```python
from agent_lexicon import check_git_merge_terminology, load_lexicon

lexicon = load_lexicon("lexicon/lexicon.yaml")
report = check_git_merge_terminology(
    lexicon,
    root=".",
    base="main",
    head="HEAD",
)

for item in report.needs_review:
    print(item.surface, item.suggestions[0].target_term_id)
```

## Tool-call safety

Agent Lexicon can check a requested tool call before the agent executes it. If
terminology is ambiguous, the guard asks for clarification instead of allowing a
risky tool call. If a term is resolved and declares allowed tools, the requested
tool must match that term's tool list.

```python
from agent_lexicon import guard_tool_call, load_lexicon

lexicon = load_lexicon("examples/customer_limits/lexicon.yaml")

decision = guard_tool_call(
    lexicon,
    "increase the limit",
    tool_name="api.update_rate_limit",
)

print(decision.status.value)  # needs_clarification
print(decision.action.value)  # ask_clarification
print(decision.is_allowed)    # False
```

Command line usage:

```bash
agent-lexicon guard examples/customer_limits/lexicon.yaml "increase the limit" --tool api.update_rate_limit
agent-lexicon guard examples/customer_limits/lexicon.yaml "increase the limit" --tool billing.update_credit_limit --scope billing
agent-lexicon guard examples/customer_limits/lexicon.yaml $'increase the customer\u202ecap' --tool billing.update_credit_limit
```

The `guard` command returns `0` for allowed or no-match decisions and `2` when
the tool call is blocked or needs clarification. Bidi-control Unicode findings
are blocked by default and can be inspected in the printed metadata summary.
This makes the guard usable in local agent wrappers and future CI checks.



## Local ingest

Agent Lexicon can read local project files into deterministic text documents for
future scout, evidence, and review workflows. Directory scans use local-project
defaults: README files, `docs/`, `src/`, Markdown, JSON/YAML, TOML, and common
text/code files. Large files, binary files, virtual environments, build outputs,
and cache directories are skipped.

Command line usage:

```bash
agent-lexicon ingest README.md src examples/customer_limits/docs --root .
agent-lexicon discover-candidates examples/customer_limits/docs --root examples/customer_limits --quality-report
agent-lexicon build-evidence examples/customer_limits/docs --root examples/customer_limits
agent-lexicon ingest examples/customer_limits/docs --root examples/customer_limits --jsonl
```

Python usage:

```python
from agent_lexicon import ingest_local_paths

report = ingest_local_paths(["README.md", "src", "examples/customer_limits/docs"], root=".")

for document in report.documents:
    print(document.relative_path, document.kind.value, document.line_count)
```

The ingest report exposes `document_count`, `total_lines`, `total_size_bytes`,
`documents`, and `skipped_paths`. Each document includes a stable SHA-256 hash,
relative path, source kind, line count, byte size, and text content.

## Candidate discovery

Agent Lexicon can run a deterministic local scout pass over ingested documents.
The scout discovers reviewable terminology candidates, assigns a score, reports
a jargon score, and applies background penalties so common project words do not
dominate the candidate list. This step is local-first and dependency-free.

Command line usage:

```bash
agent-lexicon discover-candidates examples/customer_limits/docs --root examples/customer_limits --quality-report
agent-lexicon discover-candidates examples/customer_limits/docs --root examples/customer_limits --lexicon examples/customer_limits/lexicon.yaml --json
```

Python usage:

```python
from agent_lexicon import discover_scout_candidates, ingest_local_paths

ingest_report = ingest_local_paths(["examples/customer_limits/docs"], root="examples/customer_limits")
candidate_report = discover_scout_candidates(ingest_report.documents)

for candidate in candidate_report.candidates:
    print(candidate.surface, candidate.score, candidate.jargon_score, candidate.background_penalty)
```

Each candidate includes a surface, normalized surface, kind, score, jargon
score, background penalty, occurrence count, document count, source occurrences,
and a deterministic score breakdown. Existing lexicon surfaces can be filtered
out with `existing_surfaces_from_lexicon(...)` or the CLI `--lexicon` option.

## Evidence packs

Evidence packs turn discovered candidates into reviewable snippets with file
paths and line numbers. Positive snippets show exact candidate occurrences.
Negative snippets show partial token overlap without the exact surface, which
helps reviewers spot broad, overloaded, or weak terminology candidates.

Command line usage:

```bash
agent-lexicon build-evidence examples/customer_limits/docs --root examples/customer_limits
agent-lexicon build-evidence examples/customer_limits/docs --root examples/customer_limits --json
agent-lexicon build-evidence examples/customer_limits/docs --root examples/customer_limits --jsonl
```

Python usage:

```python
from agent_lexicon import build_evidence_packs, discover_scout_candidates, ingest_local_paths

ingest_report = ingest_local_paths(["examples/customer_limits/docs"], root="examples/customer_limits")
candidate_report = discover_scout_candidates(ingest_report.documents)
evidence_report = build_evidence_packs(ingest_report.documents, candidate_report.candidates)

for pack in evidence_report.packs:
    print(pack.surface, pack.positive_count, pack.negative_count)
```

Each pack includes the candidate surface, score, positive snippets, negative
snippets, line ranges, reasons, and source metadata. This is the local evidence
foundation for proposal review and future snapshot publishing.


## Prompt safety

Agent Lexicon treats local docs and evidence snippets as untrusted input before
they are shown to a future LLM reviewer. The prompt-safety scanner is a
deterministic review aid: it flags instruction-like content such as attempts to
override instructions, reveal system prompts, exfiltrate secrets, or force tool
execution. It is intentionally not described as a complete prompt-injection
security boundary; motivated adversarial text can still require human review and
additional controls.

The scanner checks original lines, Unicode-normalized lines, and a joined
normalized window. This makes simple zero-width, fullwidth, compatibility-form,
and line-split evasions visible in review metadata without adding ML
dependencies. Evidence packs are annotated with prompt-safety metadata by
default, so review workflows can block high-risk snippets before they are sent to
an LLM.

Command line usage:

```bash
agent-lexicon safety scan examples/customer_limits/docs --root examples/customer_limits
agent-lexicon safety scan examples/customer_limits/docs --root examples/customer_limits --json
agent-lexicon safety scan examples/customer_limits/docs --root examples/customer_limits --fail-on-high-risk
agent-lexicon build-evidence examples/customer_limits/docs --root examples/customer_limits
```

Python usage:

```python
from agent_lexicon import (
    format_evidence_pack_for_llm_review,
    ingest_local_paths,
    scan_documents_for_prompt_injection,
)

ingest_report = ingest_local_paths(["examples/customer_limits/docs"], root="examples/customer_limits")
safety_report = scan_documents_for_prompt_injection(ingest_report.documents)

print(safety_report.highest_risk.value, safety_report.action.value)
```

The helper `format_evidence_pack_for_llm_review(...)` renders evidence as
data-only context with explicit untrusted boundaries. This keeps future review
agents from accidentally following instructions embedded inside project docs.
Prompt-safety findings include a `scan_scope` field (`line`, `normalized_line`,
or `joined_window`) so reviewers can see whether a match came from the original
text, Unicode-normalized text, or a multi-line joined window.


## Local policy modes

Agent Lexicon includes a small local policy layer for review workflows that need
more structure than a single-user scratchpad but do not require an enterprise
RBAC service. The policy is intentionally declarative: the CLI caller declares
an `actor` and optional `role`, then Agent Lexicon checks the requested local
action against the active mode.

Supported modes:

- `solo` — local-first mode where all roles can perform workspace actions.
- `team` — review and export are allowed for reviewers, while sync and snapshot publishing require maintainers or owners.
- `locked` — read-only for everyone except owners.

Create and inspect a local policy file:

```bash
agent-lexicon policy init --root examples/customer_limits --mode team --actor maxim --role owner
agent-lexicon policy status --root examples/customer_limits --actor maxim
agent-lexicon policy check --root examples/customer_limits --action publish_snapshot --actor maxim
```

The same policy checks are used by sensitive local commands:

```bash
agent-lexicon workspace sync examples/customer_limits/docs --root examples/customer_limits --actor maxim
agent-lexicon workspace publish-snapshot --root examples/customer_limits --actor maxim
agent-lexicon review --root examples/customer_limits --actor reviewer-1 --role reviewer
```

Python usage:

```python
from agent_lexicon import PolicyAction, check_local_policy, load_local_policy

policy = load_local_policy("examples/customer_limits")
decision = check_local_policy(
    policy,
    PolicyAction.PUBLISH_SNAPSHOT,
    actor="reviewer-1",
    role="reviewer",
)

print(decision.allowed, decision.reason)
```

This is RBAC-lite, not authentication. It is designed for local workflows, CI,
and future MCP/LLM review boundaries where the agent should know which actions
are allowed before changing review state or publishing snapshots.

## SQLite workspace state

Agent Lexicon can keep local ingest, scout candidates, and evidence packs in a
SQLite workspace under `.agent-lexicon/agent_lexicon.db`. The workspace is a
local cache for review and snapshot workflows. It is safe to delete and rebuild
from project files; team source of truth remains lexicon files, review exports,
and published snapshots.

Command line usage:

```bash
agent-lexicon workspace init --root examples/customer_limits
agent-lexicon workspace sync examples/customer_limits/docs --root examples/customer_limits --max-candidates 5
agent-lexicon workspace status --root examples/customer_limits
agent-lexicon workspace status --root examples/customer_limits --json
agent-lexicon workspace export-review-events --root examples/customer_limits
agent-lexicon workspace export-review-events --root examples/customer_limits --output review-events.jsonl
```

Python usage:

```python
from agent_lexicon import init_workspace, ingest_local_paths

state = init_workspace("examples/customer_limits")
ingest_report = ingest_local_paths(["examples/customer_limits/docs"], root="examples/customer_limits")
state.store_ingest_report(ingest_report)

summary = state.summary()
print(summary.document_count, summary.db_path)
```

The workspace stores documents, candidate payloads, evidence pack payloads,
local review decisions, and append-only review events. The database is designed
for local review workflows without requiring a service backend. SQLite connections
use WAL mode with a 30-second busy timeout and `synchronous=NORMAL`, so parallel
readers and short-lived writer bursts from local agents or CI jobs do not fail
with transient lock errors. Writes are still serialized by SQLite, preserving
transaction order while allowing readers to observe consistent snapshots.

## Local web proposal inbox

The local proposal inbox turns workspace candidates and evidence packs into a
small browser-based review queue. It runs only on localhost by default and does
not require a frontend build, a database server, or a hosted service.

Start by syncing a workspace, then open the inbox:

```bash
agent-lexicon workspace sync examples/customer_limits/docs --root examples/customer_limits --max-candidates 5
agent-lexicon review --root examples/customer_limits
```

The default URL is:

```text
http://127.0.0.1:8765
```

The inbox shows each candidate surface, score, jargon signal, background
penalty, positive evidence, and negative evidence. Reviewer decisions are saved
back to the local SQLite workspace as `accepted`, `rejected`, `ambiguous`, or
`needs_split`. The interface is intentionally minimal: one candidate list, one
evidence card, and one decision area.

For terminal-only environments, keep the server from opening a browser:

```bash
agent-lexicon review --root examples/customer_limits --no-browser
```

## Review events

Every local review decision is also stored as an append-only event. This keeps
the current decision state easy to query while preserving the review trail for
later proposal exports, snapshot publishing, and review dataset analysis. Events
include the decision, reviewer note, timestamp, candidate snapshot, and evidence
snapshot.

Command line usage:

```bash
agent-lexicon workspace export-review-events --root examples/customer_limits
agent-lexicon workspace export-review-events --root examples/customer_limits --decision accepted
agent-lexicon workspace export-review-events --root examples/customer_limits --output review-events.jsonl
```

Python usage:

```python
from agent_lexicon import export_review_events_jsonl, init_workspace

state = init_workspace("examples/customer_limits")
jsonl = export_review_events_jsonl(state)
```

The local web inbox also exposes the same export at `/review-events.jsonl` while
the server is running.

## Canonical migrations

Deprecated terms can declare a replacement canonical term through metadata. This
lets teams keep the old term visible for migration while directing agents and
reviewers toward the active canonical term. Agent Lexicon can also suggest a
conservative migration candidate from surface similarity when no explicit
replacement is declared.

Lexicon example:

```yaml
terms:
  - id: billing.credit_limit
    canonical: credit limit
  - id: billing.customer_cap
    canonical: customer cap
    deprecated: true
    metadata:
      replacement_term_id: billing.credit_limit
```

Command line usage:

```bash
agent-lexicon discover-migrations examples/customer_limits/lexicon.yaml
agent-lexicon discover-migrations examples/customer_limits/lexicon.yaml --json
agent-lexicon discover-migrations examples/customer_limits/lexicon.yaml --jsonl
```

Python usage:

```python
from agent_lexicon import discover_canonical_migration_candidates, load_lexicon

lexicon = load_lexicon("examples/customer_limits/lexicon.yaml")
report = discover_canonical_migration_candidates(lexicon)

for candidate in report.candidates:
    print(candidate.deprecated_term_id, "->", candidate.replacement_term_id)
```

Each migration candidate includes the deprecated term, replacement term,
confidence, risk, rationale, surfaces to preserve as aliases, and a
`canonical_migration` proposal representation. Deprecated surfaces are ignored
by collision validation so a replacement term can safely carry old wording as an
alias after review.

## Behavior metrics

Agent Lexicon can run deterministic behavior checks against a local `queries.jsonl` dataset.
The report measures terminology resolution, ambiguity detection, canonicalization,
and unsafe tool-call prevention.

```bash
agent-lexicon check examples/customer_limits/lexicon.yaml examples/customer_limits/queries.jsonl
```

Example output:

```text
Behavior check: 38/38 checks passed across 5 queries
Overall accuracy: 100.0%
Ambiguity detection: 100.0%
Canonicalization: 100.0%
Wrong tool prevention: 100.0%
Tool status: 100.0%
Tool allowed: 100.0%
```

For automation and dashboards, the same report can be emitted as JSON:

```bash
agent-lexicon check examples/customer_limits/lexicon.yaml examples/customer_limits/queries.jsonl --json
```

## Published local snapshots

Accepted local review decisions can be promoted into a lexicon-compatible JSON
snapshot. The snapshot can be validated and loaded by the same runtime APIs used
for normal lexicon documents. This keeps the local review workflow simple: scout
finds candidates, reviewers accept the terms they trust, and `publish-snapshot`
writes a deterministic artifact for agents and CI.

Command line usage after accepting candidates in the local review inbox:

```bash
agent-lexicon workspace publish-snapshot --root examples/customer_limits
agent-lexicon workspace publish-snapshot --root examples/customer_limits --output examples/customer_limits/snapshot.json
agent-lexicon workspace publish-snapshot --root examples/customer_limits --lexicon examples/customer_limits/lexicon.yaml --json
agent-lexicon validate examples/customer_limits/snapshot.json
```

Python usage:

```python
from agent_lexicon import open_workspace, publish_local_snapshot

state = open_workspace("examples/customer_limits")
snapshot = publish_local_snapshot(
    state,
    output_path="examples/customer_limits/snapshot.json",
)

print(snapshot.snapshot_id, snapshot.generated_term_count)
```

Only `accepted` review decisions are promoted. Rejected, ambiguous, and
needs-split candidates remain in the workspace review history and can still be
exported through the review events JSONL workflow. Snapshot files are written
atomically: Agent Lexicon writes the new JSON to a temporary file in the same
directory, flushes it, and then promotes it with an atomic replace. Readers see
either the previous complete snapshot or the new complete snapshot, not a
partially written file.

## Dictionary-as-code layout

Agent Lexicon separates local workspace state from git-tracked dictionary files.
The `.agent-lexicon/` directory stores SQLite cache, local review decisions, and
workspace metadata. The `lexicon/` directory is the reviewable source of truth
that can be committed, reviewed in pull requests, validated in CI, and used by
runtime agents.

Create the standard layout:

```bash
agent-lexicon dictionary init --root .
```

The command creates:

```text
lexicon/
  README.md
  lexicon.yaml
  queries.jsonl
  proposals/
  snapshots/
  review-events/
```

Validate the layout before opening a pull request:

```bash
agent-lexicon dictionary validate --root .
agent-lexicon check lexicon/lexicon.yaml lexicon/queries.jsonl
```

Python usage:

```python
from agent_lexicon import init_dictionary_layout, validate_dictionary_layout

summary = init_dictionary_layout(".")
assert summary.valid

validated = validate_dictionary_layout(".")
print(validated.layout.lexicon_path)
```

The layout command preserves existing files by default. Use `--force` only when
you intentionally want to overwrite the generated starter files.

## Semantic diff

Agent Lexicon can compare two validated lexicon files by terminology semantics
instead of raw line changes. This keeps pull request review focused on canonical
terms, aliases, scopes, tool mappings, evidence, proposals, and metadata.

```bash
agent-lexicon dictionary diff lexicon/lexicon.yaml lexicon-next.yaml
agent-lexicon dictionary diff lexicon/lexicon.yaml lexicon-next.yaml --json
agent-lexicon dictionary diff lexicon/lexicon.yaml lexicon-next.yaml --fail-on-change
```

Python usage:

```python
from agent_lexicon import diff_lexicon_files

report = diff_lexicon_files("lexicon/lexicon.yaml", "lexicon-next.yaml")
for change in report.changes:
    print(change.to_text())
```

Use `--fail-on-change` in automation when a workflow needs to detect whether a
lexicon update contains any semantic changes.

## Semantic merge

Agent Lexicon can perform a three-way merge for lexicon files using terminology
objects instead of raw text. Non-overlapping changes are merged automatically;
competing edits to the same semantic field become conflicts that can be handled
in review before publishing a dictionary update.

```bash
agent-lexicon dictionary merge lexicon-base.yaml lexicon-ours.yaml lexicon-theirs.yaml --output lexicon-merged.json
agent-lexicon dictionary merge lexicon-base.yaml lexicon-ours.yaml lexicon-theirs.yaml --check
agent-lexicon dictionary merge lexicon-base.yaml lexicon-ours.yaml lexicon-theirs.yaml --json
```

Python usage:

```python
from agent_lexicon import merge_lexicon_files, write_merged_lexicon_json

report = merge_lexicon_files(
    "lexicon-base.yaml",
    "lexicon-ours.yaml",
    "lexicon-theirs.yaml",
)

if report.has_conflicts:
    for conflict in report.conflicts:
        print(conflict.to_text())
else:
    write_merged_lexicon_json(report, "lexicon-merged.json")
```

Use semantic merge when multiple branches update terminology at the same time.
Merged lexicon output is written atomically using the same local artifact write
path as published snapshots. This keeps CI jobs and agent processes from reading
a partially written merge result while another process is publishing it.
It can combine independent additions such as new aliases, tools, metadata, or
evidence while blocking ambiguous edits such as two different canonical names for
the same term.

Clean merges can still include semantic warnings. These warnings do not block
writing the merged lexicon, but they flag cases that deserve reviewer attention.
For example, if one branch renames a term's canonical surface while another
branch adds aliases, tools, or evidence to the same term, Agent Lexicon reports a
`canonical_rename_with_parallel_term_change` warning. This keeps terminology
review aware that both branches may no longer be describing the same concept,
even though the field-level merge is mechanically clean.

## CI and pull request validation

Agent Lexicon includes a single dictionary PR check command for local CI and
GitHub Actions. It validates the git-tracked dictionary layout, runs the behavior
dataset, optionally prints a semantic diff against a base lexicon, and can check
three-way semantic merge inputs for conflicts.

```bash
agent-lexicon dictionary pr-check --root .
agent-lexicon dictionary pr-check --root . --base-lexicon /tmp/base-lexicon.yaml
agent-lexicon dictionary pr-check --root . --base-lexicon /tmp/base-lexicon.yaml --json
```

For stricter automation, fail when any semantic change is detected:

```bash
agent-lexicon dictionary pr-check --root . --base-lexicon /tmp/base-lexicon.yaml --fail-on-semantic-change
```

For merge validation, provide all three merge inputs together:

```bash
agent-lexicon dictionary pr-check --root . --merge-base base.yaml --merge-ours ours.yaml --merge-theirs theirs.yaml
```

The repository also includes a `Dictionary` GitHub Actions workflow that runs the
PR check on `main` pushes and pull requests. On pull requests, it attempts to
load the base branch `lexicon/lexicon.yaml` and includes a semantic diff in the
check output.

## Development

Install the development environment with Poetry:

```bash
poetry install --with dev
```

Run the test suite:

```bash
poetry run pytest -q
```

The repository also includes Make targets for the same workflow:

```bash
make install
make test
make check
```

## Relationship to SkeinRank

Agent Lexicon is intended to be the lightweight runtime SDK that agents can call
locally. SkeinRank remains the enterprise control plane for terminology drift,
proposal review, governed snapshots, and search/RAG integration.

## License

Apache License 2.0.

## Eval query datasets

Agent Lexicon uses JSONL query datasets to describe expected runtime behavior.
Each row contains one user query, optional scopes, expected terminology
resolution, and optional tool-call safety expectations. Metrics are computed by
the evaluation runner, while this layer keeps the dataset format validated and
portable.

Example row:

```json
{"id":"ambiguous.limit","text":"increase the limit","expected_status":"ambiguous","expected_action":"ask_clarification","expected_term_ids":["billing.credit_limit","api.rate_limit"],"tool_calls":[{"tool_name":"api.update_rate_limit","expected_status":"needs_clarification","expected_action":"ask_clarification","expected_allowed":false}]}
```

Validate a dataset from the command line:

```bash
agent-lexicon validate-queries examples/customer_limits/queries.jsonl
agent-lexicon check examples/customer_limits/lexicon.yaml examples/customer_limits/queries.jsonl
agent-lexicon ingest README.md src examples/customer_limits/docs --root .
agent-lexicon discover-candidates examples/customer_limits/docs --root examples/customer_limits --quality-report
agent-lexicon build-evidence examples/customer_limits/docs --root examples/customer_limits
agent-lexicon workspace sync examples/customer_limits/docs --root examples/customer_limits --max-candidates 5
```

Load the same dataset from Python:

```python
from agent_lexicon import load_eval_queries

queries = load_eval_queries("examples/customer_limits/queries.jsonl")
assert queries[0].expected_status.value == "ambiguous"
```

The loader validates duplicate ids, JSONL structure, expected resolver statuses,
expected resolver actions, tool guard statuses, tool guard actions, and primary
term references before returning typed query objects.




## Review Agent

Agent Lexicon can produce a local pre-review recommendation for one workspace
candidate. The review agent prepares a safe data-only prompt for optional LLM
review, validates structured LLM responses, and falls back to a deterministic
local assessment when no model response is provided.

```bash
agent-lexicon workspace sync examples/customer_limits/docs --root examples/customer_limits --max-candidates 5
agent-lexicon review-agent prompt --root examples/customer_limits --surface billing.update_credit_limit
agent-lexicon review-agent assess --root examples/customer_limits --surface billing.update_credit_limit
agent-lexicon review-agent consensus --root examples/customer_limits --surface billing.update_credit_limit
agent-lexicon review-agent consensus --root examples/customer_limits --surface billing.update_credit_limit --json
agent-lexicon review-agent assess --root examples/customer_limits --surface billing.update_credit_limit --json
```

The prompt command marks project evidence as untrusted data and uses the prompt
safety layer before content is sent to an external LLM. The assess command
returns one of `accept`, `reject`, `needs_split`, or `needs_more_evidence` and
includes the matching workspace review status for downstream tools. The
consensus command aggregates multiple structured review samples when supplied and
abstains when agreement or confidence is too low.

Python usage:

```python
from agent_lexicon import open_workspace, run_review_agent, run_review_agent_consensus

state = open_workspace("examples/customer_limits")
item = state.get_review_item("billing.update_credit_limit")
assert item is not None
decision = run_review_agent(item)
consensus = run_review_agent_consensus(item)
print(decision.recommendation.value, consensus.status.value)
```

Structured model responses can be passed through `run_review_agent(...,
llm_response=...)` or the CLI `--llm-response` option. High-risk prompt-safety
findings block LLM review and return a safer `needs_more_evidence` decision.

Consensus mode accepts one or more `--llm-response` files. If samples disagree or
the top decision is below the confidence threshold, Agent Lexicon returns an
abstention instead of a silent low-confidence proposal.

```bash
agent-lexicon review-agent consensus --root examples/customer_limits --surface billing.update_credit_limit --llm-response sample-a.json --llm-response sample-b.json
```


## Review dataset quality loop

Local review decisions can be exported as quality-labeled JSONL examples for
future evals, regression tests, and optional model improvement workflows. The
dataset export keeps human decisions separate from Review Agent suggestions so
the examples remain auditable.

```bash
agent-lexicon review-agent dataset --root examples/customer_limits
agent-lexicon review-agent dataset --root examples/customer_limits --json
agent-lexicon review-agent dataset --root examples/customer_limits --quality usable --output review-dataset.jsonl
agent-lexicon review-agent dataset --root examples/customer_limits --include-review-agent --json
```

Each exported row includes the candidate snapshot, evidence snapshot, human
decision, reviewer note, quality label, quality flags, and optional Review Agent
recommendation. Quality labels are `usable`, `incomplete`, `conflicting`,
`unsafe`, and `low_signal`.

Python usage:

```python
from agent_lexicon import build_review_dataset, open_workspace

state = open_workspace("examples/customer_limits")
report = build_review_dataset(state, include_review_agent=True)
print(report.usable_count)
```

This layer is intentionally local and dependency-free. It prepares portable
examples that can later be evaluated, filtered, or imported into a larger
governance system without sending private evidence to external services.

## MCP server

Agent Lexicon can run a local Model Context Protocol stdio server so desktop
agents and coding agents can use the same terminology checks as the CLI. The
server is dependency-free, local-first, and reads the git-tracked lexicon plus
optional `.agent-lexicon/` workspace state.

```bash
agent-lexicon mcp tools
agent-lexicon mcp serve --root . --lexicon lexicon/lexicon.yaml
```

The local MCP server exposes these tools:

- `resolve_term` — resolve text and return `resolved`, `ambiguous`, or `unknown`.
- `check_language` — check whether text is safe for an agent to interpret directly.
- `guard_tool_call` — block unsafe tool calls when terminology is ambiguous or mismatched.
- `find_evidence` — return lexicon or workspace evidence for a term or surface.
- `submit_proposal` — save a local review decision under workspace policy.
- `get_snapshot` — return published local snapshot metadata.

For local clients, point the MCP command at your repository root and lexicon file.
The server reuses loaded lexicons and compiled resolvers through the in-process
runtime cache, which keeps repeated local tool calls predictable on larger
lexicons. Sensitive write actions use the same RBAC-lite policy layer as the
workspace and review commands.
