Metadata-Version: 2.4
Name: defect-check
Version: 1.2.8
Summary: Standalone defect-checking engine for AI Skills, Tools, and Prompts
Author-email: SanityOps <dev@sanityops.ai>
Maintainer-email: SanityOps <dev@sanityops.ai>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/sanityops-org/artifact-defect-check
Project-URL: Repository, https://github.com/sanityops-org/artifact-defect-check
Project-URL: Documentation, https://github.com/sanityops-org/artifact-defect-check#readme
Project-URL: Issues, https://github.com/sanityops-org/artifact-defect-check/issues
Project-URL: Changelog, https://github.com/sanityops-org/artifact-defect-check/releases
Project-URL: Source, https://github.com/sanityops-org/artifact-defect-check
Keywords: defect-check,qa,quality-assurance,llm,ai,skills,tools,prompts,inspection,static-analysis
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: anthropic<1.0.0,>=0.40.0
Requires-Dist: httpx>=0.25.2
Requires-Dist: openai<3.0.0,>=2.8.0
Requires-Dist: pydantic>=2.5.0
Requires-Dist: pydantic-settings>=2.1.0
Requires-Dist: PyYAML>=6.0.3
Requires-Dist: python-dotenv>=1.0.0
Dynamic: license-file

<div align="center">

# defect-check

**Standalone defect-checking engine for AI Skills, Tools, and Prompts**

[![PyPI version](https://img.shields.io/pypi/v/defect-check.svg)](https://pypi.org/project/defect-check/)
[![Python](https://img.shields.io/pypi/pyversions/defect-check.svg)](https://pypi.org/project/defect-check/)
[![License](https://img.shields.io/github/license/sanityops-org/artifact-defect-check.svg)](LICENSE)
[![CI](https://img.shields.io/github/actions/workflow/status/sanityops-org/artifact-defect-check/publish.yml?branch=main&label=publish)](https://github.com/sanityops-org/artifact-defect-check/actions/workflows/publish.yml)

</div>

---

`defect-check` is a standalone, framework-free inspection engine for AI artifacts. It accepts **Skills**, **Tools**, and **Prompts** as input, runs multi-dimensional quality checks, and returns structured defect reports with scores and severity ratings.

- 🔍 **Five inspection modules** — QDS, QDT, QDP, Cross (PS/PT/ST), and QD-PM (Permission)
- 🤖 **LLM-powered analysis** — supports OpenAI, Anthropic, and DashScope providers
- 🎚️ **Three inspection levels** — L1 (quick), L2 (standard), L3 (deep)
- 🛡️ **Permission risk assessment** — OR-L1/L2/L3 operation risk levels for safety-critical checks
- ⚖️ **Built-in rate control** — configurable retry (default 3) and LLM concurrency limit (default 5)
- 📦 **Zero infrastructure** — no database, no API server, no task queue
- 🏗️ **Framework-free** — bring your own runtime, the engine stays pure

---

## Table of Contents

- [defect-check](#defect-check)
  - [Table of Contents](#table-of-contents)
  - [Installation](#installation)
  - [Quick Start](#quick-start)
    - [Recommended: the `DefectChecker` class](#recommended-the-defectchecker-class)
  - [Inspection Levels](#inspection-levels)
  - [LLM Configuration](#llm-configuration)
    - [Retry and Concurrency Control](#retry-and-concurrency-control)
    - [Custom YAML Configuration](#custom-yaml-configuration)
  - [Inspection Modules](#inspection-modules)
  - [Response Format](#response-format)
    - [Defect Fields](#defect-fields)
  - [API Reference](#api-reference)
    - [`defect_check.check(...)`](#defect_checkcheck)
    - [`defect_check.check_single(...)`](#defect_checkcheck_single)
    - [`defect_check.check_cross(...)`](#defect_checkcheck_cross)
    - [`DefectChecker` class](#defectchecker-class)
    - [Exported Types](#exported-types)
  - [Development](#development)
  - [Contributing](#contributing)
  - [License](#license)

---

## Installation

```bash
pip install defect-check
```

> Requires Python ≥ 3.11

---

## Quick Start

```python
import asyncio
import defect_check

async def main():
    result = await defect_check.check(
        tools=[
            {
                "name": "lookup_order",
                "description": "Query orders by order ID",
                "parameters": {"type": "object"},
            }
        ],
        prompts=[
            {"name": "system", "content": "You are an order assistant."}
        ],
        skills=[
            {"id": "orders", "name": "orders", "content": "# Orders workflow"}
        ],
        llm_provider="openai",
        llm_base_url="https://api.example.com/v1",
        llm_api_key="your-api-key",
        llm_model_id="your-model-id",
    )
    print(result)

asyncio.run(main())
```

### Recommended: the `DefectChecker` class

For repeated use, create a `DefectChecker` once with your LLM settings and reuse it — no need to pass base parameters on every call. All check methods are async.

```python
import asyncio
from defect_check import DefectChecker

async def main():
    # Configure everything up front (max_retries defaults to 3, max_concurrency to 5)
    checker = DefectChecker(
        llm_provider="openai",
        llm_base_url="https://api.example.com/v1",
        llm_api_key="your-api-key",
        llm_model_id="your-model-id",
        max_retries=3,          # retries for recoverable LLM errors
        max_concurrency=5,      # max in-flight LLM requests across all calls
    )

    result = await checker.defect_check(
        tools=[{"name": "lookup_order", "description": "Query orders by order ID"}],
        prompts=[{"name": "system", "content": "You are an order assistant."}],
        skills=[{"id": "orders", "name": "orders", "content": "# Orders workflow"}],
    )

    single = await checker.check_single(tool={"name": "lookup_order"})
    cross = await checker.cross_defect_check(prompt="You are...", skills=skills, tools=tools)

    await checker.close()

asyncio.run(main())
```

You can also create the checker first and configure it later via setter methods:

```python
checker = DefectChecker()
checker.set_llm_config(llm_provider="openai", llm_api_key="sk-...", llm_model_id="gpt-4o")
checker.set_max_retries(5)      # override the default of 3
checker.set_max_concurrency(10) # override the default of 5

result = await checker.defect_check(tools=tools)
```

---

## Inspection Levels

Use `check_level` to control inspection depth:

| Level | Description | Speed |
|-------|-------------|-------|
| `L1` | Quick check — basic validation | ⚡ Fastest |
| `L2` | Standard check — moderate depth | ⚙️ Balanced |
| `L3` | Deep check — comprehensive analysis | 🔬 Thorough |

```python
result = await defect_check.check(
    tools=tools,
    prompts=prompts,
    skills=skills,
    check_level="L3",
    llm_provider="openai",
    llm_api_key="your-api-key",
    llm_model_id="your-model-id",
)
```

When omitted, QDS determines the level using the bundled checklist, while QDT, QDP, and Cross determine it via the LLM. You can also pass `check_level` through `options`:

```python
result = await defect_check.check(
    tools=tools,
    prompts=prompts,
    skills=skills,
    options=defect_check.DefectCheckOptions(check_level="L2"),
    llm_provider="openai",
    llm_api_key="your-api-key",
    llm_model_id="your-model-id",
)
```

> The legacy `qdp_check_level` option remains supported for backwards compatibility. Conflicting values (e.g. `check_level="L3"` + `options={"qdp_check_level": "L1"}`) will raise an error.

### Permission Check Options

Two additional options in `DefectCheckOptions` control the QD-PM permission inspection:

| Option | Default | Description |
|--------|---------|-------------|
| `enable_permission_check` | `True` | Enable/disable the QD-PM permission inspection stage |
| `or_level` | `None` (auto) | Operation Risk Level override: `OR-L1`, `OR-L2`, or `OR-L3` |

Invalid `or_level` values (anything other than `OR-L1`/`OR-L2`/`OR-L3`/`None`) raise a `ValueError`.

---

## LLM Configuration

LLM settings are passed **explicitly by the caller** — the package does not read `.env` files or environment variables for LLM configuration.

| Parameter | Description | Required |
|-----------|-------------|----------|
| `llm_provider` | Provider name: `"openai"`, `"anthropic"`, or `"dashscope"` | ✅ |
| `llm_api_key` | API key for the provider | ✅ |
| `llm_base_url` | Custom base URL (e.g. for self-hosted endpoints) | Optional |
| `llm_model_id` | Model identifier (e.g. `"gpt-4o"`, `"claude-sonnet-4-20250514"`) | ✅ |

You can also pass a pre-configured client object via the `provider` parameter, bypassing the four `llm_*` parameters:

```python
from defect_check.llm import DefectCheckTextClient

# Build your own client, then pass it in
client = DefectCheckTextClient(my_custom_provider)

result = await defect_check.check(
    tools=tools,
    prompts=prompts,
    skills=skills,
    provider=client,
)
```

The same `llm_*` / `provider` parameters are accepted by `DefectChecker(...)` and `checker.set_llm_config(...)`.

### Retry and Concurrency Control

The LLM client applies two protections that you can tune (defaults shown):

- **`max_retries`** — recoverable errors (HTTP 429, provider 5xx, connection failures) are retried with exponential backoff. Default: `3`.
- **`max_concurrency`** — an instance-wide cap on simultaneous in-flight LLM requests, shared across every check call made on the same checker. The semaphore covers each request *and* its retries. Default: `5`.

```python
checker = DefectChecker(provider=my_client, max_retries=5, max_concurrency=10)
```

> Best practice: set `max_concurrency` before running your first check. Changing it afterwards rebuilds the underlying client's shared semaphore, which briefly releases coordination for in-flight callers.

If the LLM configuration is missing when a check runs, the response carries an `LLM_CONFIGURATION_MISSING` error instead of raising an exception.

### Custom YAML Configuration

You can override the built-in inspection rules and prompt templates with custom YAML files:

```python
checker = DefectChecker(
    llm_provider="openai",
    llm_api_key="your-api-key",
    llm_model_id="gpt-4o",
    yaml_overrides={
        "qds_scoring": "/path/to/custom_qds_rules.yaml",
        "prompt_qds": "/path/to/custom_qds_prompt.yaml",
    },
)

# Or switch at runtime
checker.set_yaml_overrides({"qds_scoring": "/another/config.yaml"})

# Reset to built-in defaults
checker.set_yaml_overrides(None)
```

**Supported override keys:**

| Key | Purpose |
|-----|---------|
| `qdp_scoring` | Prompt inspection rules |
| `qds_scoring` | Skill inspection rules |
| `qdt_scoring` | Tool inspection rules |
| `cross_scoring` | Cross-artifact inspection rules |
| `prompt_qdp` | QDP LLM prompt template |
| `prompt_qds` | QDS LLM prompt template |
| `prompt_qdt` | QDT LLM prompt template |
| `prompt_cross` | Cross-artifact LLM prompt template |
| `prompt_level` | Level determination prompt template |
| `pm_scoring` | Permission (QD-PM) inspection rules |
| `prompt_extract` | QD-PM permission extraction prompt template |
| `prompt_judge` | QD-PM permission judgment prompt template |

For detailed YAML schema, domain-specific extensions, and examples, see **[Custom YAML Configuration](docs/custom-yaml-configuration.md)**.

---

## Inspection Modules

The package provides five inspection modules, each targeting a different artifact dimension:

| Module | Full Name | Target | Method |
|--------|-----------|--------|--------|
| **QDS** | Quality of Design Specification | Skills | Checklist + rules |
| **QDT** | Quality of Design Tools | Tools | LLM + rules |
| **QDP** | Quality of Design Prompts | Prompts | LLM + rules |
| **Cross** | Cross-artifact inspection (PS/PT/ST) | All pairs | LLM + rules |
| **QD-PM** | Permission inspection | All artifacts | LLM + rules |

Every supplied Skill, Tool, and Prompt is inspected. The response always uses a consistent envelope — `results` is always a list: one input produces one result item, multiple inputs produce multiple result items.

Inspection rules and prompt templates are packaged in the wheel. QDT, QDP, and Cross load YAML resources; QDS loads the bundled `checklist.py`.

### Permission Inspection (QD-PM)

QD-PM is a safety-focused module that inspects permission and operation risks in AI artifacts. It runs after single-artifact and cross-artifact inspections, subject to **Gate-0** pre-validation.

**Operation Risk Levels (OR-L):**

| Level | Risk | Examples |
|-------|------|----------|
| **OR-L1** | Read-only / low risk | Query, list, read operations with no side effects |
| **OR-L2** | Medium risk | Write, modify, export, create — reversible or local side effects |
| **OR-L3** | High risk | Delete, outbound send, irreversible operations, production/sensitive data access |

**Gate-0 Pre-validation:**

Before QD-PM runs, Gate-0 validates four structural preconditions:

- **G0-1**: No unfixed P0 defects in single-artifact inspection (QDS/QDT/QDP)
- **G0-2**: No unfixed P0 defects in cross-artifact inspection
- **G0-3**: A responsibility-boundary statement exists and is locatable
- **G0-4**: Every Tool Schema has sufficient spec for operational semantics

If any precondition fails, QD-PM returns an "evaluation cannot be performed" signal instead of a defect list.

**Configuration:**

```python
result = await defect_check.check(
    tools=tools,
    prompts=prompts,
    skills=skills,
    options=defect_check.DefectCheckOptions(
        enable_permission_check=True,  # enable QD-PM (default: True)
        or_level="OR-L2",              # set operation risk level (OR-L1/OR-L2/OR-L3)
    ),
    llm_provider="openai",
    llm_api_key="your-api-key",
    llm_model_id="gpt-4o",
)
```

### Unified Inspection Framework

Since v2.x, QDS, QDT, and QDP share a single unified inspection framework in `defect_check/core/`. Each domain is a thin declarative registration (`DomainRegistration`) wired into a 7-stage pipeline:

```
LevelDeterminer → EntryGate → ViewGenerator → Checklist → CheckExecutor → Enricher → Scorer → Adapter
```

| Stage | Responsibility |
|-------|----------------|
| **LevelDeterminer** | Determine check level (L1/L2/L3) from artifact content — keyword-based (QDS) or LLM-based (QDP/QDT) |
| **EntryGate** | Deterministic pre-check — QDS-0 gate validates frontmatter before scoring |
| **ViewGenerator** | Optional intermediate view (QDS) |
| **Checklist** | Derive the applicable check list from the domain defect registry |
| **CheckExecutor** | Main inspection logic — the per-domain executor (QDPExecutor, QDS GroupedParallelExecutor, QDTExecutor) |
| **Enricher** | Fill missing defect metadata from the registry |
| **Scorer** | Compute score, grade, and gate result via the unified `UnifiedScorer` |
| **Adapter** | Convert internal findings to the public `DefectItem` contract |

All three domains feed the same `InspectionPipeline` (see `defect_check/orchestrator.py`), which is what the public `check` / `check_single` entry points call. This guarantees identical contract behavior across domains while keeping QDS-specific features (QDS-0 gate, intermediate view, grouped parallel execution) as first-class framework concepts.

The orchestrator also runs **QD-PM (Permission Inspection)** after the single-artifact and cross-artifact stages, gated by Gate-0 pre-validation. QD-PM inspects permission proportionality using operation risk levels (OR-L) and agent capability levels (AC-L).

To add a new domain, implement a `DomainRegistration` (level determiner, gate, view generator, executor, enricher, scorer) and register it in `DomainRegistry` — see [docs/architecture.md](docs/architecture.md) for details.

---

## Response Format

```json
{
  "schema_version": "1.0",
  "status": "completed",
  "results": [
    {
      "module": "QDT",
      "check_type": "artifact",
      "status": "completed",
      "check_level": "L2",
      "artifacts": [{"type": "tool", "id": "lookup_order", "name": "lookup_order"}],
      "score": {"total_score": 100.0, "max_score": 100.0, "grade": null, "gate_result": "PASS"},
      "defect_summary": {"total_defects": 0, "p0_count": 0, "p1_count": 0, "p2_count": 0},
      "defects": [],
      "error": null,
      "details": {},
      "metadata": {}
    }
  ],
  "summary": {
    "total_results": 1,
    "completed_results": 1,
    "failed_results": 0,
    "skipped_results": 0,
    "total_defects": 0,
    "p0_count": 0,
    "p1_count": 0,
    "p2_count": 0,
    "gate_result": "PASS"
  },
  "errors": [],
  "metadata": {"execution_time_seconds": 0.0}
}
```

### Defect Fields

Each defect in the `defects` list contains these canonical fields:

| Field | Description |
|-------|-------------|
| `id` | Unique defect identifier |
| `name` | Short defect name |
| `severity` | `P0` (critical), `P1` (major), or `P2` (minor) |
| `category` | Defect category |
| `description` | Human-readable description |
| `location` | Where the defect was found |
| `impact` | Impact of the defect |
| `fix_suggestion` | Recommended fix |
| `artifact_refs` | References to affected artifacts |
| `details` | Module-specific extra fields |

---

## API Reference

### `defect_check.check(...)`

Inspect caller-provided Skills, Tools, and Prompts.

```python
async def check(
    tools: list[dict] | None,
    prompts: list[dict] | None,
    skills: list[dict] | None,
    *,
    check_level: str | None = None,
    options: DefectCheckOptions | dict | None = None,
    provider: Any | None = None,
    llm_provider: str | None = None,
    llm_base_url: str | None = None,
    llm_api_key: str | None = None,
    llm_model_id: str | None = None,
) -> dict[str, Any]
```

### `defect_check.check_single(...)`

Inspect a single artifact. See the [API documentation](https://github.com/sanityops-org/artifact-defect-check/wiki) for details.

### `defect_check.check_cross(...)`

Run cross-artifact inspection (PS/PT/ST). See the [API documentation](https://github.com/sanityops-org/artifact-defect-check/wiki) for details.

### `DefectChecker` class

A stateful facade that holds LLM configuration, retry, and concurrency settings — configure once, then run checks as instance methods.

```python
from defect_check import DefectChecker

checker = DefectChecker(
    provider: Any | None = None,          # pre-configured client/provider (optional)
    llm_provider: str | None = None,
    llm_base_url: str | None = None,
    llm_api_key: str | None = None,
    llm_model_id: str | None = None,
    max_retries: int = 3,
    max_concurrency: int = 5,
    options: DefectCheckOptions | dict | None = None,  # instance-level defaults
    yaml_overrides: dict[str, str] | None = None,      # custom YAML config overrides
)

# Configuration setters (can be called after construction)
checker.set_llm_config(llm_provider=..., llm_api_key=..., llm_model_id=...)
checker.set_max_retries(5)
checker.set_max_concurrency(10)
checker.set_options({"check_level": "L2"})
checker.set_yaml_overrides({"qds_scoring": "/path/to/custom.yaml"})  # custom rules

# Inspection methods (all async; parameter signatures match the module-level functions)
await checker.defect_check(tools=..., prompts=..., skills=...)   # full inspection
await checker.check_single(tool=..., prompt=..., skill=...)      # single artifact
await checker.cross_defect_check(prompt=..., skills=..., tools=...)  # PS/PT/ST only
await checker.close()                                            # release the LLM connection
```

Precedence for check settings: explicit method arguments > instance configuration (`set_options` / constructor `options`) > built-in defaults. Call-level `check_level` overrides the instance level without conflict errors.

### Exported Types

```python
from defect_check import (
    DefectCheckOptions,
    DefectCheckResponse,
    DefectItem,
    DefectSummary,
    InspectionResult,
    InspectionError,
    ScoreResult,
    ResponseSummary,
    ArtifactReference,
    SkillArtifact,
    PromptArtifact,
    RESOURCE_ALIASES,  # supported YAML override keys
)
```

---

## Development

```bash
# Clone the repository
git clone https://github.com/sanityops-org/artifact-defect-check.git
cd artifact-defect-check

# Create a virtual environment
python -m venv .venv && source .venv/bin/activate

# Install in development mode
pip install -e ".[dev]"

# Run tests
pytest
```

> **Note:** The package uses `pytest` with `asyncio_mode = "strict"` — async tests must be decorated with `@pytest.mark.asyncio` and awaited. When adding a new test module, keep its filename unique across the whole `tests/` tree (pytest errors on two test files sharing the same basename in different directories).

---

## Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

Please make sure to update tests as appropriate and adhere to the existing code style.

---

## License

This project is licensed under the Apache License 2.0 — see the [LICENSE](LICENSE) file for details.
