Metadata-Version: 2.4
Name: defect-check
Version: 0.0.1
Summary: Standalone defect-checking engine for AI Skills, Tools, and Prompts
Author-email: SanityOps <dev@sanityops.ai>
Maintainer-email: SanityOps <dev@sanityops.ai>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/sanityops-org/artifact-defect-check
Project-URL: Repository, https://github.com/sanityops-org/artifact-defect-check
Project-URL: Documentation, https://github.com/sanityops-org/artifact-defect-check#readme
Project-URL: Issues, https://github.com/sanityops-org/artifact-defect-check/issues
Project-URL: Changelog, https://github.com/sanityops-org/artifact-defect-check/releases
Project-URL: Source, https://github.com/sanityops-org/artifact-defect-check
Keywords: defect-check,qa,quality-assurance,llm,ai,skills,tools,prompts,inspection,static-analysis
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: anthropic>=0.40.0
Requires-Dist: httpx>=0.25.2
Requires-Dist: openai>=2.8.0
Requires-Dist: pydantic>=2.5.0
Requires-Dist: pydantic-settings>=2.1.0
Requires-Dist: PyYAML>=6.0.3
Requires-Dist: python-dotenv>=1.0.0
Dynamic: license-file

<div align="center">

# defect-check

**Standalone defect-checking engine for AI Skills, Tools, and Prompts**

[![PyPI version](https://img.shields.io/pypi/v/defect-check.svg)](https://pypi.org/project/defect-check/)
[![Python](https://img.shields.io/pypi/pyversions/defect-check.svg)](https://pypi.org/project/defect-check/)
[![License](https://img.shields.io/github/license/sanityops-org/artifact-defect-check.svg)](LICENSE)
[![CI](https://img.shields.io/github/actions/workflow/status/sanityops-org/artifact-defect-check/publish.yml?branch=main&label=publish)](https://github.com/sanityops-org/artifact-defect-check/actions/workflows/publish.yml)

</div>

---

`defect-check` is a standalone, framework-free inspection engine for AI artifacts. It accepts **Skills**, **Tools**, and **Prompts** as input, runs multi-dimensional quality checks, and returns structured defect reports with scores and severity ratings.

- 🔍 **Four inspection modules** — QDS, QDT, QDP, and Cross (PS/PT/ST)
- 🤖 **LLM-powered analysis** — supports OpenAI, Anthropic, and DashScope providers
- 🎚️ **Three inspection levels** — L1 (quick), L2 (standard), L3 (deep)
- 📦 **Zero infrastructure** — no database, no API server, no task queue
- 🏗️ **Framework-free** — bring your own runtime, the engine stays pure

---

## Table of Contents

- [defect-check](#defect-check)
  - [Table of Contents](#table-of-contents)
  - [Installation](#installation)
  - [Quick Start](#quick-start)
  - [Inspection Levels](#inspection-levels)
  - [LLM Configuration](#llm-configuration)
  - [Inspection Modules](#inspection-modules)
  - [Response Format](#response-format)
    - [Defect Fields](#defect-fields)
  - [API Reference](#api-reference)
    - [`defect_check.check(...)`](#defect_checkcheck)
    - [`defect_check.check_single(...)`](#defect_checkcheck_single)
    - [`defect_check.check_cross(...)`](#defect_checkcheck_cross)
    - [Exported Types](#exported-types)
  - [Development](#development)
  - [Contributing](#contributing)
  - [License](#license)

---

## Installation

```bash
pip install defect-check
```

> Requires Python ≥ 3.11

---

## Quick Start

```python
import asyncio
import defect_check

async def main():
    result = await defect_check.check(
        tools=[
            {
                "name": "lookup_order",
                "description": "Query orders by order ID",
                "parameters": {"type": "object"},
            }
        ],
        prompts=[
            {"name": "system", "content": "You are an order assistant."}
        ],
        skills=[
            {"id": "orders", "name": "orders", "content": "# Orders workflow"}
        ],
        llm_provider="openai",
        llm_base_url="https://api.example.com/v1",
        llm_api_key="your-api-key",
        llm_model_id="your-model-id",
    )
    print(result)

asyncio.run(main())
```

---

## Inspection Levels

Use `check_level` to control inspection depth:

| Level | Description | Speed |
|-------|-------------|-------|
| `L1` | Quick check — basic validation | ⚡ Fastest |
| `L2` | Standard check — moderate depth | ⚙️ Balanced |
| `L3` | Deep check — comprehensive analysis | 🔬 Thorough |

```python
result = await defect_check.check(
    tools=tools,
    prompts=prompts,
    skills=skills,
    check_level="L3",
    llm_provider="openai",
    llm_api_key="your-api-key",
    llm_model_id="your-model-id",
)
```

When omitted, QDS determines the level using the bundled checklist, while QDT, QDP, and Cross determine it via the LLM. You can also pass `check_level` through `options`:

```python
result = await defect_check.check(
    tools=tools,
    prompts=prompts,
    skills=skills,
    options=defect_check.DefectCheckOptions(check_level="L2"),
    llm_provider="openai",
    llm_api_key="your-api-key",
    llm_model_id="your-model-id",
)
```

> The legacy `qdp_check_level` option remains supported for backwards compatibility. Conflicting values (e.g. `check_level="L3"` + `options={"qdp_check_level": "L1"}`) will raise an error.

---

## LLM Configuration

LLM settings are passed **explicitly by the caller** — the package does not read `.env` files or environment variables for LLM configuration.

| Parameter | Description | Required |
|-----------|-------------|----------|
| `llm_provider` | Provider name: `"openai"`, `"anthropic"`, or `"dashscope"` | ✅ |
| `llm_api_key` | API key for the provider | ✅ |
| `llm_base_url` | Custom base URL (e.g. for self-hosted endpoints) | Optional |
| `llm_model_id` | Model identifier (e.g. `"gpt-4o"`, `"claude-sonnet-4-20250514"`) | ✅ |

You can also pass a pre-configured client object via the `provider` parameter, bypassing the four `llm_*` parameters:

```python
from defect_check.llm import DefectCheckTextClient

# Build your own client, then pass it in
client = DefectCheckTextClient(my_custom_provider)

result = await defect_check.check(
    tools=tools,
    prompts=prompts,
    skills=skills,
    provider=client,
)
```

---

## Inspection Modules

The package provides four inspection modules, each targeting a different artifact dimension:

| Module | Full Name | Target | Method |
|--------|-----------|--------|--------|
| **QDS** | Quality of Design Specification | Skills | Checklist + rules |
| **QDT** | Quality of Design Tools | Tools | LLM + rules |
| **QDP** | Quality of Design Prompts | Prompts | LLM + rules |
| **Cross** | Cross-artifact inspection (PS/PT/ST) | All pairs | LLM + rules |

Every supplied Skill, Tool, and Prompt is inspected. The response always uses a consistent envelope — `results` is always a list: one input produces one result item, multiple inputs produce multiple result items.

Inspection rules and prompt templates are packaged in the wheel. QDT, QDP, and Cross load YAML resources; QDS loads the bundled `checklist.py`.

---

## Response Format

```json
{
  "schema_version": "1.0",
  "status": "completed",
  "results": [
    {
      "module": "QDT",
      "check_type": "artifact",
      "status": "completed",
      "check_level": "L2",
      "artifacts": [{"type": "tool", "id": "lookup_order", "name": "lookup_order"}],
      "score": {"total_score": 100.0, "max_score": 100.0, "grade": null, "gate_result": "PASS"},
      "defect_summary": {"total_defects": 0, "p0_count": 0, "p1_count": 0, "p2_count": 0},
      "defects": [],
      "error": null,
      "details": {},
      "metadata": {}
    }
  ],
  "summary": {
    "total_results": 1,
    "completed_results": 1,
    "failed_results": 0,
    "skipped_results": 0,
    "total_defects": 0,
    "p0_count": 0,
    "p1_count": 0,
    "p2_count": 0,
    "gate_result": "PASS"
  },
  "errors": [],
  "metadata": {"execution_time_seconds": 0.0}
}
```

### Defect Fields

Each defect in the `defects` list contains these canonical fields:

| Field | Description |
|-------|-------------|
| `id` | Unique defect identifier |
| `name` | Short defect name |
| `severity` | `P0` (critical), `P1` (major), or `P2` (minor) |
| `category` | Defect category |
| `description` | Human-readable description |
| `location` | Where the defect was found |
| `impact` | Impact of the defect |
| `fix_suggestion` | Recommended fix |
| `artifact_refs` | References to affected artifacts |
| `details` | Module-specific extra fields |

---

## API Reference

### `defect_check.check(...)`

Inspect caller-provided Skills, Tools, and Prompts.

```python
async def check(
    tools: list[dict] | None,
    prompts: list[dict] | None,
    skills: list[dict] | None,
    *,
    check_level: str | None = None,
    options: DefectCheckOptions | dict | None = None,
    provider: Any | None = None,
    llm_provider: str | None = None,
    llm_base_url: str | None = None,
    llm_api_key: str | None = None,
    llm_model_id: str | None = None,
) -> dict[str, Any]
```

### `defect_check.check_single(...)`

Inspect a single artifact. See the [API documentation](https://github.com/sanityops-org/artifact-defect-check/wiki) for details.

### `defect_check.check_cross(...)`

Run cross-artifact inspection (PS/PT/ST). See the [API documentation](https://github.com/sanityops-org/artifact-defect-check/wiki) for details.

### Exported Types

```python
from defect_check import (
    DefectCheckOptions,
    DefectCheckResponse,
    DefectItem,
    DefectSummary,
    InspectionResult,
    InspectionError,
    ScoreResult,
    ResponseSummary,
    ArtifactReference,
    SkillArtifact,
    PromptArtifact,
)
```

---

## Development

```bash
# Clone the repository
git clone https://github.com/sanityops-org/artifact-defect-check.git
cd artifact-defect-check

# Create a virtual environment
python -m venv .venv && source .venv/bin/activate

# Install in development mode
pip install -e ".[dev]"

# Run tests
pytest
```

---

## Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

Please make sure to update tests as appropriate and adhere to the existing code style.

---

## License

This project is licensed under the Apache License 2.0 — see the [LICENSE](LICENSE) file for details.
