Metadata-Version: 2.4
Name: lightassay
Version: 0.3.3
Summary: File-based orchestrator for structured evaluation of applied LLM workflows: humans declare intent, LLMs reason about quality, code runs and records raw facts
Author-email: Vadim Larin <vadimlarintech@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/vadimlarintech/lightassay
Project-URL: Repository, https://github.com/vadimlarintech/lightassay
Project-URL: Issues, https://github.com/vadimlarintech/lightassay/issues
Project-URL: Changelog, https://github.com/vadimlarintech/lightassay/blob/main/CHANGELOG.md
Keywords: llm,eval,evaluation,testing,workflow,ai,ai-agents
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: build; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Dynamic: license-file

# Vibe-testing LLM workflows for vibe coders

`lightassay` is AI-native evaluation for real LLM workflows: describe what
worries you, let an agent build the workbook, run the actual workflow, and
review semantic analysis. No fake coded quality metrics.

## Install

```bash
pip install lightassay
```

## Setup

Requires an installed and authenticated Codex CLI or Claude CLI.

```bash
lightassay init
```

## Quick Start

```bash
lightassay quickstart \
  --message "Check the main workflow behavior, edge cases, and obvious failure modes." \
  --target "python:myapp.pipeline.run"
```

## Use Cases

- Create a first eval suite from plain-language intent.
- Rerun the same workbook after app fixes.
- Compare target models from one workbook.
- Connect Python, JavaScript, TypeScript, HTTP, or command-based workflows.
- Keep evaluation state in local workbook and JSON artifacts.

## Commands

- `lightassay quickstart` - create a workbook, run the target workflow, and write analysis.
- `lightassay continue` - continue an existing workbook with new instructions.
- `lightassay rerun` - rerun the same prepared workbook after app changes.
- `lightassay compare-models` - run the workbook model matrix and compare results.
- `lightassay adapter-template` - generate a thin target adapter template.
- `lightassay adapter-check` - validate a target adapter before running evals.
- `lightassay agent-guide` - print installed-package guidance for coding agents.

## Why Lightassay

- AI-first: built for agent-driven eval creation and analysis.
- Agent-operated: uses your authenticated Codex CLI or Claude CLI subscription for
  preparation, analysis, and compare instead of a separate analysis API bill.
- File-based: workbook markdown plus JSON artifacts.
- Real workflow boundary: tests your target workflow, not isolated prompt snippets.
- No fake coded quality metrics.

## Docs

For installed-package guidance:

```bash
lightassay agent-guide
```

Repository docs:

- [Quickstart](https://github.com/vadimlarintech/lightassay/blob/main/docs/quickstart.md)
- [Agent operational guide](https://github.com/vadimlarintech/lightassay/blob/main/docs/agent_operational_guide.md)

## License

MIT - see [LICENSE](LICENSE).
