Metadata-Version: 2.4
Name: quality-loop
Version: 0.1.0
Summary: Measure what an agent-facing product teaches: isolated agent runs of real tasks, reported by an operator agent
Author: craftware
License-Expression: MIT
Project-URL: Homepage, https://github.com/crftwr/quality-loop
Project-URL: Repository, https://github.com/crftwr/quality-loop
Project-URL: Issues, https://github.com/crftwr/quality-loop/issues
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pytest-timeout>=2; extra == "dev"
Dynamic: license-file

# quality-loop

Measure what an agent-facing product teaches.

A skill bundle, an MCP server or a CLI is only as good as what an agent with
nothing else can do with it. quality-loop runs that agent on a real task, in a
**room** that holds the product's agent-facing surface and nothing else. The
room is started so it cannot read the repository where the answers are.
Afterwards an **operator** agent reports the problems it finds in what the
room did, each with a severity. A human decides what to do with the report.

> Status: early development, run end to end against a toy product
> (`examples/toy/run.py`). In place: rooms and their isolation, the audit,
> the stand-in, fixture pages, screens by name on macOS and Windows,
> Markdown scenarios, the operator agent and its MCP tools, and reports with
> severities. The first real product, Keyhac, is next. The design is
> [crftwr/keyhac#205](https://github.com/crftwr/keyhac/issues/205).

## What stays code, and why

Anything a run depends on to mean something is code, not something an agent
decides:

- **The prompt the room is handed** is copied verbatim. The operator knows
  what a good answer looks like, so any paraphrase of the prompt can leak it.
- **Isolation.** The room runs with `--restricted`, an explicit permission
  list under `dontAsk`, one MCP configuration and nothing else, git stopped at
  the room, and stdin closed. A transcript audit then checks what no flag can
  enforce.
- **The operator's machine.** Their files are moved aside and replaced with
  fixtures. On the way out they are put back in a `finally`, whether the run
  ended normally, by an exception, by Ctrl-C or by SIGTERM.
- **Evidence.** A report cites what tools recorded, not what the room said.

## Products using it

- [Keyhac](https://github.com/crftwr/keyhac)
- [XeFM](https://github.com/crftwr/xefm) (planned)

## Development

```
make test
```

MIT licensed.
