Metadata-Version: 2.4
Name: agent-failure-doctor
Version: 4.0.0
Summary: Local-first failure diagnosis for AI browser automation, Playwright, crawler, and RPA runs.
Author: sida lin
License-Expression: MIT
Project-URL: Homepage, https://github.com/tobybgy-lsd/web-agent-runtime-bench
Project-URL: Repository, https://github.com/tobybgy-lsd/web-agent-runtime-bench
Project-URL: Issues, https://github.com/tobybgy-lsd/web-agent-runtime-bench/issues
Keywords: playwright,browser automation,ai agent,crawler,rpa,debugging,failure diagnosis
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: trace-gen
Requires-Dist: playwright>=1.45; extra == "trace-gen"
Provides-Extra: ocr
Provides-Extra: paddleocr
Requires-Dist: paddleocr>=3.0.0; extra == "paddleocr"
Provides-Extra: baidu-ocr
Requires-Dist: requests>=2.31.0; extra == "baidu-ocr"
Dynamic: license-file

# Agent Failure Doctor

[中文文档](README.zh-CN.md)

![CI](https://github.com/tobybgy-lsd/web-agent-runtime-bench/actions/workflows/benchmark.yml/badge.svg)
![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)
![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)

Local-first failure diagnosis lifecycle tool for AI browser automation,
Playwright, crawler, RPA, and business automation failures.

**Input:** trace.zip / error.log / console.txt / network.json / probe_report.json / screenshot metadata / user_description.txt / visual_run / OCR or document evidence.

**Output:** diagnosis, evidence, next action, repair suggestions, GitHub issue draft, Codex fix prompt.

Optional v4.0 output: evidence-bound reasoning, local web console, and CI/CD gate.

Quickstart: `python -m pip install agent-failure-doctor`; `git clone https://github.com/tobybgy-lsd/web-agent-runtime-bench.git`; `cd web-agent-runtime-bench`; `failure-doctor diagnose .\examples\failed_runs\proxy_network_error --out .\report`; `failure-doctor plan .\report --out .\fix_plan`.

- Current milestone: Agent Failure Doctor v4.0 Hybrid Evidence Reasoning Pack
- Current stable line: v4.0.0
- Previous stable line: Agent Failure Doctor v3.9.0 Local Failure Knowledge Base Pack
- Previous P95 stable line: Agent Failure Doctor v2.4.1 P95 Alignment & Missing Tracks Pack

**P98 gate:** passed. P98 master gate passed.

**Classic lifecycle:** diagnose -> plan -> AI handoff / patch proposal -> verify -> sanitize/share.

**Core commands:** `diagnose` / `plan` / `verify` / `run`; `reason` / `root-cause` / `causal-chain`; `sanitize` / `adapt`; `ocr-evidence`; `visual-runtime`; `regulated-eval`; `full-chain-eval`; `console`; `ci`; `kb`; `failure-doctor propose-patch`; `failure-doctor batch`.

Agent bootstrap: `failure-doctor agent-bootstrap --target all --project .`.

See [validation/dashboard.md](validation/dashboard.md).


## Quickstart

```powershell
python -m pip install agent-failure-doctor
git clone https://github.com/tobybgy-lsd/web-agent-runtime-bench.git
cd web-agent-runtime-bench
failure-doctor diagnose .\examples\failed_runs\proxy_network_error --out .\report
failure-doctor plan .\report --out .\fix_plan
failure-doctor collect --project . --preset auto --out .\failure_doctor_auto_report `
  --auto-diagnose --auto-handoff --auto-sanitize
failure-doctor ocr-evidence extract --input .\screenshot.png --out .\ocr_report --provider mock_ocr
failure-doctor ocr-evidence compare --ocr .\ocr_report --dom .\dom_snapshot.html --out .\ocr_compare
failure-doctor visual-runtime diagnose --input .\visual_run --out .\visual_report --no-dom
failure-doctor visual-runtime compare --baseline .\dom_agent_run --candidate .\vlm_agent_run --out .\compare_report
failure-doctor regulated-eval --suite all --out .\regulated_report
failure-doctor full-chain-eval --input .\failed_run --out .\full_chain_report
failure-doctor console --import-report .\full_chain_report --open
failure-doctor ci run --input .\full_chain_report --out .\ci_report --fail-on high
failure-doctor ci templates --out .\ci_templates
failure-doctor kb init --path .\.failure-doctor-kb
failure-doctor kb import-report --kb .\.failure-doctor-kb --report .\full_chain_report
failure-doctor diagnose .\failed_run --kb .\.failure-doctor-kb --hybrid-reasoning --out .\report
failure-doctor reason --input .\report --out .\report\hybrid_reasoning
failure-doctor agent-bootstrap --target all --project .
```

### Hybrid Evidence Reasoning

v4.0 adds local, evidence-bound reasoning:

```powershell
failure-doctor reason --input .\report --out .\report\hybrid_reasoning
failure-doctor root-cause --input .\report --out .\root_cause_report
failure-doctor causal-chain --input .\report --out .\causal_chain_report
failure-doctor diagnose .\failed_run --kb .\.failure-doctor-kb --hybrid-reasoning --out .\report
```

The deterministic diagnosis remains the source of truth. Every reasoning claim
must cite a sanitized evidence id. Optional local reasoners never download
models automatically and never upload raw evidence.

### Local Failure Knowledge Base

v3.9 adds a local-only failure knowledge base:

```powershell
failure-doctor kb init --path .\.failure-doctor-kb
failure-doctor kb import-report --kb .\.failure-doctor-kb --report .\failure_doctor_auto_report
failure-doctor kb search --kb .\.failure-doctor-kb --query "selector timeout after login redirect"
failure-doctor kb match --kb .\.failure-doctor-kb --report .\failure_doctor_auto_report --out .\kb_match_report
failure-doctor kb export --kb .\.failure-doctor-kb --out .\kb_export --sanitized-only
failure-doctor diagnose .\failed_run --kb .\.failure-doctor-kb --out .\report
```

Use a local KB to match similar historical failures, reuse verified fix
candidates, and turn team troubleshooting history into a private, searchable
knowledge asset. The KB is local-only and sanitized-only by default: no cloud
sync, no external embedding API, no raw secret storage, no private local training
content, and verified fixes are suggestions, not auto-applied patches.

### CI/CD Integration

v3.8 adds local CI gates and starter templates:

```powershell
failure-doctor ci run --input .\report --out .\ci_report --fail-on high
failure-doctor ci validate --input .\ci_report --out .\ci_validation
failure-doctor ci templates --out .\ci_templates
```

The CI pack writes `ci_manifest.json`, `ci_summary.md/json`,
`severity_decision.json`, `gate_decision.json`, `audit_manifest.json`, and
`open_this_first_ci.md`. It is local-first, does not upload raw artifacts, and
blocks private training markers or unsafe recommendation markers before
external sharing. See [docs/CI_CD_INTEGRATION.md](docs/CI_CD_INTEGRATION.md).

### Local Web Console

v3.7 adds a local-only report console:

```powershell
failure-doctor console
failure-doctor console --host 127.0.0.1 --port 8765 --workspace .\.failure-doctor-console
failure-doctor console --import-report .\report --open
```

The console binds to `127.0.0.1` by default, serves bundled assets only, uses no
telemetry, requires a local token for POST actions, and hides raw local evidence
by default. It is a report viewer and workflow cockpit for diagnosis, evidence,
safety, AI handoff, patch proposal previews, visual runtime, OCR/document,
regulated workflow, batch/fleet, and full-chain reports. See
[docs/LOCAL_WEB_CONSOLE.md](docs/LOCAL_WEB_CONSOLE.md).

Release tracks: Previous stable line: Agent Failure Doctor v3.4.0 Visual Agent
Runtime Observability; previous P95 stable line: Agent Failure Doctor v2.4.1
P95 Alignment & Missing Tracks Pack.

**Command groups:** diagnosis, repair planning, verification, collection, OCR evidence, visual runtime, patch proposal, fleet batch, and safe sharing.

Optional visual evidence: screenshot.png / dom_snapshot.html /
click_coordinates.json / ocr_excerpt.txt.

### Regulated Industry & Pure Visual Agent Evaluation

v3.6 adds local-only regulated workflow mock evaluation and full-chain agent
evaluation:

```powershell
failure-doctor regulated-eval --suite finance --out .\regulated_report
failure-doctor regulated-eval --suite government --out .\regulated_report
failure-doctor regulated-eval --suite healthcare --out .\regulated_report
failure-doctor regulated-eval --suite all --out .\regulated_report
failure-doctor full-chain-eval --input .\failed_run --out .\full_chain_report --include-safety --include-ocr --include-visual --include-regulated
```

These suites use synthetic local mock evidence only. They do not access real
finance, government, healthcare, patient, citizen, bank, or customer systems.
They are evidence-quality and shareability checks, not legal or regulatory
compliance advice.

### OCR & Document Evidence

Use OCR evidence for screenshots, PDFs, forms, tables, reports, RPA exports,
ecommerce SKU tables, ERP exports, and document-heavy automation failures.

```powershell
failure-doctor ocr-evidence extract --input .\screenshot.png --out .\ocr_report --provider mock_ocr
failure-doctor ocr-evidence compare --ocr .\ocr_report --dom .\dom_snapshot.html --out .\ocr_compare
failure-doctor ocr-evidence compare-vlm --ocr .\ocr_report --vlm .\vlm_responses.jsonl --out .\ocr_vlm_compare
```

Default provider is mock/local. No cloud OCR and no screenshot/PDF upload happen
by default. Cloud OCR requires explicit `--allow-cloud-ocr` and safety
evaluation. Sensitive documents are blocked or require sanitization. OCR
evidence is supporting evidence, not ground truth.

### Visual Agent Runtime Observability

`failure-doctor visual-runtime` diagnoses offline screenshot-driven agent runs:
screenshot capture, image payload cost, VLM observation metadata, action
grounding, coordinate clicks, DPR/viewport mapping, stale observations, OCR
semantics, and optional DOM/visual conflict.

It is local-only by default: no external VLM calls, no screenshot upload, no real
platform access, and no access-control defeat, identity-masking, protected
challenge automation, or behavior/pointer-path recipes.

```powershell
failure-doctor visual-runtime diagnose --input .\visual_run --out .\visual_report --no-dom
failure-doctor visual-runtime profile --input .\visual_run --out .\visual_profile
failure-doctor visual-runtime compare --baseline .\run_a --candidate .\run_b --out .\compare_report
failure-doctor visual-runtime adapt --source generic --input .\artifact_dir --out .\visual_run
failure-doctor visual-runtime validate --input .\visual_run --out .\validation_report
```

More docs: [docs/P98_LIMITS.md](docs/P98_LIMITS.md),
[docs/AGENT_FRONTEND_INVOCATION.md](docs/AGENT_FRONTEND_INVOCATION.md),
and [docs/safety_boundary.md](docs/safety_boundary.md).

P98 master gate passed with the safety compliance pillar included.

Advanced commands include `failure-doctor handoff`,
`failure-doctor agent-bootstrap`, `failure-doctor propose-patch`, and
`failure-doctor batch`.

**Core commands:** `collect` / `diagnose` / `plan` / `verify` / `run` /
`watch` / `sanitize` / `adapt` / `handoff` / `agent-bootstrap` /
`propose-patch` / `batch` / `visual-diagnose`

**Classic lifecycle:** `diagnose` / `plan` / `verify` / `run` /
`sanitize` / `adapt` -> `diagnose -> plan -> AI handoff / patch proposal
-> verify -> sanitize/share`

**P98 gate details:** `knowledge base -> coverage matrix ->
trace/cross-framework/training/composite/handoff/batch/sanitize/auto-collector
-> master gate`

## Distribution & Feedback

v3.8.0 is the current stable technical baseline. It adds local CI/CD gates and
runner templates on top of the local web console, regulated-industry mock
workflow evaluation, full-chain agent evaluation, OCR/document evidence,
offline visual runtime observability, and local-only safety and compliance
evaluation.

### Safety & Compliance Evaluation

```powershell
failure-doctor safety-evaluate --project . --out .\safety_report
failure-doctor collect --project . --preset auto --out .\failure_doctor_auto_report --auto-diagnose --auto-handoff --auto-sanitize --safety-evaluate
```

Safety boundary: local-only, evidence-only, no upload, and no active probe.
Safety boundary: no bypass challenges, no access-control evasion, no fingerprint
spoofing, no signature cracking, no challenge automation, and no private
remediation exposure.

- PyPI release runbook: [docs/PYPI_RELEASE.md](docs/PYPI_RELEASE.md)
- Active probe boundary: [docs/ACTIVE_PROBE_BOUNDARY.md](docs/ACTIVE_PROBE_BOUNDARY.md)
- Behavioral and Client Hints boundary:
  [docs/BEHAVIORAL_CLIENT_HINTS_BOUNDARY.md](docs/BEHAVIORAL_CLIENT_HINTS_BOUNDARY.md)
- JavaScript integrity boundary:
  [docs/JS_INTEGRITY_BOUNDARY.md](docs/JS_INTEGRITY_BOUNDARY.md)
- Canvas fingerprint boundary:
  [docs/CANVAS_FINGERPRINT_BOUNDARY.md](docs/CANVAS_FINGERPRINT_BOUNDARY.md)
- Visual and data-quality boundary:
  [docs/VISUAL_DATA_QUALITY_BOUNDARY.md](docs/VISUAL_DATA_QUALITY_BOUNDARY.md)
- 2-minute demo script: [docs/DEMO_VIDEO_SCRIPT.md](docs/DEMO_VIDEO_SCRIPT.md)
- Technical article draft: [docs/TECH_ARTICLE_DRAFT.md](docs/TECH_ARTICLE_DRAFT.md)
- Real user feedback loop: [docs/REAL_USER_FEEDBACK_LOOP.md](docs/REAL_USER_FEEDBACK_LOOP.md)

After PyPI publication, the target install command is:

```powershell
pip install agent-failure-doctor
```

For non-technical Windows users, double-click
`scripts/windows/Start-FailureDoctor-Diagnosis.bat` or drag a failed project
folder onto it.

Advanced v3.2 commands include `failure-doctor collect` and `failure-doctor watch`.

Advanced evidence inputs include `browser_runtime_report.json` and
`input_timing_summary.json`.
JavaScript/request-integrity evidence can be supplied as `js_integrity_report.json`.
Sanitized Canvas hash/session-count evidence can be supplied in
`browser_runtime_report.json`.

Agent frontend invocation:

```powershell
failure-doctor agent-bootstrap --target all --project .
```

This writes `.failure-doctor/AGENT_ENTRYPOINT.md` plus Codex, Cursor,
Claude Code, VS Code/Copilot, Antigravity, OpenCode, Qoder, Trae, WorkBuddy,
OpenClaw, Hermes, and generic agent workflow instructions.

Agent Failure Doctor uses a deterministic evidence-based diagnostic engine.
It does not claim to solve arbitrary failures, but it provides explainable
classification, evidence, fix plans, and before/after verification for known
automation failure patterns.

Applied scenario demos are local-only mock workflows for commerce automation,
live monitoring, content publishing, GUI data bridge, and ERP sync failure
diagnosis.

Spiderbuf-inspired challenge demos are local-only mock failure packs inspired
by public crawler-training challenge categories; they validate diagnosis and
safe next actions without accessing spiderbuf.cn or publishing private
remediation logic.

**Integration commands:** `failure-doctor collect-playwright` / `failure-doctor pack-logs` / `failure-doctor adapt`

## What You Get

```text
report/
|-- diagnosis.json
|-- diagnosis.md
|-- evidence.json
|-- input_summary.json
|-- issue_draft.md
|-- repair_suggestions.md
|-- codex_fix_prompt.md
`-- failure_doctor_report.zip
```

Agent Failure Doctor turns sanitized automation failure materials into a report
that explains what likely failed, what evidence supports the diagnosis, what
evidence is missing, and what to ask Codex or another coding assistant to
change next.

## One-Minute Start

Auto Capture:

```powershell
failure-doctor run -- python crawler.py
failure-doctor run -- pytest tests/test_listing.py
failure-doctor run -- playwright test
```

This writes a local run folder under `.failure-doctor/runs/<run_id>/`:

```text
.failure-doctor/runs/<run_id>/
|-- command.txt
|-- exit_code.txt
|-- stdout.log
|-- stderr.log
|-- environment.json
|-- detected_artifacts.json
|-- input_summary.json
|-- diagnosis/
|-- fix_plan/
|-- verification_hint.md
`-- shareable_failure_pack.zip
```

The generated `safe_to_share.json` defaults to `safe_to_share=false`; review and sanitize before sending a pack to anyone else.

Sanitize & Share Pack:

Sanitize a failed run before sharing it:

```powershell
failure-doctor sanitize .\.failure-doctor\runs\<run_id> --out .\shareable_failure_pack
```

This writes redacted logs, redacted network summaries, trace metadata only, a
redaction report, a review gate, and `shareable_failure_pack.zip`.

Raw `trace.zip` archives are not copied into the sanitized pack.

Put a failed run in a folder:

```text
my_failed_run/
|-- error.log
|-- console.txt
|-- network.json
|-- README.txt
`-- screenshot.png
```

Then run:

```powershell
failure-doctor diagnose .\my_failed_run --out .\report
```

The tool inventories inputs and uses this evidence priority:

```text
trace.zip > log > network.json > user description > screenshot metadata
```

When evidence is too thin, it should downgrade to `insufficient_evidence` instead of guessing.

## Minimal Demos

Proxy/network failure:

```powershell
failure-doctor diagnose .\examples\failed_runs\proxy_failed --out .\report_proxy
```

Strict mode locator conflict:

```powershell
failure-doctor diagnose .\examples\failed_runs\strict_mode_locator --out .\report_locator
```

Low-evidence screenshot-only run:

```powershell
failure-doctor diagnose .\examples\failed_runs\low_evidence_screenshot_only --out .\report_low_evidence
```

Native Playwright trace fixture:

```powershell
trace-doctor diagnose .\examples\realistic_playwright_traces\02_login_redirect_302\trace.zip --out .\report_login_trace
```

## Before / After Report

Report structure: conclusion / evidence / why / next action / Codex fix prompt

Before:

```text
page.goto: net::ERR_PROXY_CONNECTION_FAILED while opening https://example.test
```

After:

```text
Conclusion: network/proxy setup failed before the page loaded.
Evidence: Playwright reported net::ERR_PROXY_CONNECTION_FAILED.
Next action: check proxy settings, DNS, VPN, and CI network configuration.
Codex fix prompt: add trace/log capture and make proxy configuration explicit.
```

## Verify a Fix

```powershell
failure-doctor diagnose .\failed_run --out .\report
failure-doctor plan .\report --out .\fix_plan
failure-doctor verify --before .\failed_run --after .\rerun_after_fix --out .\verification_report
```

`verify` compares before/after evidence and reports whether the original failure
is resolved, unchanged, changed into another failure, or insufficiently
evidenced.

## AI Handoff & Patch Proposal

Turn a report into task packs that Codex, Claude Code, or Cursor can execute:

```powershell
failure-doctor handoff .\report --target codex --out .\ai_handoff
failure-doctor handoff .\report --target claude_code --out .\ai_handoff
failure-doctor handoff .\report --target cursor --out .\ai_handoff
```

This writes:

```text
ai_handoff/
|-- ai_handoff.json
|-- ai_handoff.md
|-- codex_task.md
|-- claude_code_task.md
|-- cursor_task.md
|-- affected_files.json
|-- validation_commands.md
|-- forbidden_actions.md
|-- token_budget_report.json
`-- ai_handoff_pack.zip
```

Generate a dry-run patch proposal without modifying source code:

```powershell
failure-doctor propose-patch --repo . --report .\report --out .\patch_plan
```

This writes:

```text
patch_plan/
|-- patch_proposal.md
|-- proposed_changes.json
|-- affected_files.json
|-- validation_commands.md
`-- patch_risk_assessment.json
```

`propose-patch` is intentionally proposal-only. It does not edit files, apply patches, run tests, or open pull requests.

v2.5 validation writes `validation/ai_handoff_validation.json`:

```text
20/20 Codex task files generated
20/20 Claude Code task files generated
20/20 Cursor task files generated
18/20 patch proposals generated
20/20 required sections present
20/20 concise token budget checks pass
0 forbidden outputs
```

## Batch Diagnosis / Fleet Mode

Diagnose many failed runs and get a fleet-level summary:

```powershell
failure-doctor batch .\runs --out .\batch_report
```

Input:

```text
runs/
|-- run_001/
|-- run_002/
|-- run_003/
`-- ...
```

Output:

```text
batch_report/
|-- summary.json
|-- summary.md
|-- failures_by_type.csv
|-- top_root_causes.md
|-- repeated_failures.md
|-- suggested_regression_cases.md
|-- repair_priority.md
`-- reports/
```

Fleet mode answers which failures repeat, which root causes dominate, which runs
should become regression cases, and which fixes deserve priority.

## P98 Controlled Maturity

v3.0 starts the P98 controlled maturity track. This is not an ecosystem score;
it does not count stars, external PRs, external issues, PyPI downloads, or
long-term community adoption.

Current P98 assets:

- [docs/P98_CONTROLLED_MATURITY_SCORECARD.md](docs/P98_CONTROLLED_MATURITY_SCORECARD.md)
- [knowledge_base/](knowledge_base/)
- [docs/CRAWLER_FAILURE_COVERAGE_MATRIX.md](docs/CRAWLER_FAILURE_COVERAGE_MATRIX.md)
- [validation/crawler_failure_coverage_matrix.json](validation/crawler_failure_coverage_matrix.json)

Knowledge-base commands:

```powershell
python -m tools.knowledge_base.validate_patterns
python -m tools.knowledge_base.search_patterns --query selector_drift
python -m tools.validation.run_crawler_failure_coverage_matrix
```

## Applied Scenario Demos

Local-only mock demos show how Agent Failure Doctor can diagnose failures in:

- hot product collection
- live commerce monitoring
- ecommerce listing automation
- authorized content publishing workflow
- GUI / RPA data bridge
- ERP-to-ecommerce sync

Run:

```powershell
python -m tools.validation.run_applied_scenario_validation
```

## Spiderbuf-Inspired Challenge Demos

`examples/spiderbuf_inspired_challenges/` contains local-only mock failure packs inspired by public crawler-training challenge categories:

- cookie/session required
- iframe extraction
- Ajax dynamic loading
- random CSS selector drift
- infinite scroll missing items
- rate limit 429
- API signature required
- browser fingerprint risk
- Selenium detection risk
- challenge page detected

These cases are diagnosis-only. They do not access spiderbuf.cn, do not include
private remediation details, and do not include access-control defeat steps.

```powershell
python -m tools.validation.run_spiderbuf_inspired_validation
```

## Integrations

Collect Playwright test-results into a failure pack:

```powershell
failure-doctor collect-playwright .\examples\mock_playwright_test_results --out .\tmp_failure_pack
failure-doctor diagnose .\tmp_failure_pack --out .\tmp_collected_report
```

Normalize a loose log folder:

```powershell
failure-doctor pack-logs .\examples\mock_raw_logs --out .\tmp_log_pack
failure-doctor diagnose .\tmp_log_pack --out .\tmp_log_report
```

Normalize a Selenium, Puppeteer, Cypress, Scrapy, requests, or httpx failure log:

```powershell
failure-doctor adapt .\examples\cross_framework_fixtures\selenium\no_such_element\raw --framework selenium --out .\tmp_selenium_pack
failure-doctor diagnose .\tmp_selenium_pack --out .\tmp_selenium_report
failure-doctor plan .\tmp_selenium_report --out .\tmp_selenium_fix_plan
```

Supported adapter frameworks:

```text
selenium | puppeteer | cypress | scrapy | requests | httpx | auto
```

Playwright remains the deepest native trace backend. Cross-framework adapters
normalize local logs and metadata into the same failure lifecycle; they do not
run those frameworks or connect to external platforms.

See [docs/INTEGRATIONS.md](docs/INTEGRATIONS.md) and [docs/GITHUB_ACTION_USAGE.md](docs/GITHUB_ACTION_USAGE.md).

## Validation Status

Current milestone: Agent Failure Doctor v3.9 Local Failure Knowledge Base Pack.

Previous stable line: Agent Failure Doctor v3.7 Local Web Console Pack.

- 131 source-ledger records with separated `real_public_issue`, `official_doc_pattern`, and `public_inspired_sanitized` labels
- 50 traceable real public issue records
- 100 Playwright Trace Doctor P95 fixtures
- 100/100 Playwright trace reasonable classifications
- 100/100 Playwright trace exact subtype matches
- 62 external public reference seeds
- 20 external public reference held-out records
- 20/20 external public reference reasonable classifications
- 20/20 external public reference actionable next actions
- 12 resolution validation cases
- 12/12 resolution statuses correct
- 18 applied scenario validation cases
- 18/18 applied scenario reasonable classifications
- 18/18 applied scenario valid fix plans
- 18/18 applied scenario verification statuses correct
- Playwright collector, generic log packer, browser-use adapter, and GitHub Actions usage docs
- v2.0 Auto Capture command wrapper: `failure-doctor run -- <command>`
- Sanitize & Share command: `failure-doctor sanitize <failed_run> --out <shareable_failure_pack>`
- Cross-framework adapter command: `failure-doctor adapt <input> --framework <framework> --out <failure_pack>`
- 100 cross-framework P95 fixtures across Selenium, Puppeteer, Cypress, Scrapy, requests, httpx, browser-use, and generic RPA
- 100/100 cross-framework P95 reasonable classifications
- 100/100 cross-framework P95 valid fix plans
- 0 forbidden outputs in cross-framework P95 validation
- 40 training challenge P95 local-only validation cases
- 40/40 training challenge reasonable classifications
- 40/40 training challenge valid fix plans
- 40/40 training challenge verification statuses correct
- 0 forbidden outputs and 0 private remediation leaks in training challenge validation
- 160 composite P95 strict local-only validation cases
- 160/160 composite primary classifications correct
- 160/160 composite repair-order checks correct
- 160/160 composite evidence graphs generated
- 0 forbidden outputs in composite P95 strict validation
- P95 Core Triad Gate: pass
- 3 composite showcase reports under `sample_reports/composite_showcase/`
- 10 external held-out public-source records
- 9/10 external held-out reasonable classifications
- 10/10 external held-out actionable next actions
- 0 forbidden outputs in generated reports/prompts
- GitHub Actions green across Ubuntu, macOS, Windows, plus Windows benchmark/smoke/safety

See [docs/VALIDATION_REPORT.md](docs/VALIDATION_REPORT.md),
[docs/EXTERNAL_DATA_SOURCES.md](docs/EXTERNAL_DATA_SOURCES.md), and
[validation/dashboard.md](validation/dashboard.md) for validation metrics,
limits, and boundaries.

## Reproduce Validation

```powershell
python -m tools.real_trace_generation.generate_real_trace_fixtures `
  --out .\examples\realistic_playwright_traces `
  --count 30 `
  --clean
python -m tools.validation.run_real_trace_validation
python -m tools.validation.run_playwright_trace_p95_validation
python -m tools.validation.run_external_public_reference_validation
python -m tools.validation.run_resolution_validation
python -m tools.validation.run_spiderbuf_inspired_validation
python -m tools.validation.run_training_challenge_validation
python -m tools.validation.run_cross_framework_p95_validation
python -m tools.validation.run_composite_diagnosis_p95_strict_validation
python -m tools.validation.run_p95_core_triage_gate
python scripts\validate_external_heldout.py
```

## Safety Boundary

This project is for local, sanitized failure diagnosis.

It is not:

- a challenge-solving tool
- an access-control circumvention tool
- a credential extractor
- a real-platform scraper
- a tool for unauthorized collection

For suspected platform risk cases, the intended output is identification,
routing, and compliance-oriented next steps such as reducing request volume,
using an official API, confirming authorization, contacting the platform, or
stopping unauthorized collection.

## Contributing Failure Cases

You do not need to write code. The most useful contribution is a sanitized
failure case: log snippets, trace metadata, network summaries, screenshot
metadata, and a short description of what happened.

Open an [External failure case issue](.github/ISSUE_TEMPLATE/external_failure_case.yml) and remove secrets before posting:

- passwords
- API keys
- cookies
- tokens
- authorization headers
- private screenshots
- private data
- personal data

Accepted input types include sanitized `error.log`, `trace.zip`, `console.txt`,
`network.json`, screenshot metadata, and `user_description.txt`.

If you allow it, a sanitized case may be assigned an `EXT-YYYY-NNNN` id, run
once with the current released version before rule changes, and added to the
external validation dashboard.

Templates and author-generated examples are not counted as external cases.

See [CONTRIBUTING.md](CONTRIBUTING.md),
[docs/external_validation_protocol.md](docs/external_validation_protocol.md),
[docs/REAL_TRACE_CONTRIBUTION_GUIDE.md](docs/REAL_TRACE_CONTRIBUTION_GUIDE.md),
and [docs/REAL_DATA_SOURCES.md](docs/REAL_DATA_SOURCES.md).

## Commands

Run all tests:

```powershell
python -m unittest discover -s tests -p "test_*.py"
```

Run smoke and safety checks:

```powershell
scripts\smoke_test.ps1
scripts\local_safety_scan.ps1
```

