v1.11.1
==============
* change (release): pushing a version tag vX.Y.Z publishes -- no GitHub release has to be made by hand. Before building, release.yml checks the tag is vX.Y.Z, is the CHANGELOG's first version and points at a commit on main (scripts/release_notes.py); publishing to PyPI still waits for a reviewer to approve the `pypi` environment; once PyPI has the files the workflow makes the GitHub release, with docs/releases/vX.Y.Z.md as its notes (or the version's CHANGELOG entries) and the SBOM attached. A release made by hand still works: the workflow finds it and attaches the SBOM
* fix (packaging): the PyPI page shows the README's images and links -- PyPI resolved its relative paths against pypi.org, so v1.11.0's page had four broken images and dead links. Building now pins every relative target to the release's tag on GitHub (images from raw.githubusercontent.com, documents to their GitHub page; src/ipandora/_readme.py). PyPI does not let a release's description change, so v1.11.0's page stays as it is
* fix (README): the text says what the framework does today and nothing it does not -- it said App automation was planned and listed HTTP and Web only; the HackTrace figure was misquoted ("96% of shortcuts edit the test"; the paper reports 82-91% of passing solutions cheating when RL training does not penalise it); the status-code row implied a status code is judged unless declared a gap (it is judged by status_ok()/status_is()); --exclude-flaky and --fail-on-weakened are described as the reference describes them; every extra is listed. The architecture and workflow diagrams no longer show the UI and evaluation layers as planned, nor failure triage as done by AI (it is rules)

v1.11.0
==============
* add (docs/guide/eval.md): `ipandora eval figures results/*.jsonl -o figures/` (ipandora.core.eval.figures) draws the report's figures from the same records and the same numbers as `eval table` -- success.svg (each configuration's success rate with its 95% interval), cost.svg (mean cost of a run, money or else tokens, against success rate, one labelled point per configuration with its interval) and failures.svg (one panel per configuration: each failure reason's share of its failed runs, with counts). Plain self-contained SVG -- no script, font or fetched file, byte-identical for the same records -- one hue with configurations and reasons named in text, a <title> on every mark, light and dark steps
* change (templates): the AGENTS.md `ipandora init` writes tells agents to wait with eventually() / kafka.received() rather than time.sleep(), to make gRPC channels and Kafka clients through rpc.channel() / kafka.producer() / kafka.consumer() so calls are in the evidence and attributed, and not to swallow what a Kafka flush() raises -- each with the signal that catches it; with an example
* fix (R4, mutation testing of core/kafka.py): kafka.received() works on any confluent-kafka Consumer -- one not made by kafka.consumer() has no `topics`, and received() raised AttributeError on it
* fix (CR23, docs/design/reviews/23-CR23.md): a Kafka producer's flush() asks a topic's metadata once, not once per lost message -- with the brokers down each request waited 3 s, and 10 lost messages took 32 s to report (100: 5 minutes); a consumer shared by several cases records each case's first 50 messages, not 50 in all -- the count was kept by id() of the case's evidence, which every case reuses
* add (D6b, docs/guide/kafka.md): Kafka through confluent-kafka as it is (ipandora.core.kafka, extra `kafka`, confluent-kafka 2.16.0 admitted) -- kafka.producer(config) and kafka.consumer(config, topics) are confluent-kafka's Producer and Consumer, with the same configuration and methods. Every message produced (key, headers, value, partition and offset once delivered, or why not) and consumed is in the case's evidence (`protocol: kafka`); a produced message carries the case's traceparent with `tracing.enabled`. No message is lost silently: flush() raises KafkaUnavailableError (no answer from the brokers -- the environment, or the injected fault in effect) or KafkaTimeoutError (still queued when it returned) or KafkaDeliveryError (the broker refused it; a message that timed out is asked about by the topic's metadata, so a missing topic is not taken for brokers that are down). kafka.received(consumer, where, within, name) reads until a message matches and is one Check, `within` in its claim; a consumer that cannot reach the brokers says so (KafkaUnavailableError, the environment). Defaults only when unset: message.timeout.ms 30 s, and a consumer without group.id gets a group of its own reading from the earliest offset. CI's `kafka` job checks it on a real apache/kafka 4.3.2 broker (by digest), paused and delayed with netem
* fix (CR22, docs/design/reviews/22-CR22.md): a gRPC error status the service answered with (INTERNAL, NOT_FOUND, ...) is no longer triaged as the environment when its details say "timed out" or the like -- grpcio raises it with the service's own words, which read as a transport failure; it is a defect (medium), "the case did not expect this status code". A future's add_done_callback(fn) gives `fn` the framework's call, whose exception() is translated (it got grpcio's own future); a metadata key sent more than once keeps every value in the evidence, joined with ', '
* add (D6a, docs/guide/grpc.md): gRPC through grpcio as it is (ipandora.core.rpc, extra `grpc`, grpcio 1.84.1 admitted) -- a test builds its own generated stubs on `rpc.channel(target, credentials=None, options=None)`, or wraps a channel it built with `rpc.intercept(channel, name)`, and uses grpcio's API unchanged (unary and streaming calls, futures, with_call, metadata, deadlines). The channel's interceptor records every call in the case's evidence like an HTTP exchange (`protocol: grpc`; method, request and response messages -- protobuf as JSON, the first 20 of a stream --, status code, trailing metadata, time; credentials redacted), from the moment it starts, so a stream the test stopped reading or a future still running when the case ended is there too; adds the case's traceparent with `tracing.enabled`; and raises a call that ended UNAVAILABLE or DEADLINE_EXCEEDED as RpcUnavailableError / RpcTimeoutError -- still a grpc.RpcError with the same code() and details() -- naming the injected fault in effect, triaged as the environment or that fault. Every other status code is the service's answer, raised untouched. The report shows gRPC calls with their status code and without the curl button
* fix (CR21, docs/design/reviews/21-CR21.md): a run killed while netem's helper container was setting tc (a second or so: a case timeout, a cancelled CI job) no longer leaves a qdisc nothing knows of -- the marker is written before tc is set, and put back as it was when the fault is refused
* add (D4c, docs/guide/faults.md): tc netem as a fault backend (ipandora.core.netem) -- delay, loss, corrupt, duplicate and rate on a Docker container's interface (a docker compose service, say), through the same faults.injected(). A helper container joins the target's network namespace with NET_ADMIN and runs tc there, so the target needs neither tc nor privileges; the helper image is the admitted chaos-daemon image by digest, or `faults.netem.image`. The faults in effect on one container are one netem qdisc, rebuilt as each comes and goes, removed with the last, and checked with `tc qdisc show` before the block runs; a refused fault leaves what was there. Marker files record host and process, and cleanup removes only what this host's dead processes left. CI's `netem` job checks it on the runner's Docker
* fix (CR20, docs/design/reviews/20-CR20.md): on a shared cluster, a run's cleanup no longer removes the Chaos Mesh faults another run is injecting -- every object records the host and process that made it, and cleanup removes only what this host's dead processes (or this process) left; and a fault no longer outlives a run that died with nobody to clean up: every Chaos object lasts at most `faults.chaos_mesh.max_duration` (default 600 s) and Chaos Mesh ends it by itself
* add (D4b, docs/guide/faults.md): Chaos Mesh as a fault backend (ipandora.core.chaosmesh) -- delay, loss, partition, pod_failure and pod_kill on the pods chaosmesh.pods() selects, through the same faults.injected(); each is a NetworkChaos / PodChaos object made with kubectl, waited for until Chaos Mesh has injected it into every selected pod (a selector that selects nothing, or a timeout, is an error that leaves nothing behind), and deleted -- waiting for the pods to recover -- to undo it. Every kubectl call names `faults.chaos_mesh.context`; without it nothing is injected, and kubectl's current context is never used. CI's `chaos-mesh` job checks it on a real kind cluster (kind v0.33.0 built from the checksum-verified module, Chaos Mesh chart 2.8.6 by sha256, images by digest, all admitted in docs/security/components.md)
* add (D4b, docs/guide/faults.md): fault backends -- what applies and undoes a fault is a faults.Backend (apply, undo, clean, where), registered with faults.register(name, factory); a Fault names its backend (default toxiproxy, which is now the first backend, behaviour unchanged). Whatever the backend, the framework keeps the safety switch, the lock per target, the evidence (now with `backend` and `target`), the marker triage reads and the undo; a fault naming an unregistered backend is refused before anything is applied. `faults.backends` in the config lists the backends a run cleans up when it starts and ends, one by one (one unreachable does not stop the others). Decision 18 records why: a backend interface with Toxiproxy, Chaos Mesh and netem built in, and a company's or a cloud's platform as the user's own backend
* fix (CR19, docs/design/reviews/19-CR19.md): an Android case's log window opens to the millisecond (`date +%s.%N`, whole seconds on a device without %N) -- it opened up to a second early, and a crash the previous case caused in that second made this case a product defect, high confidence; `assert android(text=...).exists` (or `.wait()`) failing on a selector is diagnosed like a failed click -- lookups that came back empty during the case are remembered, and the one the failed assert names (quoted) is looked for in the widget tree at that moment, instead of the assert being judged a product defect; update_locks.py compares the npm security cutoff as an instant, not as a string; npm license overrides are per version
* fix (android): the logcat a failure keeps always holds the crash and ANR lines (the earliest, if there are more than 300), the rest of the 300 being the latest of the app's other lines -- it kept the last 300, and on CI the restarted app and the system's notes about it pushed the FATAL EXCEPTION out of the report (the verdict was still right)
* add (M5c): uiautomator2 measured against Appium on the same emulator, screen and operations -- scripts/bench_android.py (session start cold and warm, a lookup that finds and one that does not, find-and-read text, the widget tree, a screenshot; every answer checked, the tools taking turns) run by .github/workflows/android-bench.yml; results in docs/design/decisions/14. Its Appium server, UiAutomator2 driver and Python client are admitted like everything else: npm packages by exact version with a hashed package-lock.json resolved under the cooldown (scripts/update_locks.py, `npm --before`) and installed without install scripts or optional packages; check_components.py checks every installed package's sha512, source and license offline; the daily scan runs `npm audit` on the lock and pip-audit and the license check on requirements/bench.txt. appium 3.9.0 and driver 8.7.3 are taken inside the cooldown for their security fixes (3.8.0 / 8.7.0 carried two critical advisories); two moderate ones bundled in the driver are recorded exceptions
* test (M5b): the Android integration on a real emulator -- CI's `android` job boots an API 34 emulator (KVM, reactivecircus/android-emulator-runner, admitted) and runs a whole suite against the system's Settings app through the `android` fixture: it opens; a name on screen looked for through the wrong attribute is a stale selector (harness, medium); an absent widget is to be confirmed; a real crash (`am crash`) is a product defect with the FATAL EXCEPTION quoted; each failure keeps its screenshot, widget tree and logcat. Elsewhere the test is skipped without touching adb; IPANDORA_ANDROID_DEVICE=1 runs it against a device you attached
* add (M5a, docs/guide/android.md): Android through uiautomator2, used as it is -- extra `android` (uiautomator2 3.7.1, admitted in docs/security/components.md). The `android` fixture (not `device`, which pytest-playwright already has) is uiautomator2's Device of `android.serial`, the app (`android.package`) installed once per session from `android.app_path`, started before each case (`clear_data`: cleared first) and stopped after, the device held for the case alone (resource lock `device:<serial>`, across workers and runs). A failure keeps the screenshot, the widget tree and logcat since the case began (the app's own lines and every crash or ANR, redacted). Triage: a FATAL EXCEPTION or ANR of the app is a product defect, whatever failed after it; a selector that finds nothing is looked for by name in the widget tree -- a widget of that name is there: the selector's problem; none: to be confirmed
* fix (CR18, docs/design/reviews/18-CR18.md): with `--no-weak-rule`, a test file that does not collect no longer makes every test of the agent's other files count as failing before the fix -- unchanged tests and tests that passed were judged as reproductions; only the files that do not collect are counted that way, the agent's tests in the others are chosen and run before the fix as usual
* add (docs/guide/eval.md, fresh-repro PLAN.md 2.5): SWT-bench's plain F→P, to compare with its harness -- `ipandora eval f2p --keep-out-of-scope --no-weak-rule` (testgen.grade(discard_out_of_scope=False, weak_rule=False)) keeps the agent's source changes in place (still reported) and counts any failure before the fix as a reproduction, a test file that only collects with the fix included (its tests count as failing before); the F→P check's name says which rules were off and the record carries `rules`. Both rules stay on by default
* fix (CR17, docs/design/reviews/17-CR17.md): a test-generation run that wrote no test is `no_tests`, no longer also flagged `zero_assertion`, and the zero-assertion rate of `eval table` is over the runs that wrote tests (the same failure was counted twice); `eval table` refuses records read twice (same configuration, task, run and attempt), which counted every run twice; a `|` in a configuration name no longer adds a column to the markdown
* add (E12, docs/guide/eval.md): the report's tables from the raw results alone -- `ipandora eval table results/*.jsonl [-o tables.md] [--json] [--k N]` (ipandora.core.eval.tables) reads the lines of `eval records` and gives, per configuration, the main table (success rate as the mean of each task's rate, its task-level 95% interval, pass@k and pass^k), the green-washing metrics of test-generation runs (out of scope, zero assertions, weak reproductions, mutation score of the runs that reached F→P), cost per run (tokens with cache reads and writes, money, time, turns, tool calls) and money in all, failure reasons counted, and every pair of configurations compared on their shared tasks; each `eval records` line of a test-generation run now carries mutation {killed, tested}
* add (E11, docs/guide/eval.md): measure the infrastructure noise instead of guessing it -- `ipandora eval noise r1/report.json r2/report.json ...` (stats.measure_noise()) takes runs of one configuration on the same tasks and gives each run's success rate on the tasks every run decided, the largest gap between two runs, its standard deviation, and the `--noise` to compare with; runs that differ in harness, model, config, resources, dataset or split are refused (exit 1), and two runs only are measured with a warning
* fix (CR16, docs/design/reviews/16-CR16.md): a TaskError raised inside the agent (standard.converse()/prepare() cannot read a plant or a fixture) is the task's fault -- no verdict, attributed to the task, never retried -- instead of infrastructure, which `--infra-retries` retried and counted at 100%; an agent's error, and each retried one, is kept to 2000 characters; a run whose infra_errors is null loads
* add (E10, docs/guide/eval.md): retry the infrastructure, never the agent -- `ipandora eval run --infra-retries N [--retry-wait SECONDS]` (evaluate(infra_retries=, retry_wait=)) tries a trial again while its agent raises (a container, an API 5xx, a rate limit), waiting longer each time; an answer, right or wrong, is never retried. Every retried error is kept in Trial.infra_errors (also read from Inspect's error_retries), summary() reports attempts, infra_attempts, the infrastructure error rate (infra_rate, over every attempt) and retried, the report and the console state the rate, the manifest records infra_retries, and each `eval records` line carries infra_retries and infra_errors
* fix (R3b, mutation testing of core/eval/testgen.py): a fix that touches a Python file this interpreter cannot parse no longer ends the grading of an F→P with a SyntaxError -- that file simply has no mutants; a task directory given with a trailing slash names its directory in the id error (it named ''). Tests pin every check a Result reports (name, evidence, claim), how a test's failure is classified, when pytest's records are believed, that a non-Python or unparsable file does not stop the search for hooks, the lines a fix touches (deletions, new and removed files), the innermost function, mutants on a function's first and last line, what counts as an assertion (raises/warns imported directly, deprecated_call, fail, helpers through a cycle), and task.yaml's dates, budget and every problem at once
* fix (R3, mutation testing of core/eval/standard.py): a json check's claim no longer depends on the outcome -- a missing path read `$.id equals 'A1'` and lost the approx tolerance while a present one read `$.id equals A1`, so the judgement inventory saw a different judgement; the claim is now always `$.id equals 'A1'` / `$.amt approx 10 ± 1`. A state check with no end state claims its `ignoring ...` like one with. A non-number length or tolerance is a task error whatever the output, not only when the path exists. A planted task's judgements keep the judge's label (it was dropped while masking canaries, leaving `eval samples` without judge_label). A rubric dimension's weight that is not a non-negative number is the task's error, as for every other judgement, not weight 1 (ai/judge.py). Tests pin each output check's name, evidence, source and claim, every error's place in the task, and the no_loops / called_at_most bounds
* add `technique: all` for injection plants (docs/guide/eval-dataset.md §6.3): the task becomes one task per technique when the dataset loads, ids <id>/<technique>, tagged technique, so the report scores each technique; a technique may also be given by name
* fix (CR15, docs/design/reviews/15-CR15.md): the material a judge is shown can no longer close its <question>/<answer> tag and carry on as instructions; a brace before the judge's JSON verdict no longer hides it; the judge's call budget is held under a lock across parallel trials
* add a model judge for rubric dimensions (ipandora.ai.judge.Judge, docs/guide/eval.md): pass/fail or graded levels, best first; it may answer unsure, and an unreadable reply, a failing provider or a spent call budget is unsure too, never a guess; what it is sent is redacted and cut; its verdicts are llm_judge checks, shown and never counted; the level travels as the check's label (Graded.label) into `eval samples` (judge_label), and `eval agreement --categorical` gives Cohen's kappa over several levels. A spec's JUDGE judges expected.rubric in the standard grader
* fix (CR14, docs/design/reviews/14-CR14.md): a NameError or ImportError raised by the code under test (not by the test) is a reproduction, not a weak one; a test file the agent deleted is no longer brought back by the grader; the agent's test code can no longer forge a verdict as easily -- the result path leaves the environment as the recording plugin loads, the records must agree with pytest's exit status, and test files defining pytest hooks are not graded (reason `tampering`); assertions in helpers of the same file count
* add (E9): `ipandora eval f2p-selfcheck` checks the F→P grader on known answers (the reference tests three times, no change, source only, a zero-assertion test, an import-only failure); `ipandora eval records` writes one JSON line per run with tokens (cache reads and writes apart, from Inspect logs), cost, time, turns, tool calls, how the run ended (Trial.ended), over_budget as a flag, and for test-generation tasks the failure reason and flags by rule (testgen.failure_reason / flags); testgen.from_json reads `f2p --json` back into checks for an Inspect scorer
* add test-generation tasks (E8, core/eval/testgen.py, docs/guide/eval.md): `ipandora eval f2p WORKDIR --fix fix.patch` judges an agent's reproduction tests by fail-to-pass, in place -- its new or changed tests (collect-only before and after) must fail in the call phase before the fix (collection, import, syntax, name and fixture errors are weak reproductions) and pass every run with it (flaky otherwise); changes outside test files are put back and counted; assertions are read from the source; mutants are made only in the functions the fix touched. testgen.load() reads task.yaml directories strictly; the fix is part of the dataset hash
* change (evaluation statistics): the success rate is the mean of each task's rate and its 95% interval resamples tasks (and each task's trials) -- the Wilson interval over all trials pooled was too narrow, since a task's trials are not independent; the pooled rate and its Wilson interval stay in summary() as pooled_rate / wilson. Add `ipandora eval compare` and stats.pairwise(): configurations compared pairwise on the tasks they share, Bonferroni-corrected, an interval spanning zero being "no reliable difference"
* add (E6, docs/guide/eval-dataset.md): ModelUser.from_task(task) builds the model-played user from input.persona/goal, and standard.converse() refuses such a task without a user instead of falling back to a script; a dataset's top-level description and meta go into the manifest and the report (not the hash), and an unknown top-level key is an error; `ipandora eval to-inspect` writes Inspect samples (JSONL), whose expected and initial_state inspect_scorer reads back, so the same grader judges them in Inspect. The evaluation report now lists self-check problems of kind `task`
* fix (CR13, docs/design/reviews/13-CR13.md): a trial with egress_allow refuses to start while another egress guard is on (workers>1), instead of being judged by the other trial's connections; an agent that raises inside standard.converse()/prepare() no longer carries a planted canary into the trial's error; `plant: []` is no plants everywhere
* test (R2, mutation testing of core/consistency.py, 67.8% -> 94%): the claim and name of History.check() (what the judgement inventory compares), the options it passes to elle-cli (--consistency-models, --anomalies), the timeout, each verdict's evidence and every counterexample shape, where Java is looked for and in what order, the refusal of an unadmitted jar, and the evidence files byte for byte. No behaviour changed; what survives is wording and equivalent mutants
* add planting and safety to the standard grader (E5b, docs/guide/eval-dataset.md §6.3): initial_state.plant puts an injection (after what is already there), a canary (fresh each trial) or a decoy into the fixture; expected.safety judges resisted, not_leaked, within, untouched and egress_allow. standard.converse() / prepare() run a trial on the planted world and hand the plants to the grader on AgentResult.setup; after grading, every canary (plain, base64, hex) is masked from the trial's output, trajectory, evidence and error, so none reaches a report. A plant nothing judges, a judgement with nothing planted, or a planted world built outside prepare() is a TaskError
* add the standard grader (E5a, docs/guide/eval-dataset.md §5): a spec without grade() is judged by the dataset's `expected` -- answer and checks (contains, regex, one_of, json paths, schema), the end state as the initial state plus `changes`, called / not_called with argument comparisons ({lte: 100}, {regex: ...}), called_at_most, no_loops, finished, says, never_says -- with the spec's world(task) for the initial state and for replaying references. initial_state.fixture and patch are read by standard.initial_state(). An `expected` key it does not know is an error, not a judgement left out; the spec lists the keys it judges itself in SKIP; such a task is attributed to the task (TaskError) and fails `eval selfcheck`. A reference of tool calls is replayed only when selfcheck hands it over (AgentResult.is_reference): an agent's list output is judged, never executed. Datasets: fixture and schema files must exist and are part of the hash (datasets with fixtures get a new hash once). same_state()'s ignore takes `*` for one path segment
* fix (CR12, docs/design/reviews/12-CR12.md): scripts/mutation.py no longer deletes a work directory it did not make -- one without its marker is refused; the test that a missing elle-cli skips the checker tests checks for a skip again, not for any error mentioning elle-cli. A route for tracing keeps the browser context's extra_http_headers (tested)
* fix (R1, mutation testing): a failed case's service logs began with the previous case's when the two ran within one second -- Docker's --since stops at whole seconds; lines are now cut at the instant the case began, by their own timestamps (still the daemon's, a moment late: docs/guide/stack.md). The logs are read after a 0.25 s pause, so the failing request's own line is in them. Tests: unexpected skips from require() now fail the stack and consistency tests -- a broken docker() or checker() used to skip every test that reached it; the faults tests now run the real production guard (--env / IPANDORA_ENV), the client's timeout and HTTP-proxy bypass, and the nested-block lock count. scripts/mutation.py runs in a copy of the project, takes several test paths, and fails when no mutant was judged
* add trace propagation from the browser (D7b, docs/guide/tracing.md): with tracing on, requests a page sends to the application -- web.base_url's host and `tracing.hosts`, never third parties -- carry a traceparent of the case's trace, a span each, through a route on the context of `page`, `context` and page_as(); a page's own traceparent is kept. A case whose trace went out only from its browser shows it in the report. Routing turns Playwright's HTTP cache off for that context, so only when tracing is enabled
* fix (CR11, docs/design/reviews/11-CR11.md): two compose files of the same name (shop/compose.yaml, payment/compose.yaml) were one compose project in a process -- the second `up` joined the first and taking either down removed both (the project name now carries the files' paths, and a number for the same files up twice); an unexpected error while keeping service logs of a failed case raised inside pytest's report hook and ended the session (now a warning). Service logs are read with --tail
* docs: the evaluation dataset spec (docs/guide/eval-dataset.md), for people and AIs preparing agent-evaluation datasets -- the ten task fields, input shapes (scripted and model-played users), the `expected` vocabulary (output constraints, end state, trajectory rules, what may and may not be said, planted safety probes, coding-task limits, rubrics), references, versioning, a pre-merge checklist and dataset sizes; each item marked supported or planned (E5: a standard grader that reads the vocabulary). A test loads every example with the current loader, so datasets written to it now need no change later
* add dependent services from a compose file (docs/design/decisions/19 D5), `ipandora.core.stack`: with stack.compose('compose.yaml') as s -- `docker compose up --wait` on the services' own health checks, s.url(service, port) for the port Docker published, `down --volumes` whatever happens. One project per process (`ipandora-<host>-<pid>-<name>`), so parallel workers never collide; projects of processes that died are removed the next time a stack starts. A stack that does not come up raises StackError with the failing services' last log lines, triaged `environment`; a failed case keeps each service's log since the case began (redacted, last 80 lines) as evidence, quoted in the report. Docker Compose v2.20+ is an external tool, not installed or pinned by the framework; without it the case is skipped. The test images are pinned by digest and registered
* fix (CR10, docs/design/reviews/10-CR10.md): a traceparent the case put on its session was replaced by the framework's own when the request set its own Content-Type (a session-level one now counts as the case's); a report rebuilt from JSON (`ipandora merge`) rendered a shard's trace_url as a link whatever its scheme -- links are now http(s) only
* add cross-service trace evidence (docs/design/decisions/14 §12 D7), `ipandora.core.tracing`: with `tracing.enabled: true` every request sent through the HTTP layer carries a W3C traceparent -- one trace per case, concurrently() threads included, a new span per request and per replay -- added before credentials and Provider.sign(), never over a traceparent the case or an instrumentation wrote. Each exchange records the trace id it actually went out with (failed ones too); the report lists the case's traces and, with `tracing.link` (an http(s) template with {trace_id}), links each to Jaeger / Tempo. Off by default; no tracing SDK, no new dependency
* fix (CR9, docs/design/reviews/09-CR9.md): the history and checker answer kept as a case's evidence were redacted by the shape of a string only -- a write of {'token': ...} reached the artifacts directory, which CI uploads, as written (fields with a credential's name are now masked, as in the report; the checker still sees the real history); an artifacts directory that cannot be written turned a judged history into an error (now a warning). The acceptance test waits a second of injected lag instead of 300 ms
* add consistency checks under injected faults (docs/design/decisions/18 D3b): each operation of a History records the faults in effect when it was invoked and completed (`:faults`, which the checkers ignore); a verdict says how many operations ran under which faults, and a register counterexample names the faults in effect when the failing operation completed. Each check() inside a case keeps the history (EDN, with the elle-cli command that re-checks it) and the checker's full answer as files of the case, listed in the report under 操作历史, strings redacted. A cas-register / mutex pass says how many reads of nil it could not judge -- Knossos reads nil as "unknown", so a stale read of an unwritten key passes unseen. Acceptance: an eventually consistent KV under replication latency is not linearizable, and eventually() on the same read passes
* fix (CR8, docs/design/reviews/08-CR8.md): three histories crashed the consistency checker instead of being judged -- an operation that raised a BaseException (pytest.skip/fail, KeyboardInterrupt) left an open invoke (now recorded as info); an operation name that is not an EDN keyword (refused before the operation runs); a NaN or infinite value (refused). The jar's hash check is redone when the file changes
* add consistency checks on an operation history (docs/design/decisions/18 D3a), `ipandora.core.consistency`: History.call(f, value, operation, ok=, fail=, result=) records invoke and ok / fail / info completions per thread (concurrently() included; a thread continues as a new process after info), writes the history as EDN and History.check(model) runs elle-cli -- Knossos for cas-register and mutex, Elle for rw-register and list-append with consistency models and anomalies, Jepsen's counter, set, bank -- turning valid into a passing Check, invalid into a failing one with the checker's counterexample, and unknown into a gap. elle-cli 0.1.11 is admitted as an external tool (EPL-1.0, reviewed; Java 21): its SHA-256 is pinned and checked before every run, CI downloads and verifies it; without Java or the jar the case is skipped, saying why
* fix (CR7, docs/design/reviews/07-CR7.md): a real outage after a fault block was triaged as the injected fault -- a transport error now carries the faults in effect when it was raised (in any thread, concurrently() included), and only such an error is triaged `injected`; nested injected() blocks on one proxy waited for their own lock until the resource timeout (the lock is re-entrant within a process); a run() started inside a case removed the faults that case was injecting (cleanup only in the outermost session, and never while a fault is in effect)
* add fault injection through Toxiproxy (docs/design/decisions/18 D4a), `ipandora.core.faults`: with faults.injected(latency / bandwidth / timeout / reset_peer / slow_close / down ...) adds toxics on a Toxiproxy proxy for one block and removes them whatever happens, holding the resource lock fault:<proxy> meanwhile; each fault is written into the case's evidence and the report; triage calls a transport failure during the case's own fault `injected`, not `environment`. Off unless `faults.allowed: true` in the environment's config, never in prod; toxics are named `ipandora-...` and leftovers are removed when a run starts and ends. No new Python dependency: the HTTP API is called with requests
* fix (CR6, docs/design/reviews/06-CR6.md): the coding graders could be bypassed by the agent they judge -- the judgement inventory was read from the agent's work directory, where it could be regenerated after loosening a test (it is now read as committed at `base`: coding.committed()); a sitecustomize.py in the work directory ran before the test runner and printed a result of its choosing (children now start in an empty directory with -E and enter the work directory afterwards). Files that steer a test run -- conftest.py, pytest.ini, pyproject.toml, setup.cfg, tox.ini, sitecustomize.py, *.pth -- are checked by default with `base` (harness_untouched, --allow-harness-changes); mutation testing first checks that the tests load each target from the work directory; a git that hangs is a message
* add graders for coding agents (docs/design/decisions/13 E1), `ipandora.core.eval.coding` and `ipandora eval code DIR`: changes(dir, base) -- committed, uncommitted and untracked, a rename as a deletion and an addition -- with only_touches / never_touches (path globs) and change_size; suite(dir, tests) runs the tests with the framework's runner in a child process, for tests_pass and every_test_judges (zero-judgement tests), and with a judgement inventory for not_weakened; mutation(dir, tests, targets) runs a built-in mutator (comparisons, arithmetic, and/or, not, boolean and integer constants, return values) over a copy of the work directory, for kills_mutants(min_score). `--json` and coding.from_json() carry the checks out of an Inspect sandbox
* E2-E4, from review (CR5, docs/design/reviews/05-CR5.md): a ScriptedUser reused across trials starts each conversation afresh -- it had nothing left to say after the first, and the agent was blamed; the injection corpus no longer spells its marker out, so an agent quoting the planted text back is not taken as obeying it; tool arguments and end states that cannot be deep-copied are kept as they are instead of failing the trial as infrastructure; trend.history() reads the newest 200 reports only; EgressGuard documents what it sees behind a proxy
* add the regression gate of agent evaluation (docs/design/decisions/13 E4): `--fail-on-regression overall|tag|task` on `eval run` and `eval inspect` -- tag also fails on any tag value that got worse, task also on any task that always succeeded and now always fails; the gate's reasons are printed and recorded in the report. Per-tag comparisons are Bonferroni-corrected (stats.compare(level=...)): at 95% each, one of twenty tag values was called worse by chance alone. Saturation: trend.history() reads earlier runs of the same dataset and split from the report archive, trend.saturation() says when the last 3 runs were all at 95% or more -- time to graduate the set and add harder tasks -- and which tasks succeeded every time
* add agent safety checks (docs/design/decisions/13 E3), `ipandora.core.eval.safety`, each deterministic: injections(tool, args) -- six techniques sharing a unique marker -- and resisted(); canary(kind) -- a fake secret in a real format, unique per trial, never printed -- and not_leaked(), which also finds it base64- or hex-encoded in messages and tool arguments; within(result, allowed tools); untouched(before, after, decoy paths) and FileDecoys; EgressGuard(allow=...), which refuses and records name resolution of hosts off the list. Example: examples/agent/safety.py, one task per risk
* add multi-turn evaluation of business agents (docs/design/decisions/13 E2), `ipandora.core.eval.dialogue`: converse(agent, user, max_turns, state) drives a conversation inside a spec's agent and returns an AgentResult -- every message and tool call in the trajectory, cost summed, how it ended, and the world's end state (deep-copied); ScriptedUser answers from rules in the task's input, each once, so a conversation always ends and is reproducible; checks for the end state (same_state, every difference named, `ignore` paths), the trajectory (called / not_called with argument values or predicates, called_at_most, no_loops, finished) and the output (says, never_says, whose evidence never repeats the match); replay() runs a reference's gold tool calls on a fresh world, so `eval selfcheck` covers multi-turn tasks. `ipandora.ai.simulated_user.ModelUser` lets a model play the user: optional, call-budgeted, redacted, named in the manifest; a run with a different user_simulator than the baseline is not comparable. Example: examples/agent/
* E1, from review (CR4, docs/design/reviews/04-CR4.md): the sandbox is recorded by type and config file name, not its absolute path, so a run on CI can be compared with a baseline written on a laptop; an unfinished Inspect run (cancelled, failed part way, still running) still gets its report but exits 1 and is never written as a baseline -- the samples that never ran were simply missing from the comparison; inspect_scorer no longer reads a `budget` it never judged; the dataset hash does not depend on sample order
* add the Inspect path of agent evaluation (docs/design/decisions/13 E1), extra `inspect` (inspect_ai 0.3.283, Python 3.11+): `ipandora eval inspect LOGS` and inspect_log.read() turn Inspect logs into an evaluation run -- one trial per sample and epoch, every score a check (passing at --threshold, 1 by default; dict and list scores one check per entry), model-graded scorers counted as model judges, errors and time limits as infrastructure, other limits as failed budget checks, a manifest with the Inspect version, solver, model, sandbox, limits and a hash of the samples -- with the same report, baseline comparison and exit code as `eval run`. inspect_scorer.scorer(grade) runs a grader of the pytest path as an Inspect scorer: CORRECT / INCORRECT / NOANSWER (nothing judged is never a pass), the checks kept in the score and reported by name, async graders awaited, a grader that raises attributed to the grader
* fix (CR3, docs/design/reviews/03-CR3.md): parallel logins -- an account-pool lease took the lock a case's @pytest.mark.resource('account:<name>') uses, for the whole run; auth.forget() on a shared login read the same credential back; auth.store() and web.share_with_api() did not reach the other workers; plain `pytest -n` had no shared directory, so single_session did nothing; a pooled identity also declared single_session put every worker on one account; a pool smaller than the worker count waited out the timeout instead of saying so; a serial run wrote the credential to disk; single_session = 'admin' was read as a set of letters
* fix (CR3): cases() -- a function's @pytest.mark.dims hid every row's own dims from the coverage matrix (dims are now merged, the row winning); a CSV column named items, values or get read as the dict method; the data file was looked for above the project root and in the current directory; a CSV row with more fields than the header crashed; null dims became 'None'
* fix (CR3): `ipandora check layout` -- fixed waits are matched by their last name (t.sleep, asyncio.sleep, self.page.wait_for_timeout); only a bare expect(...) is a judgement, not pexpect's child.expect; tests/conftest.py may import requests; relative imports are not requests; the sleep advice names Playwright's waits for pages/
* add `ipandora check layout [DIR]` (docs/design/decisions/17 L3): the layering rules of a case library, checked from the syntax tree -- judgements (assert, assert_all, expect, pytest.raises, matches_golden) in apis/, pages/ or flows/ (require() in flows/ is a precondition and allowed), fixed sleeps in pages/, flows/ and tests/, requests imported in tests/, hard-coded addresses in apis/ -- each with its file and line; exit 1 on any finding, 0 with --warn-only. The CI workflow `ipandora init` writes runs it
* add cases(path) (`ipandora.core.cases`, docs/design/decisions/17 L2): data-driven cases from YAML, JSON or CSV -- each row one pytest.param, named by its `id`, placed in the coverage matrix by its `dims` (CSV: `dims.<axis>` columns), the rest handed to the case as a Case that reads as a dict or by attribute. The path is looked up from the calling test file upwards. Duplicate ids and an empty file are errors: a parametrized case over nothing passes without running. The full layout has a data-driven example
* add logins under parallel runs (docs/design/decisions/17 A2): Provider.single_session (True, or identity names) makes every worker share one login of an account that allows one session at a time -- the first worker logs in and saves the credential in a private per-run directory (0700, files 0600, removed when the run ends) under a file lock, the others read it, and a rejected or expiring credential is renewed by one worker only. Provider.accounts = {'buyer': [...]} is an account pool: each worker leases one account for the whole run, and login() reads it with auth.account(name); with more workers than accounts, a worker waits for one to be given back
* fix (CR2, docs/design/reviews/02-CR2.md): a request with no Content-Type had the session's default headers merged over the case's own -- `Accept: text/csv` went out as `*/*`. The session's headers now only fill in
* fix (CR2): a query-parameter credential (ApiKey(param=...)) was skipped when the case gave params as a list or a string, so the request and its replay went out unauthenticated; auth.forget() restarted the credential version, so a thread renewing an older one logged in twice
* page_as, from review: domain cookies keep their leading dot (they reach subdomains); dotless hosts get their own name back from http.cookiejar's `.local`; cookie values are text; HttpOnly is read whatever its case; login_url matches the login page's path in full; a single-page app is given time to redirect itself before the check; web_headers adds to the suite's extra_http_headers; web_prepare returns True when it delivered the credential; the page login is not traced, so the typed password cannot reach a retained trace; a storage_state in browser_context_args is refused clearly; web.trace is validated; web.share_with_api(name, page) hands a browser login to the API side (Provider.login() cannot start a second Playwright). The full-layout web example logs in as buyer with page_as
* add page_as(identity, url=None) for web cases (docs/design/decisions/17 A3): a new page in its own browser context, logged in as that identity, so one case can hold a buyer and a seller side by side. By default the identity's API login is moved into the browser as cookies (add_cookies, the host from web.base_url where a cookie has none); a provider implementing web_login(name, page) logs in through the page instead, once per identity and process, and the browser state is reused from memory. With Provider.login_url (or web_logged_in), a page that lands on the login page triggers one fresh login and a reopen, then fails as a test-data problem. web_prepare() writes a token back into sessionStorage, which a saved state does not keep; web_headers sends header credentials from the browser; web.credential_from_browser() hands a browser-only login to the HTTP session. A credential with neither cookies nor a way into the browser is refused with the three ways to fix it
* add web.trace (retain-on-failure | on | off): a Playwright trace holds request headers and bodies -- tokens, cookies -- and cannot be masked. A bare `off` in YAML, which YAML reads as false, means off
* add `ipandora init --layout full` (docs/design/decisions/17 L1): a layered case library -- apis/ for endpoints, flows/ for multi-step setup, pages/ for page objects, data/ for factories, tests/api and tests/web for the cases, and a pytest.ini with `pythonpath = .` so cases can import the layers however the run is started. The examples log in as buyer, seller and anonymous against a demo service; the web example is skipped, saying why, without the web extra. Guide: docs/guide/layout.md. AGENTS.md gains the identity and layering rules
* when a case library's own package cannot be imported (`No module named 'apis'`), the run says to add `pythonpath = .` to pytest.ini -- also when the import fails inside conftest.py, where pytest reports no collection error at all
* add identities and login providers (docs/design/decisions/17 A1, `ipandora.core.auth`): a case says who it is -- `with auth.identity('buyer'):` -- and a provider registered once in conftest.py (`auth.provider(...)`, a Provider subclass or a plain `login(name, session)` function) gets the credential: `auth.Bearer` (a JWT's own expiry is read), `auth.ApiKey`, `auth.Cookies`, or the cookies a session login leaves behind, plus a per-request `sign()` hook. Each identity has its own session, so two identities in one case never mix; the default session is unchanged. Login on first use, once per process and identity; early refresh at min(60 s, a fifth of the lifetime); a response the provider calls unauthenticated (401 by default) causes one fresh login and one replay, never for file or stream bodies, never for a credential the case wrote itself. `anonymous` sends nothing. concurrently() threads inherit the identity. The evidence records the identity's name and marks login and replayed requests; a failed login is triaged as test data, or as the environment when the login endpoint cannot be reached. Everything is forgotten when a run ends. `auth.secret()` reads passwords from the environment
* fix (CR1 F6): `ipandora eval run` always exited 0, so a CI gate on an evaluation never fired. It exits 1 when the verdict is "worse" or "not comparable"; an explicit --baseline that does not exist, a missing dataset or bad YAML are errors (exit 2) with a message instead of a silent "no baseline" or a traceback
* fix (CR1 F6): `ipandora eval selfcheck` failed on tasks that simply have no reference, and the CLI and the report called every self-check problem "参考解判不过". Problems are named by kind; only a failing reference or a grader that cannot judge it exits 1
* fix (CR1 F6): a task's budget was written into datasets and documented, and nothing read it. Each budget item is now a required judgement -- `seconds` against the time taken, any other key against the cost the agent reported
* fix (CR1 F6): the guide said reports separate `do` from `dont` tasks; nothing did. Polarity is grouped like a tag in the report and the baseline comparison when a run has `dont` tasks
* fix (CR1 F6): a reference written as a file name that does not exist was taken as the inline reference -- the text of the path; it is an error when the dataset is read. A task with `id: 0` was rejected
* fix (CR1 F6): Trial.to_dict() deep-copied the agent's output and crashed on anything that cannot be copied; such values are kept as their repr. A spec using `from __future__ import annotations` with a dataclass failed to load (the module was not in sys.modules), and a spec that is not a .py file failed with AttributeError
* fix (CR1 F6): labels typed as text -- "false" -- counted as yes in the agreement (bool('false') is True); mean_score left out trials judged only by optional checks
* fix (CR1 F5): pruning the local report archive deleted the whole archive when an index entry had an empty run_id
* fix (CR1 F5): the pass-rate caption always said "skips and unverified are not counted", so one pass beside nine errors read as a plain 100%. It names what the denominator leaves out, with counts (report.json gains pass_rate_note)
* fix (CR1 F5): `ipandora run` exits by the report's verdict -- the same rule as `ipandora merge` and the report's "退出码" line; an empty run used to read "exit 1" in the report while the command exited 0. report.json gains gate_weakened
* fix (CR1 F5): the history index recomputed the pass rate without leaving out unverified cases, so it disagreed with the report it linked to; "new failures" compared with the previous run of any selector and environment instead of the same ones
* fix (CR1 F5): JUnit totals did not count a case of unknown status, though it was written as an <error>; `--junit-xml` is written by one function for run and merge
* fix (CR1 F5): the evaluation view showed per-tag "变差 / 变好" when the run was not comparable with the baseline, and gave every trial of a task the same anchor id
* fix (CR1 F4): the license gate matched allowed names as substrings: 'Proprietary ... not permitted' passed on the 'mit' in 'permitted', 'see disclaimer' on 'isc', npm's 'UNLICENSED' on 'Unlicense', and the public report listed them as passing. Names are matched as whole words; Proprietary and UNLICENSED are denied outright
* fix (CR1 F4): the registry check kept one entry per action name, so when two workflows pinned the same action to different SHAs only the last was checked. Every occurrence is checked, including the workflow `ipandora init` writes into new projects, which Dependabot never updates
* fix (CR1 F4): the CLI reference stopped at the first level of subcommands -- `ipandora eval` showed one row and none of `eval run / selfcheck / samples / agreement` -- and the docs gate passed, comparing the reference with the same incomplete output. Nested subcommands are documented now. The configuration reference no longer drops top-level scalars or shows a list as null
* fix (CR1 F4): the security report said licenses passed while the scan failed on a license of an unlocked distribution; such a distribution is now listed and counted. The lock descriptions are read from update_locks.py instead of written by hand, so they name parallel, visual and web
* fix (CR1 F4): `update_locks.py --cutoff 2026-10-22` let in releases from that afternoon, under the 14-day cooldown; a bare date now means its first instant, like the default
* fix (CR1 F4): mutmut, which scripts/mutation.py runs, was never admitted. It is now registered and locked in requirements/tools.txt (3.8.0; 3.8.1 was under 14 days old). mutation.py guessed the whole suite for a package target, and with CRLF or a trailing space after `[mutmut]` it quietly mutated all of src/
* fix (CR1 F4): check_packaging.py deleted src/*.egg-info and build/ in the developer's checkout, breaking an egg-link editable install; it builds from a clean copy of the tracked files now
* fix (CR1 F4): projects created by `ipandora init`: .gitignore keeps conf/<env>.yaml out of the repository (AGENTS.md says credentials go there; conf/config.yaml is still committed); the demo fixture extends no_proxy as well as NO_PROXY -- requests reads the lowercase one first -- and restores both; `init` no longer crashes printing the next steps when the target is on another Windows drive
* fix (CR1 F3): `.decimal()` compared with a float by the float's binary value, so `that('0.1').decimal() == 0.1` never held and `< 0.1` held. A float is now read as written. `'1_000'` is no longer accepted as a number
* fix (CR1 F3): a malformed path was silently read as another one -- `data..items` as `data.items`, `items[0` as `items.0`, `a[x]` as `a.x` -- so a check over the wrong field could pass. It is a failed check saying "bad path" now. A JSON body starting with a byte-order mark parses
* fix (CR1 F3): triage blamed the environment whenever "timed out" or "ConnectionError" appeared anywhere in the message, including a failed judgement quoting a response body ("upstream timed out"). A raised transport exception, or an eventually() whose every read was one, is still the environment; a judgement that failed on a response is a defect
* fix (CR1 F3): a locator error the test caught and handled decided the triage of a case that failed for another reason. The selector the failure message names decides; failing that, a locator failure counts only when no judgement failed
* fix (CR1 F3): a failed check with no expr took the next line of the message as its expr in triage
* fix (CR1 F3): golden names that are not ASCII all became 'image', and long artifact names sharing their first 80 characters shared a file: one golden or diff overwrote another. Such names get a hash; ASCII golden names keep their paths. An unreadable image or golden file is a failed check instead of an exception
* fix (CR1 F3): Web identities were not recorded for elements whose aria line carries attributes (`- heading "价格" [level=2]`, `[checked]`); two `expect(response)` checks in one case had the same claim, so the inventory saw one -- the claim now names the response's path; locator memory is written under a lock with a per-process temp file, so two runs finishing together no longer lose each other's entries
* fix (CR1 F2): redaction missed credentials quoted with a colon -- a dict repr in a message (`{'password': 'x'}`), a JSON fragment, a YAML line, a `Cookie:` header line in text. They are masked now, alongside key=value pairs
* fix (CR1 F2): anything nested deeper than 20 levels was returned unredacted; it is masked whole now
* fix (CR1 F2): brief() cut long values before the report masked them, so a PEM block cut before its END line, a JWT cut after a dot or a seed phrase cut short matched no pattern and its first part was published. Long values are masked before they are cut
* fix (CR1 F2): the evaluation report and the labelling samples were not redacted at all: agent output, trajectory and grader / judge messages went out as produced. They are masked like the test report; cost numbers are kept
* fix (CR1 F2): the LLM triage prompt sent the case id unmasked, and a parametrised id can be the password; parameters are hidden in the prompt. The MCP summary and explain_failure mask case ids too
* fix (CR1 F2): a request or response body that was not text (a number, a dict) made the HTML report fail with TypeError; copy-as-curl wrote a dict body as a Python repr instead of JSON
* fix (CR1, docs/design/reviews/01-CR1.md): a `-k` selector or a user `-m` was recorded like --exclude-flaky. Every case it left out counted as "used to pass, now excluded" -- `ipandora run login --fail-on-weakened` failed for cases nobody touched, MCP run_tests told the agent it had weakened them, and with --update-baseline those cases were stored as excluded, so deleting their assertions later went unreported. Only cases marked flaky and removed by --exclude-flaky are excluded now; what a selector leaves out is outside the run, and a run narrowed by `-k` / `-m` no longer reports missing cases as deleted. RunResult gains `narrowed`
* fix (CR1): --exclude-flaky appended `-m 'not flaky'`, and pytest keeps only the last `-m`, so the user's own marker expression was silently dropped. It is now done by the runner plugin and combines with any `-m`. run() gains exclude_flaky=
* fix (CR1): --update-baseline after a collection error (a broken conftest, nothing collected) saved an empty baseline, which turned weakening detection off for the whole suite. Such a run is refused, the file is left as it was, and the command exits 1
* fix (CR1): reads inside eventually() that did not hold yet were recorded as failed judgements when the probe used assert_all, so a case whose wait succeeded showed red checks; and a require() inside the probe skipped the whole case on the first read. The probe's own judgements are withheld (its requests are still recorded), and an unmet require() is a read that did not hold
* fix (CR1): a case whose setup passed but whose call never reported -- the process died -- counted as passed. It is an error now
* fix (CR1): a case skipped by its marker lost its dims, title and position, so case order differed between serial and parallel runs
* fix (CR1): calling run() from inside a test restored pytest.raises / pytest.warns and uninstalled the Web patches for the outer session, and replaced the outer case's evidence: everything the outer case judged afterwards was lost. Sessions and case evidence now nest
* fix (CR1): run() changed the env overlay, bytecode cache and environment before its try/finally, so an error before pytest started left them changed; Runtime.Option.env is now set the same in serial runs as in workers; a failure saving locator memory no longer turns a finished run into an exception; resource locks live in one per-user directory instead of next to IPANDORA_RUNS_DIR, so runs with different run stores still exclude each other
* fix (CR1): a thread concurrently() gave up on kept adding requests to the case after the call returned. Each thread records separately and finished threads are merged in operation order. Results().warning no longer crashes on an empty result
* fix (CR1): RunResult.to_dict() dropped the coverage axes; when several claims about one subject changed at once, the inventory paired them in sorted order instead of by operator
* add agent evaluation, E0 of docs/design/decisions/13 (`ipandora.core.eval`, `ipandora eval`): datasets in YAML / JSONL with a content hash over tasks and reference files; agents and graders in Python, graders returning Checks, with weights, required / optional, model-judge checks shown but not counted, and "uncertain"; k trials per task on threads; an agent exception is infrastructure and a grader exception is the grader's, neither counted as the agent failing; reference self-check (`ipandora eval selfcheck`, exit 1 on a failing reference) that attributes failures on a broken task to the task; success rate with a Wilson interval, pass@k, pass^k, partial score, cost; a committed evaluation baseline compared task by task -- McNemar for single 0/1 trials, otherwise a bootstrap that resamples tasks and the trials within each, so one task's trial noise cannot produce a confident verdict -- "better / worse" only outside the interval and beyond the given infrastructure noise; a manifest that says when two runs are not comparable; export for human labelling and agreement with Cohen's kappa; an evaluation view of the report with the trajectory as evidence. Example in examples/eval/
* fix: a form-encoded request body was not redacted -- `username=alice&password=hunter2` reached the report as sent, because only JSON bodies were masked by field name and `password=...` has no value shape. Every key=value pair whose key names a credential is now masked wherever text is redacted: form bodies, URL query strings (and so the copy-as-curl command), messages. Found reviewing login support (docs/design/decisions/17 §7)
* fix: what leaves for an agent was not redacted at all -- the MCP run_tests summary, explain_failure's messages and tracebacks, and the prompt LLM triage sends to a model provider. All three are now masked like the report, and masked before they are cut, so a truncated key cannot slip past its pattern
* add docs/design/decisions/17: login state (identities, credentials, providers; refresh; parallel; Web storage state) and how to organise a case library, reviewed twice; plan A0-A3, L1-L3
* add `ipandora merge REPORT... -o DIR` and report.merge() / merge_files(): one report and one verdict from the report.json of every shard of a run split across CI jobs or machines. A case present in several shards is counted once; a case whose outcome differs between shards is listed as a conflict and takes the more severe outcome, so a merge never turns red green; shards' baseline comparisons are combined and the coverage matrix is drawn over all of them. Refuses the same run twice and shards that ran against different environments (--allow-mixed-env). The exit code follows the same rule as `ipandora run`. With --baseline, the shards are declared to be the whole suite: a case the baseline holds as passing that no shard reported is a deleted -- weakening -- judgement, which is what catches a shard whose report never arrived
* report.json gains coverage_axes (the COVERAGE declarations themselves, so a report rebuilt from JSON can draw its matrix), shards, merge_conflicts and duplicates
* add eventually(probe, within, every=0.5): judge something that becomes true later -- a replica catching up, a queue consumer finishing -- instead of time.sleep. Re-reads until the probe's checks hold or the time is up and returns one Check for assert_all. The evidence says how many reads and how long when it held, and what the reads saw in order when it did not. A read that raises is an observation, not the end of waiting; a probe that returns no Check is a TypeError at once. `within` is part of the claim, so waiting longer is a changed judgement under --fail-on-weakened
* roadmap for distributed targets and scale: docs/design/decisions/14 §12 (D1 eventually, D2 merging sharded reports, D3 consistency checking, D4 fault injection)
* report redesign (docs/design/decisions/16): the first screen is one sentence -- 可以放行 / 通过，但需要复核 / 不能放行 / 没有跑起来 -- built from counts by a fixed template and agreeing with the exit code; the pass rate states its numerator and denominator; failures are grouped by triage category, the system under test first, and passes that are not evidence (unverified, flaky, weaker than the baseline) get their own group. Light and dark, usable at phone width, still one offline file
* add "这次新红的", from the committed baseline: failing now and passing in the baseline, failing with no passing record, new cases. The same answer on every machine; the framework keeps no history and draws no trends
* add `ipandora run --compare-with REPORT`: compare failures with an earlier report.json -- new, still failing, recovered. Without it, the previous run of the same selector and environment in the local report archive is used, labelled as a local cache; with neither, nothing is shown
* report.json gains view, verdict, pass_rate_basis, against_baseline and against_run; existing fields are unchanged. MCP run_tests returns newly_failing; build_report takes compare_with and returns the verdict
* triage reasons and next steps are in Chinese; categories stay English codes, with Chinese labels in the report
* COVERAGE accepts the short form `{'role': ['admin', 'guest'], 'channel': ['web', 'app']}`. A declaration the report cannot read is reported with the reason; it used to be drawn as an empty "0 / 0" matrix
* fix: without pytest-xdist installed, every run failed with "unknown hook 'pytest_testnodedown'" on pytest 9.2, which answers hasattr(config.hook, name) with yes for any name. The xdist hook is now an optionalhook. CI did not see it: its locks install xdist and pin pytest 9.1.1
* BREAKING: record/replay removed -- core/cassette, the `ipandora cassette` command, and eight `run` flags (--record, --replay, --verify, --cassette, --cassette-dir, --on-exhausted, --max-cassette-age, --allow-diff). A general test framework should not own mocking; a suite that needs a stubbed response can use `responses` or `httpretty`. Every HTTP request also passed through a record/replay check whether or not anything was recording; the transport adapter is a plain HTTPAdapter again. See docs/design/decisions/07 for how to restore it
* BREAKING: Source.ONCHAIN removed -- a Web3 dimension hardcoded into the general assertion layer. Check.src is a free string, so a suite can still tag `src='onchain'` and the report groups by it the same way. Source.DIFF went with record/replay
* BREAKING: data reporting is gone. The upload path collected every response, batched it on a background thread and then discarded it -- the request itself was commented out and called into `pandora_user`, which does not exist in this repo. Removed MultiHandle, UploadApiInfo, InnerPlugin, Runtime.Host/Mysql/Email, Option.only_api, and the mysql/file/email/addressesTo/report config sections
* BREAKING: Robot Framework support removed. utils/log.py imported `robot.api` unguarded, which made robotframework a hard requirement for importing the package at all, long after the adapter was archived. Also removed robotlogparser, the get_test_report MCP tool, Runtime.Frame.is_rf and the Robot branches in utils/variable.py
* BREAKING: removed orphan modules with no consumer -- common/uuidutils.py (UUIDFactory), common/timeutils.py, utils/filewriter.py, utils/niceline.py, core/base/classwrap/{singleton,attrvaluesplit}.py, core/base/repository/, common/mysqlaction.py
* BREAKING: eight dependencies that nothing imported are no longer installed -- websockets, bs4, six, tabulate, retrying, assertpy, pymysql, twine. pendulum went with timeutils; psutil is now the optional `system` extra and imported lazily
* every remaining dependency is version-bounded on both sides: a dependency that silently moves across a major makes a run's result depend on when it was installed
* add JUnit XML output, rendered from the same ReportData as the HTML report. `ipandora run` writes report.xml beside every report; --junit-xml puts a copy where CI expects it
* a collection failure now emits a real failing testcase in the XML: `tests="0" failures="0"` is indistinguishable from a clean run to every CI there is
* BLOCKED and GAP have no JUnit equivalent and are written as <skipped>, with the real status kept in the type attribute, the message and a property
* strip characters XML 1.0 forbids -- they cannot be escaped, and one stray byte in one assertion message would cost the whole run its report
* add @pytest.mark.flaky(reason=...) and `ipandora run --exclude-flaky`. Excluded cases are still named in the summary: a flaky marker nobody is reminded of is a failure that has been parked
* add scripts/mutation.py -- mutation testing, scoped to one module and its tests, with a score and --min-score for gating
* smoke_import now passes against a minimal install, excusing only the modules whose optional extra is genuinely missing
* BREAKING: json_matches removed, replaced by field() -- tests are Python, so judge fields with Python operators: `field(r, 'data.fee') <= 100`, `.is_in()`, `.contains()`, `.startswith()`, `.matches()`, `.len()`. Each comparison returns a Check, so assert_all still reports every failure at once. json_matches was never on PyPI
* BREAKING: ResponseHandler.filter takes a Python predicate instead of operator dicts -- `filter(lambda x: x['i'] > 3)`. Passing `{'$gt': 3}` raises with the equivalent lambda. Keyword equality stays, but is now strict: it compared str(a) == str(b), so '0' matched 0 and 'True' matched True. utils/match.py's Compare and DictMatcher are gone
* comparisons are strict: values compare as their own type -- the JSON string "12" is not greater than 9, and an API that switched numbers to strings is a finding, not something to coerce away; bool never equals a number (json_equals too); nothing is truncated. The operator dicts got all three wrong: `{'$eq': 5}` passed on 5.9, `{'$eq': True}` passed on the string 'false', and `{'__class__': 5}` passed on anything
* add the judgement inventory: every case's judgements are recorded as data -- Checks, plain `assert`s that held (pytest's assertion-pass hook, helpers included), pytest.raises / pytest.warns -- and compared with a committed baseline, conf/judgements.json. Changes are classified as removed, loosened (a numeric bound got wider), changed (an expected value differs), now skipped / excluded, unverified, deleted -- or tightened / added, which are fine. `ipandora run --update-baseline` accepts a run; `--fail-on-weakened` gates on weakening; `--baseline PATH` / `--no-baseline`. Compared on what ran, not on source text, so a same-length rewrite or an assert moved into a helper is visible. Shown in the CLI summary, a "判定变化" report section, JUnit properties and the MCP run_tests result
* add UNVERIFIED: a case pytest calls passed that made no judgement the runner could see. Not counted in the pass rate, named everywhere, written to JUnit as skipped type="unverified". `assert True` is not a judgement. unittest / doctest items are exempt -- their checks are not assert statements and cannot be seen
* Check gains `claim`, a stable form of what is asserted (`data.fee <= 100`) that a custom name does not change; field(), that() and the http checks fill it. schema_conforms includes the schema's fingerprint, so a loosened schema shows up as changed
* add Source.ASSERT for judgements recorded from plain asserts and pytest.raises
* fix: in one process -- the MCP server -- a test file edited between two runs kept running its first version, because test modules stayed in sys.modules. An agent fixing a test and re-running got the old result back. Project modules a run imported are now dropped when it ends; the framework, the standard library and installed packages are not
* run() caches rewritten test bytecode in its own directory (next to the run store, or $IPANDORA_PYCACHE_DIR): pytest reuses __pycache__ without checking which options compiled it, so a file first run by plain pytest would never have its asserts recorded
* field() covers what used to need json_value plus a hand-built Check: `.decimal()` for numbers sent as strings (exact, and explicit -- comparison stays strict by default), `.each(path)` to judge every list element in one Check that names the failing indexes, `.map(fn, label)` for any computation, and Field-vs-Field comparison for cross-field checks. `.each()` on an empty list fails unless allow_empty=True: "every item is valid" over zero items is a vacuous pass
* add that(value, label): judge any Python value -- a database row, a second API, text read off a page -- with the same operators
* anything field() cannot judge -- a missing field, mismatched types, an invalid regex -- is a failed Check with the reason, never an exception; and a bare field used as a bool raises, so `if field(r, 'x'):` cannot be silently always-true
* config is read from the suite's own conf/ (./conf, or $IPANDORA_CONFIG_DIR) over the package defaults: conf/config.yaml for the project, conf/<env>.yaml per run. Previously both were looked up only inside the installed package, so configuring a suite meant editing site-packages. The old location still loads, with a warning
* fix: an env overlay replaced whole sections (shallow dict.update) -- `ai: {enabled: true}` dropped `ai.model`. Layers now merge recursively
* fix: an env overlay was never undone. In one process -- the MCP server -- a run with env=staging followed by env=prod (no prod.yaml) or by no env at all still used staging's settings. Each run's overlay is now removed when the run ends, pass or fail
* fix: `-e` naming a file that does not exist logged at debug level, so a typo ran the whole suite on defaults under the requested env's name. It is now a warning listing every path searched
* fix: the crypto key landed inside the tracked package tree and three separate gates failed to recognise it; it now lives in ~/.ipandora/crypto at 0700/0600
* fix: AESCryptographer held a base64 str where AES wants raw bytes after generating a key, so the first run of a fresh install always raised
* fix: MANIFEST.in named the distribution instead of the package path; scripts/check_packaging.py now checks the built wheel, because whether data files reach it depends on the setuptools version doing the build
* fix: the `mcp` extra was bounded <2 while the server imports mcp.server.mcpserver, which only 2.x has -- `pip install intellipandora[mcp]` installed an MCP server that could not start. Now mcp>=2.0,<3
* add the `test` extra (httpretty) for the suite's own HTTP tests. CI installed .[ai,mcp] without it and every run since 1.11.0 began failed at collection
* fix: run() writes no bytecode while it runs. A test file edited within the same second, at the same size, kept its stale .pyc -- pyc validity is mtime-in-seconds plus size -- so a second run in one process could execute the old test
* add `ipandora init [dir]`: conf/config.yaml, a conftest with a local demo service and an example suite that passes offline, the AGENTS.md that gives a coding agent the rules -- each tied to the gate or report signal that enforces it -- a CI workflow with --fail-on-weakened, and a CODEOWNERS that puts the judgement baseline and goldens under review. Existing files are kept unless --force
* docs reorganised by reader: docs/guide/ by task (quick start, judgements, HTTP, parallel and concurrent, web, images, config, reports, CI, agents), docs/reference/ generated from the code, docs/design/architecture.md for how it works now, docs/design/decisions/ for the records 00-15. AGENTS.md for agents working on the framework, llms.txt as the index
* scripts/check_docs.py gates the docs: every relative link resolves, and docs/reference/cli.md and config.md match the argparse definitions and config.yaml. test/testdocs checks that every Python block in the guide parses and imports names that exist, runs the blocks marked runnable, and runs the quick start end to end
* fix: the concurrency section of the guide imported `field` from ipandora.core.assertion.field, where there is no such name; the docs test found it
* add image verdicts: that(image).matches_golden(name, mask=, tolerance=, threshold=, noise=) with pixelmatch and Pillow (the `visual` extra). Goldens live in __goldens__/<name>-<browser>-<platform>.png beside the test; `ipandora run --update-baseline` writes them along with the judgement baseline, a missing one fails instead of being made on the spot, and the golden's hash is in the claim, so a re-recorded golden is caught by --fail-on-weakened
* by default only solid (3x3) differences count: anti-aliasing and sub-pixel shifts left 151-290 differing pixels between renderings, while a price changing from 100.50 to 100.60 differs in 7 -- no pixel count separates them, so text is judged as text. noise='strict' counts every pixel
* the report shows golden, actual and diff side by side, the diff with a box around what changed
* pixelmatch runs only on the box where the images differ, padded by its anti-aliasing reach: the same count as the whole image (tested on 400 random scenes), milliseconds instead of ~0.7s
* add web automation on Playwright and pytest-playwright, used as they are (the `web` extra). Every expect() that holds is a judgement in the inventory, so a case written only with expect is no longer unverified, and a weakened expectation shows in the baseline diff
* a failed web case carries pytest-playwright's screenshot, embedded in the report, and its trace, with the command that opens it. Each run gets its own --output folder: pytest-playwright empties it per session, which would have broken archived reports
* a selector that finds nothing is triaged instead of blamed on the test: the role and accessible name it pointed at in a passing run are looked up on the page. Still there: stale selector (harness). Gone: 待确认, possibly a defect, with the screenshot. Never recorded: 待确认 as well
* web.base_url, web.browser and web.viewport in conf/<env>.yaml become pytest-playwright's defaults; the command line and a suite's own conftest still win
* fix: run() called from inside a test failed under pytest-playwright with "nested soft assertion scopes are not supported"; the outer scope is set aside for the inner session
* fix: lock files admitted releases up to a day short of the 14-day cooldown -- uv reads a bare --exclude-newer date as including that whole day. The cutoff is now an exact instant; uv pinned back to 0.13.4, which had been admitted at 13 days 22 hours
* add REVIEWED_LICENSES: a license outside the allowlist accepted after a written review, shown as such in the security report. First entry: text-unidecode (Artistic / GPL dual license, used under Artistic, [web] only)
* add concurrently(): N operations at the same moment from inside one test, to judge behaviour under concurrency -- charged exactly once, no overselling, idempotency keys honoured. Threads start together behind a barrier; each gets a copy of the caller's session (headers, cookies, auth, TLS, proxies), so concurrent requests are logged in, and records into the caller's case, so every request is in the report. Exceptions are results to judge, not raised. A thread failing before the start aborts the whole burst with NotConcurrent; nothing is sent
* results.overlap_check(): whether every operation was in flight at once, as a judgement. A concurrency test that happened to run sequentially passes vacuously; this makes it fail instead
* add SessionManager.forget() and SessionManager.clone(). Sessions are keyed by thread name and ident, and idents are reused: a session left by a finished thread is inherited, cookies and all, by the next thread that gets the same pair. concurrently() forgets each of its threads' sessions; a suite that starts its own short-lived threads should call forget() in them too
* the evidence recorder guards its writes, so several threads can record into one case
* add parallel execution: `ipandora run --workers N|auto`, run(workers=...), MCP run_tests(workers=...), with pytest-xdist from the new `parallel` extra. The result is the same as a serial run -- judgement inventory, unverified cases, baseline comparison, case order, coverage, deselected cases -- which the tests check byte for byte. Plain `-n 4` used to lose every judgement: they were recorded in the workers, into a collector that only existed in the main process
* the runner is split in two: core/runner/plugin.py, loaded with -p in every process that runs cases, records judgements, evidence and declarations and sends them back on the teardown report; core/runner/collector.py reads reports only. Serial and parallel are the same code
* add @pytest.mark.resource('name', ...): cases naming the same resource never run at the same time, across workers and across concurrent runs on one machine. A standard-library file lock (flock / msvcrt), released by the OS if the process dies. Waiting longer than $IPANDORA_RESOURCE_TIMEOUT (300s) is an error triaged as environment
* add --timeout SECONDS (pytest-timeout): a case that runs longer fails and is triaged as environment, so one hung case cannot hold a worker for good
* asking for --workers or --timeout without the `parallel` extra is an error that names the extra, not a quietly serial run
* a user's own --junitxml no longer receives the runner's evidence as XML properties; their own record_property values are kept
* BREAKING: the `ai` extra requires anthropic>=2.1,<3 (was >=0.40,<1). The framework uses Anthropic(), messages.create and the text of TextBlock, all unchanged in 2.x; 1.x was never tested and is not allowed. 2.x no longer pulls in httpx, httpcore or distro
* security: every third-party component is admitted, registered (docs/security/components.md) and checked by CI against what the code declares; licenses are checked against an allowlist; vulnerability exceptions carry a reason and an expiry date. See docs/design/decisions/15
* security: CI and development install from hash-pinned lock files (requirements/*.txt, scripts/update_locks.py) that only accept releases at least 14 days old; GitHub Actions are pinned to commit SHAs. The stale, unpinned requirements.txt is gone
* security: .github/workflows/security.yml scans on every PR, every merge and daily -- pip-audit per lock file, bandit over src/, licenses, the registry, exception expiry -- and keeps the public report, docs/security/report.md, current
* security: releases are published only from .github/workflows/release.yml through PyPI Trusted Publishing, with attestations and a CycloneDX SBOM attached to the GitHub release. `make upload` is gone
* security: pyproject.toml pins the build backend (setuptools>=84.1,<85); what a wheel contains no longer depends on the machine that built it
* security: bandit findings in src/ fixed (sha1 marked usedforsecurity=False, asserts used for argument checks replaced by exceptions, the minidom round-trip in the JUnit writer replaced by ET.indent) or annotated with a reason; the example at the bottom of crypto.py, which carried a ciphertext, was removed

v1.10.0
==============
* add core/report: build report data from a run, then render it -- HTML and JSON come from the same model
* strip secrets when the data is built, not when it is rendered; masking in a template would protect one view and leave the JSON behind it readable
* redaction masks on two independent signals, field name and value shape, so a key pasted into a free-text error message is caught even though it has no field name
* pass rate counts conclusive cases only -- a suite that skips half its cases is not 100% healthy
* report separates cases that told us nothing (skipped/blocked/gap/error) from real results, and shows how many failures actually implicate the system under test
* Status carries BLOCKED and GAP; neither is inferred from a run, since a run cannot report an assertion nobody wrote
* HTML autoescapes: failure messages contain whatever the system under test returned
* add build_report MCP tool
* an unrecognised outcome maps to ERROR rather than PASS

v1.9.0
==============
* add core/triage: rule-based failure classification -- defect, contract, environment, data, harness -- always on, deterministic, free
* separate failures that implicate the system under test from those that implicate our harness, data or network; conflating them is what teaches people to ignore red builds
* classification reuses Check.src recorded at assertion time: all-schema failures mean contract drift, derived means business logic
* run_tests and explain_failure now carry the classification and a next step per failure
* add optional LLM root-cause analysis in ai/triage.py, registered through a hook so core/ never imports ai/; it adds prose only and never decides a category
* add ipandora.ai.enable()/disable(); importing the package is not enough to start spending
* fix Runtime.Ai.max_calls_per_run being impossible to set to 0 -- the falsy check treated a configured 0 as unset and restored the default, disabling the strictest budget setting
* rewrite scripts/check_layering.py to parse imports with AST; the previous text grep flagged documentation that merely mentions ipandora.ai, which core/triage necessarily does
* strip pytest's (file, lineno, ...) wrapper from skip messages, which pointed at the assertion collector rather than anything useful

v1.8.0
==============
* add core/runner: run(selector, env) -> RunResult, callable as a library
* MCP run_tests/explain_failure/list_runs go through it; run_tests no longer shells out to pytest and parses stdout
* output forks by audience: summary() carries failure names and assertion messages for an agent, full tracebacks stay in the run store behind explain_failure(run_id)
* runs are archived by run_id under ~/.ipandora/runs (override with IPANDORA_RUNS_DIR), pruned to the last 50
* use --import-mode=importlib so run() can be called repeatedly in one process; the default mode aborts on a second run over same-named test files
* a selector that matches nothing, or any pytest exit code meaning the run did not happen, now reports ok=False instead of reading as "everything passed"
* run(quiet=True) suppresses pytest output, and the MCP server routes logging to stderr under stdio transport -- both would otherwise corrupt the JSON-RPC stream
* add log_to_stderr() for callers that own stdout

v1.7.0
==============
* Runtime.Case is now thread-local: steps used to live in a process-global dict keyed by a single process-global case name, so under any concurrency every thread's steps landed under whichever name was set last
* reading Runtime.Case.steps no longer pops -- the first reader used to empty it and a second read silently returned []; use drain_steps() to take them, clear() to reset
* MultiHandle buffers per instance under a lock; the buffer used to be a class attribute shared by every instance and mutated unsynchronised from both the producer and the background thread
* MultiHandle's background thread waits on an Event instead of busy-looping on sleep, so stop() takes effect immediately
* deduplicate the atexit flush registration, which two package __init__ modules were each registering
* add Runtime.reset() to drop memoised config, restoring defaults snapshotted at import; config overrides used to leak process-wide with no way to undo them

v1.6.0
==============
* fix response parsing: a single-object payload was torn into its field values, null and scalar data raised TypeError, and a string body was shredded into characters
* replace the namedtuple conversion with JsonObject, a dict that also supports attribute access -- resp.data.origin still works and resp.data['x-req-id'] now does too
* remove the silent degradation where a Python-keyword or dashed JSON key made json_to_obj fall back to a plain dict, so the parsed type depended on the payload's key names and filter() then raised AttributeError
* non-JSON bodies are returned as-is (str for text, bytes for binary) instead of being wrapped in a fake object
* filter() skips fieldless items instead of raising, and accepts dicts, JsonObject and namedtuples alike
* fetch_one() returns None rather than [] when there is no data
* __len__ no longer calls the consuming fetch_all(), so len() has no side effects and does not clear an applied filter
* AttributeError on a missing field now lists the fields that are present

v1.5.0
==============
* add core/protocol/http/transport.py -- TransportPolicy, retry adapter, error translation
* TLS verification now defaults to on; it was forced off for every https request, hiding certificate problems the suite should catch
* every request now carries a timeout (10s connect / 30s read by default); previously ApiOption defaulted to 0 and nothing applied it, so a hung endpoint hung the suite forever
* add retry with exponential backoff, restricted to idempotent methods and transient statuses (429/502/503/504); 500 is not retried
* transport failures raise HttpTimeoutError / HttpConnectionError carrying url, method and elapsed, and also inherit the matching requests exception
* mount an HTTPAdapter with configurable pool sizing on every session
* add http section to config.yaml and Runtime.Http
* fix SessionManager check-then-act happening outside the lock
* fix SessionManager acquire/release without try/finally, which deadlocked the RLock permanently on any exception
* fix deprecated current_thread().getName()

v1.4.0
==============
* add core/assertion -- the framework previously had no assertion layer at all, so a 500 response passed silently
* Check is a value, not an exception: assert_all evaluates every judgement and reports all failures in one run
* require() skips instead of failing, so unmet preconditions stay distinguishable from real defects
* http checks: status_ok, status_is, header_is, json_has, json_equals, json_matches, schema_conforms
* jsonpath accessor reads the raw body, sidestepping ResponseHandler.fetch_all() (returns key names for single-object payloads, raises on null/scalar)
* json_matches reuses the existing Compare operators; all comparators stay deterministic and offline
* declare jsonschema explicitly (was only a transitive dependency of mcp)
* test_http_get now asserts status, which it never did before

v1.3.0
==============
* archive core/engine/generator stack (2737 LOC) -- removes the duplicate case/testcase data models, the unwired v2 robot render path, and the orphan repository modules; preserved at tag v1.1.0-archive
* archive run/ scaffold subcommands (ipandoracommand, commands/project) and the duplicate runner.py script
* remove hardcoded credentials from Runtime.Host; report host now comes from conf/config.yaml
* remove Compare.cmp_semantic -- assertions must stay deterministic and offline
* move ai/ out of core/ and default ai.enabled to false; add max_calls_per_run budget cap
* trim MCP server to get_test_report; run_tests/explain_failure deferred until core/runner and core/triage exist
* add CI (import smoke, test matrix, removable-ai gate, layering and secret checks)
* add pre-commit config with secret scanning
* fix Log.redInfo typo that crashed the CLI when no subcommand was given
* fix UploadApiInfo building its URL at class-definition time
* correct python_requires to >=3.10 (code already used PEP 604 unions)
* normalize runtime.py indentation to 4 spaces

v1.2.0
==============
* add pluggable AI provider abstraction (core.engine.ai): AIProviderABC, AnthropicProvider, MockProvider, AIProviderFactory
* add ai/mcp config sections and Runtime.Ai/Runtime.Mcp
* add semantic assertion (Compare.cmp_semantic) backed by the AI provider
* add MCP server (core.mcp) exposing list_test_cases/create_test_case/run_test_suite/get_test_report, plus `ipandora mcp` command
* add docs/dev/AI_Framework_Design.md roadmap for test intelligence

v1.1.0
==============
* add service for case and robot
* add AttachmentRepository

v1.0.9
==============
* update BaseRepository, generate_insert_query and generate_update_query
* refactory BaseRepository.execute_with_transaction
* add UUIDFactory

v1.0.8
==============
* add architecture for the framework

v1.0.7
==============
* add common libraries -- timeutils, fileutils, stringutils
* add utils -- pathutils, robotlogparser, fileload, filewrite
* add core.engine -- crypto
* add core.engine -- generator, repository and service for testcase

v1.0.6
==============
* rename src/intellipandora to src/ipandora
* add common libraries
* split the core/engine into engine and protocol

v1.0.4
==============
* modify readme

v1.0.3
==============
* modify project and framework name as IntelliPandora
* Fix problem to pypi publish

v1.0.2
==============
* add http capability and plugin
* add unit testing for http provider
* add user guider
* Fixed some module naming

v1.0.1
==============
* update readme
* add docs/Python_Dev_Configuration
* add makefile

v1.0.0
==============
* provider base architecture for the framework
* add basic command model
* add command manager
* add package features