Metadata-Version: 2.4
Name: ovos-routing-judge
Version: 0.0.3
Summary: Who took an utterance, and was it the right skill? One routing judge for OVOS golden-utterance tests.
Author-email: Andreas Lorensen <andlo@outlook.dk>
License: Apache-2.0
Project-URL: Source, https://github.com/andlo/ovos-routing-judge
Project-URL: Bug Tracker, https://github.com/andlo/ovos-routing-judge/issues
Keywords: ovos,openvoiceos,testing,intent,routing,golden utterances
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Dynamic: license-file

# ovos-routing-judge

Who took an utterance, and was it the skill that should have?

One judge for every tool that replays golden utterances
(`test/end2end/golden_utterances_<lang>.jsonl`) against
[OpenVoiceOS](https://openvoiceos.org) and asks whether each sentence reached
the right skill. It is pure Python over the bus messages of **one session**:
no messagebus connection, no UI, no store knowledge, no dependencies. So the
same row gets the same verdict wherever it runs:

- [ovos-klondike-mercantile](https://github.com/andlo/ovos-klondike-mercantile)'s
  CI, on an in-process MiniCroft;
- [ovos-tui-client](https://github.com/andlo/ovos-tui-client)'s test runs
  (`ovos-tui --run`), on a live device over the messagebus.

A failure in CI can then be reproduced on a device and judged the same way
(andlo/ovos-klondike-mercantile#48).

## Use

```python
from ovos_routing_judge import Claim, in_session, judge

claim = Claim.from_messages(recorded, known_ids=loaded_skill_ids,   # Message objects or dicts
                            session_id=row_session_id)
v = judge(claim, own_ids={"ovos-skill-date-time.openvoiceos"},
          expected="ovos-skill-date-time.openvoiceos:what_time_is_it",
          hung=timed_out, known_ids=loaded_skill_ids)
v.kind     # hit | wrong_intent | captured | other | unhandled | hang
v.taker    # who took it
v.via      # its best signal: intent | provider | ocp | skill | speak
v.detail   # one line on what happened
```

`Claim.observe(message, known_ids)` folds messages in one at a time, for a
live run that wants to know as soon as something matched.

`known_ids` is a filter by default: only loaded skills can take a row, so a
pipeline plugin's own intent with nothing behind it is `unhandled`. A caller
whose skill list may be incomplete (a live device with skills in other
containers) passes `strict_known=False`: known skills still win, anyone else
is the fallback.

## Verdicts

| kind | meaning |
|---|---|
| `hit` | the row's skill took it (with the expected intent, when the row names one) |
| `wrong_intent` | the row's skill took it, and another of its intents is what fired (no visible intent at all is a `hit`) |
| `captured` | the row's skill was waiting for an answer (get_response / converse) and took the sentence as that answer |
| `other` | another skill took it (`taker`); a store can split this further |
| `unhandled` | nobody took it, or only a pipeline stage with no skill behind it |
| `hang` | nobody took it and the caller timed out |

## What counts as taking a sentence

Signals come in tiers; the taker is the first known skill in the best tier:

1. **intent**: `<skill_id>:<intent>` dispatched, `ovos.intent.matched`, or a
   pending `<skill_id>.converse.*` capturing it
2. **provider**: a pipeline plugin asked it for content
   (`ovos.common_reading.fetch_content.<skill_id>`)
3. **ocp**: OCP picked its result (`ovos.common_play.play` media, or the
   first track of `play` / `search.populate` without a player)
4. **skill**: handler start, a fallback that answered, common query's
   `question:action`
5. **speak**: it spoke in this session (`speak` or core 3's `ovos.utterance.speak`)

`ovos.intent.unmatched` means nobody, whatever else was seen.

With `session_id`, other sessions' messages are ignored, and a message
without a session can show an intent, a provider or an OCP pick but never a
tier 4 or 5 claim: unrelated skills' background activity (a scheduled
event's handler, a fallback probe's reply) arrives without a session and is
not theft (learned in ovos-test-harness's fleet suite).

Intent names are compared in one spelling: `Name.intent` (ovos-workshop 1.x)
vs `Name`, CamelCase in older releases vs snake_case, padatious
`what.time.is.it` vs padacioso `what_time_is_it`. A row with
`"intent_type": "ocp"` must go through OCP.

## The contract

`tests/fixtures/cases.json` holds recorded message sequences and the verdict
each must get. When a core changes its signals, a case goes in there and every
tool using the package gets the fix.

## License

Apache-2.0
