Metadata-Version: 2.5
Name: okareo-vision
Version: 0.1.0
Summary: Simulate a customer against an agent that lives behind a web UI, with Okareo
Project-URL: Homepage, https://okareo.com
Project-URL: Documentation, https://docs.okareo.com/simulation/web-ui
Project-URL: Repository, https://github.com/okareo-ai/okareo-vision
Author-email: Okareo <info@okareo.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agents,chatbot,evaluation,llm,okareo,playwright,simulation,testing,web ui
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Software Development :: Testing
Requires-Python: <3.13,>=3.10
Requires-Dist: okareo>=0.0.151
Requires-Dist: playwright>=1.49
Description-Content-Type: text/markdown

# okareo-vision

okareo-vision is the browser side of an [Okareo](https://okareo.com) simulation
of an agent that lives behind a web page: you describe the page in a small
Python class, and it opens the page in Chromium, runs the moves Okareo sends
each turn, and answers with what the screen shows. Okareo plays the customer
with a Driver you write, reads every screen, and scores the conversation;
nothing in this package reads the screen or calls a model.

## Install

```bash
pip install okareo-vision
playwright install chromium
```

Python 3.10 to 3.12, in a virtualenv of your own. It installs the Okareo SDK (`okareo`) and Playwright;
`playwright install chromium` downloads the browser once (on Linux,
`playwright install --with-deps chromium` also installs the libraries it
needs). A Run needs an Okareo account and its API key.

## Run the example

The example is in the repository, beside a local copy of a made-up store's
page (Shopwise, with a shopping helper called Sprig), so it needs no site of
your own.

```bash
git clone https://github.com/okareo-ai/okareo-vision.git
cd okareo-vision/examples/retail_chat
export OKAREO_API_KEY=<your Okareo API key>
export OKAREO_BASE_PATH=https://api.okareo.com   # leave it unset for a local Okareo server at http://localhost:8000
python run_retail_chat.py --debug
```

It creates a Scenario with two rows, a Driver and a Target, runs both
conversations in a headless browser, and prints the link to the finished Run in
the Okareo app (on a local server, the app at `http://localhost:3000`). `--debug`
keeps a record of the run under `okareo_runs/` (see
[The run record](https://docs.okareo.com/simulation/web-ui#the-run-record)). `--headed` shows the browser,
`--max-turns N` caps each conversation (default 20), and `--name` names the
Run, its Scenario and the Target (the Driver is always "Shopwise shopper");
rerunning under the same name reuses the Scenario, so change it after editing
`scenarios.jsonl`. `OKAREO_PROJECT` picks an Okareo project by name or id;
unset, the default project.

Its three files are everything you write for a site of your own:

| File | What it is |
|---|---|
| `run_retail_chat.py` | `ShopwiseHelper`, a `VisionTarget` subclass (the page, a note about it, a `prepare` step), and the SDK calls that start the Run |
| `driver.md` | the Driver: who the customer is and how they behave |
| `scenarios.jsonl` | the Scenario rows: per line, an `input` with the facts the Driver uses and a `result` saying what a finished conversation looks like |

The example's page is a made-up store's page served from a file, so it needs
no site and no sign-in.

## How a turn works

Okareo does the looking. Each turn it sends a batch of commands; okareo-vision
runs them in order and answers with a frame: a JPEG of the screen, its hash,
the page's visible text, the controls on screen (role, name, state and box,
from the browser's accessibility tree), whether the screen changed, and the
outcome of each command. A command that fails says why in its result; the turn
goes on, and so does the Run. There are six commands:

| Command | What it does |
|---|---|
| `click` | Presses a control by its place on the screen, by its name, or as an entry from the control list. With a name and a place, what is under the place must carry that name; otherwise nothing is pressed and the result says what is there. |
| `type` | Types into the focused field, replacing what it held, and reads the value back. Nothing focused, or a value the page rewrote, is a failed command. |
| `press` | Presses one key: `Enter`, `Tab`, `Escape`, `ArrowDown`. |
| `scroll` | Turns the mouse wheel over a place on the screen (an inner list moves, not the page), or over the middle of the screen. |
| `compose` | Puts a message in the chat's message box and sends it: Enter, then the send button beside the box if Enter only added a new line. |
| `wait` | Waits for the screen to settle. |

The screen has settled when captures taken a moment apart stay identical for a
quiet window, 1.2 seconds, for at most `settle_timeout_s` (Okareo can ask for
other values in a `wait`). The first frame of every conversation waits the
same way. A spinner that never stops is reported as not settled; what to do
about it is the Driver's call.

## VisionTarget

Configure it by subclassing, with class attributes, or with constructor
arguments; an argument wins over the attribute.

```python
from okareo_vision import VisionTarget, click_selector

class SupportChat(VisionTarget):
    url = "https://support.example.com/"
    app_notes = "The chat opens from the 'Help' button at the bottom right."

target = SupportChat(headless=False, max_parallel=3)

# the same page, no subclass
target = VisionTarget(url="https://support.example.com/",
                      app_notes="The chat opens from the 'Help' button at the bottom right.",
                      prepare=click_selector("#help-button"))
```

| Setting | Default | What it does |
|---|---|---|
| `url` | required | The page every conversation opens. `{session_index}` (1, 2, … in start order) and `{scenario_input.KEY}` (a field of the row's input) are filled per conversation. Without a url the constructor raises `TypeError`. |
| `app_notes` | `""` | What Okareo is told about the page: where the chat is, what the side panels are, what has to happen before a request can start. |
| `prepare(page)` | does nothing | Gets the page ready; see [prepare](https://docs.okareo.com/simulation/web-ui#prepare). A method on the subclass, or `prepare=` on the constructor. |
| `name` | the class name | The Target's name in Okareo. Constructor only. |
| `viewport` | `(1280, 900)` | The browser window's size; frames come in it. |
| `max_parallel` | `10` | How many conversations Okareo may hold open at once. They share one browser, each in its own context (its own cookies and storage). |
| `settle_timeout_s` | `30.0` | The longest a wait for the screen to settle lasts. |
| `headless` | `True` | `False` shows the browser. |
| `channel` | `None` | Which browser to launch. `None` is Playwright's Chromium; `"chrome"`, `"msedge"`, `"chrome-beta"`, `"msedge-beta"` are browsers installed on the machine (or fetched once with `playwright install chrome`). |
| `user_agent` | `None` | `None` presents a normal Chrome of the running version; `""` keeps Playwright's own; any other string is used as is. |
| `browser_args` | `()` | Extra Chromium command-line flags. |
| `storage_state` | `None` | A Playwright storage-state file (a saved sign-in); every conversation starts from it. |
| `screen_selector` | `None` | A CSS selector for the one element Okareo is shown as the screen, for a chat panel docked over a busy page. Frames, visible text, the control list and Okareo's points are that element's. It must be on the page whenever a frame is taken; when it is not, the turn fails. `None` is the whole window. |
| `composer_selector`, `send_selector` | `""` | The chat's message box and send button. When set, `compose` fills that box and clicks that button instead of using Okareo's point and the Enter key. |
| `debug` | off | Records a video, a HAR and a screenshot per turn; see [The run record](https://docs.okareo.com/simulation/web-ui#the-run-record). |
| `run_dir` | `None` | A `RunDir` (from `okareo_vision.diagnostics`) to write the run record to. Debug mode makes one when none is given. |

`VisionTarget.preflight()` launches and closes the browser once, so a missing
browser fails in your terminal instead of once per conversation inside a Run.

Environment variables, for settings without code:

| Variable | What it does |
|---|---|
| `OKAREO_WEB_TARGET_DEBUG=1` | Debug mode on. |
| `OKAREO_WEB_TARGET_HAR=full` | The HAR keeps cookies, headers and security details (otherwise Playwright's minimal HAR). |
| `OKAREO_RUN_DIR` | Where run directories go instead of `okareo_runs/`. |
| `OKAREO_WEB_BROWSER` | The `channel`, e.g. `chrome`. |
| `OKAREO_WEB_PERCEPTION` | The perception mode Okareo runs this Target with, `vision` or `ax`, sent when the Target registers; a Run's own simulation settings outrank it. |

The package also exports `Perception`, `Resolver`, `WizardExecutor` and
`parse_driver_values`, a separate rule-based engine that `VisionTarget` does
not use and this page does not cover.

## prepare

`prepare(page)` runs once per conversation, after the page loads and before
Okareo's first look. The [Playwright `Page`](https://playwright.dev/python/docs/api/class-page)
is yours: close a banner, sign in, open the chat, and wait for the message box
so the first frame has the chat in it. Raising ends that conversation with the
error on its row; the Run goes on. The example's, from `run_retail_chat.py`:

```python
class ShopwiseHelper(VisionTarget):
    url = REPLICA.as_uri() + "?plain=1&notice=1&greet=1"   # the store's page, opened from a file
    app_notes = (
        "This is Shopwise, an online store for the backyard. Its shopping helper, Sprig, opens as a "
        "drawer on the right side of the page. The message box at the bottom of the drawer reads "
        "'Ask anything'; a message is sent with the arrow button beside it (Enter only adds a new "
        "line). ..."
    )

    async def prepare(self, page: Page) -> None:
        # A cookie notice can cover the page on first load; its X closes it.
        notice = page.get_by_role("dialog", name="Cookies on Shopwise")
        if await notice.count() and await notice.is_visible():
            await notice.get_by_role("button", name="Close", exact=True).click()
            await notice.wait_for(state="hidden", timeout=5000)
        # The helper opens from "Ask Shopwise", or from the round button in the corner.
        pill = page.get_by_role("button", name="Ask Shopwise")
        if await pill.count() and await pill.is_visible():
            await pill.click()
        else:
            await page.locator('[data-testid="chat-bubble"]').click(force=True)
        await page.get_by_role("textbox", name="Ask anything").wait_for(state="visible", timeout=10000)
        # The helper can remember an earlier visit; Reset chat starts clean.
        reset = page.get_by_role("button", name="Reset chat")
        if await reset.count() and await reset.is_visible():
            await reset.click()
            await page.get_by_text("I'm Sprig").first.wait_for(state="visible", timeout=10000)
```

When one click is all it takes, `click_selector("#help-button")` is a ready
`prepare`: it clicks the control, and carries on with a warning when the
control is not there.

## The Driver is yours

The package ships no Driver. A Driver is a prompt you write and register with
the Okareo SDK: who the customer is, what they want, how they behave, and when
they are done. Okareo adds what every Driver in a web Run needs to read a
screen and make moves, so your text is about the customer only.
`{scenario_input.KEY}` is filled from the row's input. The start of the
example's `driver.md`:

```markdown
## Persona

-   **Identity:** You are role-playing a shopper on a store's website, chatting with the store's shopping helper to get one thing done.
-   **Mindset:** Friendly and to the point. You know what you want and how much you will spend; you answer the helper's questions from the Scenario Details and stop as soon as your request is done.

## Scenario Details

What you want: {scenario_input.task}
Budget: {scenario_input.budget}
Notes: {scenario_input.notes}

## Objectives

1. Open by telling the helper what you are shopping for, in one short sentence in your own words.
2. Answer the helper's questions from the facts above, and pick from what it shows you.
3. Stop once the helper has answered everything your request asks for. A question from the helper is something to answer, not the end.
```

It goes on with tactics, hard rules and when to end. A row that fills it:

```json
{"input": {"test_case": "SHOP-02", "task": "Find out which bag sizes the lump charcoal comes in.", "budget": "under $40", "notes": "Ask for lump charcoal first, then press the 'Show bag sizes' suggestion under Sprig's reply."}, "result": "Sprig suggests lump charcoal under $40 and says its bags come in 15 lb and 20 lb."}
```

And the SDK calls that run it, shortened from the example (the example reads
the server, the project and the names from its flags and environment):

```python
import json, os
from okareo import Okareo
from okareo.model_under_test import Driver, Target
from okareo_api_client.models import ScenarioSetCreate

okareo = Okareo(api_key=os.environ["OKAREO_API_KEY"], base_path="https://api.okareo.com")
rows = [json.loads(line) for line in open("scenarios.jsonl") if line.strip()]
scenario = okareo.create_scenario_set(ScenarioSetCreate(
    name="Shopwise shopper", seed_data=Okareo.seed_data_from_list(rows)))
driver = Driver(name="Shopwise shopper", prompt_template=open("driver.md").read(), temperature=0.0)

ShopwiseHelper.preflight()
run = okareo.run_simulation(
    name="Shopwise shopper",
    scenario=scenario,
    target=Target(name="Shopwise shopper", target=ShopwiseHelper()),
    driver=driver,
    checks=["result_completed"],
    max_turns=20,
    first_turn="driver",
)
print(run.app_link)
```

`run_simulation` serves the browser side from this process until the Run
finishes. The browser runs where this process runs, so the page can be on your
own network.

## The run record

With `--debug` in the example, `debug=True` on the Target, or
`OKAREO_WEB_TARGET_DEBUG=1`, the Target writes a record of its conversations to
a directory under `okareo_runs/` in the working directory (or under
`$OKAREO_RUN_DIR`), named for the time the first one started, in UTC. A
`RunDir` passed as `run_dir=` writes the same record without the video and the
HAR.

```
okareo_runs/20261006-063959/
  turns.jsonl            one line per turn, and one per conversation's video
  shots/                 turn00-9d0efd23.jpg: each frame Okareo saw
  ax/                    turn00-9d0efd23.json: the control list read from the page
  dom/                   turn00-9d0efd23.html.gz: the page's HTML at that frame
  video/                 9d0efd23.webm per conversation (and .mp4 with ffmpeg); debug mode
  9d0efd23.har           the page's network traffic per conversation; debug mode
```

`9d0efd23` is a conversation, `turn00` its first frame, taken before any turn
and so with no line of its own; turn `N`'s files are `turnNN-9d0efd23.*`. The
example's first turn in that conversation, one line of `turns.jsonl` shown
across several:

```json
{
  "turn": 1,
  "session": "9d0efd23",
  "commands": [
    {"op": "compose", "text": "I’m looking to buy a bag of lump charcoal for grilling, under $40.", "x": 1073, "y": 867},
    {"op": "wait", "quiet_s": 1.2, "timeout_s": 30.0}
  ],
  "results": [
    {"op": "compose", "ok": true},
    {"op": "wait", "ok": true, "settled": true}
  ],
  "frame_sha1": "7fb9cafdec473e0df1dfc424887afe59c2349b44",
  "video_offset_s": 10.29,
  "browser_ms": 1354,
  "at": "2026-10-06T06:40:09+00:00"
}
```

| Field | What it holds |
|---|---|
| `turn`, `session` | the turn's number in its conversation, and the conversation |
| `commands` | the commands as Okareo sent them |
| `results` | one per command: `op`, `ok`, `error` when it failed, and what the command reports (for `click`: `how`, which is `name`, `point` or `typed`, with `name`, `at` and `scrolled`; `landed` and `value` for `type`; `settled` for `wait`) |
| `frame_sha1` | the SHA-1 of the capture the settle wait compared, the screenshot before it is encoded as the JPEG in `shots/`; it is not the saved file's hash |
| `video_offset_s` | where this turn is in the conversation's video, in seconds |
| `browser_ms` | the time this turn spent in the browser |
| `at` | when the line was written, UTC |

When a conversation ends in debug mode, one more line names its video:
`{"session": "9d0efd23", "video": ".../video/9d0efd23.webm", "at": "..."}`.

Three more files are written when your runner asks for them, with the helpers
in `okareo_vision.diagnostics` and a `RunDir` passed to the Target:
`setup(run_dir)` logs to `run.log`; `preflight(run_dir, okareo_base=...,
app_url=...)` writes `preflight.json` (Python, Playwright, proxy settings, and
whether Okareo and the page can be reached); `report(run_dir, run_link=...)`
writes `report.txt`, a per-turn summary with no credentials in it.

The record holds whatever the page showed and sent: the screenshots, the HTML
and the video show all of it. The HAR is recorded in Playwright's minimal mode
(no cookies, headers or security details) unless `OKAREO_WEB_TARGET_HAR=full`.
The video is written when its conversation ends.

## The server it needs

okareo-vision speaks version 1 of Okareo's web simulation protocol, and says so
when the Target registers. Okareo's cloud (`https://api.okareo.com`) speaks it,
and a local Okareo server works the same; the example talks to
`http://localhost:8000` unless `OKAREO_BASE_PATH` says otherwise. The oldest
Okareo SDK it works with is 0.0.151, the floor of the `okareo` it installs.

The traffic goes two ways. To Okareo: HTTPS to the API, and NATS over a
WebSocket to `wss://connect.ngs.global:443`, which every Run needs, a local
server included, so the machine needs that outbound connection. To the page:
the browser, directly. Chromium does
not read `HTTPS_PROXY`, so a page that must be reached through a proxy needs
it as a browser flag, `browser_args=["--proxy-server=http://proxy.example.com:8080"]`.

## When it goes wrong

### The page never shows a chat

- Watch it: `headless=False` (the example's `--headed`), or `--debug` and open
  `shots/turn00-*.jpg`, the first frame Okareo saw.
- `prepare` is where the chat gets opened. End it by waiting for the message
  box, as the example does, so the first frame has the chat in it; when
  `prepare` raises, the error is on the row.
- Say in `app_notes` where the chat is and what opens it.
- A chat in a panel over a busy page: `screen_selector` shows Okareo the panel
  alone.
- When Okareo cannot find the message box: `composer_selector` and
  `send_selector` name it.

### The site refuses headless Chrome

The browser already presents itself as a normal Chrome: the automation marker
(`navigator.webdriver`) is off and the user agent has no "HeadlessChrome" in
it. When the site still refuses, show the browser and use the real Chrome:
`headless=False` and `channel="chrome"` (or `OKAREO_WEB_BROWSER=chrome`; run
`playwright install chrome` if it is not on the machine). Sites with a bot gate
commonly admit only the real Chrome, headed. A headed browser needs a display;
on a Linux server, run under `xvfb-run`.

### Sign-in needs MFA

A scripted sign-in cannot answer an MFA prompt. Sign in by hand once, save the
session, and start every conversation from it:

```python
from okareo_vision import VisionTarget
from okareo_vision.auth import ensure_session

with ensure_session("https://portal.example.com/", storage_state="portal_session.json",
                    login_timeout=300) as (session, page):
    print("signed in at", session["landed_url"])

class Portal(VisionTarget):
    url = "https://portal.example.com/"
    storage_state = "portal_session.json"
```

With no `recipe`, `ensure_session` opens a visible browser at the url and waits
up to `login_timeout` seconds (600 unless you say) for you to sign in, MFA
included, then saves the session to `storage_state` with file mode 0600. Run
it before each Run: it reuses the saved session and asks you to sign in again
only when it no longer works (`verify=`, a function of `(page, api_base)`
returning a bool, says what "works" means for your site). The file is a
signed-in session; keep it out of version control. `form_login(...)` and
`entra_login(...)` in the same module are scripted sign-ins for sites without
MFA.

## What is not in the package

- **A Driver.** Yours, registered with the Okareo SDK.
- **Model calls.** This process holds no model key; Okareo reads the screens
  and picks the moves.
- **Scenarios and Checks.** Made with the Okareo SDK, as in the example.
- **The example and test pages.** They are in the
  [repository](https://github.com/okareo-ai/okareo-vision), not in the wheel.
- **Anything about a particular site.** Its url, sign-in and selectors belong
  in your subclass.

Documentation: <https://docs.okareo.com/simulation/web-ui>. Changes:
[CHANGELOG.md](https://github.com/okareo-ai/okareo-vision/blob/main/CHANGELOG.md).
License: Apache-2.0.
