Metadata-Version: 2.5
Name: okareo-vision
Version: 0.3.0
Summary: Simulate a customer against an agent that lives behind a web UI, with Okareo
Project-URL: Homepage, https://okareo.com
Project-URL: Documentation, https://docs.okareo.com/simulation/web-ui
Project-URL: Repository, https://github.com/okareo-ai/okareo-vision
Author-email: Okareo <info@okareo.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agents,chatbot,evaluation,llm,okareo,playwright,simulation,testing,web ui
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Software Development :: Testing
Requires-Python: <3.13,>=3.10
Requires-Dist: okareo>=0.0.151
Requires-Dist: playwright>=1.49
Provides-Extra: dev
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-asyncio; extra == 'dev'
Description-Content-Type: text/markdown

# okareo-vision

okareo-vision is the browser side of an [Okareo](https://okareo.com) simulation
of an agent that lives behind a web page: you describe the page in a small
Python class, and it opens the page in Chromium, runs the moves Okareo sends
each turn, and answers with what the screen shows. Okareo plays the customer
with a Driver you write, reads every screen, and scores the conversation;
nothing in this package reads the screen or calls a model.

## Install

```bash
pip install okareo-vision
playwright install chromium
```

Python 3.10 to 3.12, in a virtualenv of your own. It installs the Okareo SDK (`okareo`) and Playwright;
`playwright install chromium` downloads the browser once (on Linux,
`playwright install --with-deps chromium` also installs the libraries it
needs). A Run needs an Okareo account and its API key.

## Run the example

The example is in the repository, beside a local copy of a made-up store's
page (Shopwise, with a shopping helper called Sprig), so it needs no site of
your own.

```bash
git clone https://github.com/okareo-ai/okareo-vision.git
cd okareo-vision/examples/retail_chat
export OKAREO_API_KEY=<your Okareo API key>
export OKAREO_BASE_PATH=https://api.okareo.com   # leave it unset for a local Okareo server at http://localhost:8000
python run_retail_chat.py --debug
```

It creates a Scenario with two rows, a Driver and a Target, runs both
conversations in a headless browser, and prints the link to the finished Run in
the Okareo app (on a local server, the app at `http://localhost:3000`). `--debug`
keeps a record of the run under `okareo_runs/` (see
[The run record](https://docs.okareo.com/simulation/web-ui#the-run-record)). `--headed` shows the browser,
`--max-turns N` caps each conversation (default 20), and `--name` names the
Run, its Scenario and the Target (the Driver is always "Shopwise shopper");
rerunning under the same name reuses the Scenario, so change it after editing
`scenarios.jsonl`. `OKAREO_PROJECT` picks an Okareo project by name or id;
unset, the default project.

Its three files are everything you write for a site of your own:

| File | What it is |
|---|---|
| `run_retail_chat.py` | `ShopwiseHelper`, a `VisionTarget` subclass (the page, a note about it, a `prepare` step), and the SDK calls that start the Run |
| `driver.md` | the Driver: who the customer is and how they behave |
| `scenarios.jsonl` | the Scenario rows: per line, an `input` with the facts the Driver uses and a `result` saying what a finished conversation looks like |

The example's page is a made-up store's page served from a file, so it needs
no site and no sign-in.

## How a turn works

Okareo does the looking. Each turn it sends its moves; okareo-vision
runs them in order and answers with a frame: a JPEG of the screen, its hash,
the page's visible text, the controls on screen (role, name, state and box,
from the browser's accessibility tree), whether the screen changed, and the
outcome of each command. A command that fails says why in its result; the turn
goes on, and so does the Run. A failed `type` stops the rest of its turn: the
moves after it are not run (a `wait` still is), so nothing is submitted on a
wrong value. There are six moves:

| Command | What it does |
|---|---|
| `click` | Presses a control by its place on the screen, by its name, or as an entry from the control list. With a name and a place, what is under the place must carry that name; otherwise nothing is pressed and the result says what is there. |
| `type` | Types into the focused field, replacing what it held, and reads the value back. Nothing focused after 300 ms (a box can take focus a beat after the press), or a value the field does not hold, is a failed command. A type that waited for focus says how long (`focus_wait_ms`). |
| `press` | Presses one key: `Enter`, `Tab`, `Escape`, `ArrowDown`. |
| `scroll` | Turns the mouse wheel over a place on the screen (an inner list moves, not the page), or over the middle of the screen. |
| `compose` | Puts a message in the chat's message box and sends it: Enter, then the send button beside the box if Enter only added a new line. |
| `wait` | Waits until the turn is done. |

A turn is done the way browser automation usually decides it, with the picture
keeping the last word. The page and its frames must first go half a second with
no request of the page's own site in flight, no change in the DOM (open shadow
roots included), and nothing drawn in the accessibility tree saying it is busy
(the busy state, or a progress bar with no value; a bar at opacity 0 is not
drawn). A request is over when the browser says so, when its response came with
a known length, or when the page it belonged to went to another page (the
browser's error page too) or its frame was taken off the page; never because it
is old. Then two captures 400 ms apart must match, with the box
being typed in left out of both (a rich-text editor that is a whole document
is left out as a whole): a blinking cursor is not a change, and a canvas app's
paint still holds the wait. A signal that keeps changing with no half-second
pause through the first 2 seconds (analytics beacons, an animation in the DOM)
is left out of that wait; one request held open, or one busy node, holds it up
to the limit, by design, and the record names it. Nothing is configured per
site. A wait lasts at most
`settle_timeout_s`, or the limit Okareo sends with it, and the first frame of
every conversation waits the same way. A spinner that never stops is reported
as not settled; what to do about it is the Driver's call.

Each wait's result says how it ended, beside `settled`: `ended_by` (`picture`,
or `ceiling`), `held_by` (the signals that held it), `held` (what held it: the
request open longest, by its host and path, and the busy node, by its role and
name), `dropped` (the signals left out for churning), `waited_ms`, and what the
signals saw (`requests`, `mutations`, `busy_ms`). The log has a line for every
wait, naming what held it.

A `type` works on a box that formats itself as it is typed into (a date box
that puts in its own slashes, a phone or card mask, a currency box) the way a
person does: it types the letters and digits in runs, and before each
separator it looks at the box, skipping its own where the box already put one.
Once a look shows the box holding exactly what was typed, it is not
formatting, and the rest of the value is typed in one burst: a long answer is
as quick as a short one. The read-back accepts the box's formatting:
`555-123-4567` is held by `(555) 123-4567` and `1234.50` by `1,234.50`, while a
box that dropped or moved one of the value's separators, or holds a different
date, has not taken it. A suggestion the box selected after the typed text is
deleted. A miss is cleared and typed once more, a key every 40 ms, or in one
burst for a value over 25 characters; a box whose `maxlength` cannot hold the
value is not typed again. A date is written in the shape the box names in its
placeholder or its name (`MM/DD/YY`, `MM-DD-YYYY`), month first only, and a
two-digit year from the Driver is written as four, which the result says
(`typed 1992 for 92`); a box that names nothing and takes two digits of the
year gets two. The box read back is the one typing began in, even after the
page moves focus on. A row of one-character boxes (a code's digit boxes, each
moving focus to the next) is read as one value from its first box: the boxes
from the one typing began in, forward in document order, while each is of the
first one's kind (each declaring the first one's `maxlength`, or none when it
declares none, and each declaring `maxlength` 1 or holding one character),
wherever focus went after the last digit (the next box, a Submit button). Any
of them holding something is cleared before the code is typed. A value of one
letter or digit never reads or clears a row. A middle-initial box is out of a
first-name box's row when it declares `maxlength` 1 and the first-name box does
not: two adjacent boxes with no `maxlength` (or the same one, over 1), each
holding one character, look like a code row and are read as one. The result
has `landed` and `value`, and when they apply `typed_as`, `retried` and `note`.
The browser's own date input is set through its value setter instead.

A `click` by name alone presses the one control in the control list that answers
to the name, scrolled into view when it is off screen or covered. A row of an open list
that the page has not built yet (a list that builds only the rows near what it
shows, as Flutter's and react-window's do) is reached by paging the list: the
wheel turns over the list a page at a time, down then up, at most 12 pages each
way, and the row is pressed once it appears. A list is open when a row of it is
on top, or when a control that says it is expanded names it (`aria-controls`);
a list that only stands on the page is never paged or scrolled to. A list an
expanded control names that opened past the edge of the scroller it sits in is
scrolled into view first. The result then has `scrolled` and
a note saying how many pages it took. Nothing else is pressed, and a name that
never appears fails the command: `no control named ...`.

## VisionTarget

Configure it by subclassing, with class attributes, or with constructor
arguments; an argument wins over the attribute.

```python
from okareo_vision import VisionTarget, click_selector

class SupportChat(VisionTarget):
    url = "https://support.example.com/"

target = SupportChat(headless=False, max_parallel=3)

# the same page, no subclass
target = VisionTarget(url="https://support.example.com/", prepare=click_selector("#help-button"))
```

| Setting | Default | What it does |
|---|---|---|
| `url` | required | The page every conversation opens. `{session_index}` (1, 2, … in start order) and `{scenario_input.KEY}` (a field of the row's input) are filled per conversation. Without a url the constructor raises `TypeError`. |
| `prepare(page)` | does nothing | Gets the page ready; see [prepare](https://docs.okareo.com/simulation/web-ui#prepare). A method on the subclass, or `prepare=` on the constructor. |
| `cleanup(page)` | does nothing | Runs at the end of each conversation, before its page closes. A method on the subclass. |
| `sign_in(page)`, `sign_out(page)` | do nothing | Once per Run, for a site behind a sign-in; see [Signing in once per Run](#signing-in-once-per-run). |
| `name` | the class name | The Target's name in Okareo. Constructor only. |
| `viewport` | `(1280, 900)` | The browser window's size; frames come in it. |
| `max_parallel` | `10` | How many conversations Okareo may hold open at once. They share one browser, each in its own context (its own cookies and storage). |
| `open_spacing_s` | `0` | For a site that refuses many pages opened at once: each conversation's first page (its `url`) is requested this many seconds after the previous one's response arrived, one request at a time, however slowly it came. The page's own loading and `prepare` are not held. `0` opens them as fast as Okareo starts them. |
| `page_open_timeout_s` | `30.0` | Each page's timeout for loads and actions, in seconds. |
| `settle_timeout_s` | `30.0` | The longest a wait for the screen to settle lasts. |
| `headless` | `True` | `False` shows the browser. |
| `channel` | `None` | Which browser to launch. `None` is Playwright's Chromium; `"chrome"`, `"msedge"`, `"chrome-beta"`, `"msedge-beta"` are browsers installed on the machine (or fetched once with `playwright install chrome`). |
| `user_agent` | `None` | `None` presents a normal Chrome of the running version; `""` keeps Playwright's own; any other string is used as is. |
| `browser_args` | `()` | Extra Chromium command-line flags. |
| `storage_state` | `None` | A Playwright storage-state file (a saved sign-in); every conversation starts from it. A session from `sign_in` is used instead when there is one. |
| `screen_selector` | `None` | A CSS selector for the one element Okareo is shown as the screen, for a chat panel docked over a busy page. Frames, visible text, the control list and Okareo's points are that element's. It must be on the page whenever a frame is taken; when it is not, the turn fails. `None` is the whole window. The crop is optional: a chat in an iframe needs no setting. |
| `composer_selector`, `send_selector` | `""` | The chat's message box and send button. When set, `compose` fills that box and clicks that button instead of using Okareo's point and the Enter key. For a box Okareo cannot find, never for a frame: a box inside an iframe is found without it. |
| `debug` | off | Records a video, a HAR and a screenshot per turn; see [The run record](https://docs.okareo.com/simulation/web-ui#the-run-record). |
| `run_dir` | `None` | A `RunDir` (from `okareo_vision.diagnostics`) to write the run record to. Debug mode makes one when none is given. |

`VisionTarget.preflight()` launches and closes the browser once, so a missing
browser fails in your terminal instead of once per conversation inside a Run.

Environment variables, for settings without code:

| Variable | What it does |
|---|---|
| `OKAREO_WEB_TARGET_DEBUG=1` | Debug mode on. |
| `OKAREO_WEB_TARGET_HAR=full` | The HAR keeps cookies, headers and security details (otherwise Playwright's minimal HAR). |
| `OKAREO_RUN_DIR` | Where run directories go instead of `okareo_runs/`. |
| `OKAREO_WEB_BROWSER` | The `channel`, e.g. `chrome`. |
| `OKAREO_WEB_PERCEPTION` | The perception mode Okareo runs this Target with, `vision` or `ax`, sent when the Target registers; a Run's own simulation settings outrank it. |

## prepare

`prepare(page)` runs once per conversation, after the page loads and before
Okareo's first look. The [Playwright `Page`](https://playwright.dev/python/docs/api/class-page)
is yours: close a banner, sign in, open the chat, and wait for the message box
so the first frame has the chat in it. Raising ends that conversation with the
error on its row; the Run goes on. The example's, from `run_retail_chat.py`:

```python
class ShopwiseHelper(VisionTarget):
    url = REPLICA.as_uri() + "?plain=1&notice=1&greet=1"   # the store's page, opened from a file

    async def prepare(self, page: Page) -> None:
        # A cookie notice can cover the page on first load; its X closes it.
        notice = page.get_by_role("dialog", name="Cookies on Shopwise")
        if await notice.count() and await notice.is_visible():
            await notice.get_by_role("button", name="Close", exact=True).click()
            await notice.wait_for(state="hidden", timeout=5000)
        # The helper opens from "Ask Shopwise", or from the round button in the corner.
        pill = page.get_by_role("button", name="Ask Shopwise")
        if await pill.count() and await pill.is_visible():
            await pill.click()
        else:
            await page.locator('[data-testid="chat-bubble"]').click(force=True)
        await page.get_by_role("textbox", name="Ask anything").wait_for(state="visible", timeout=10000)
        # The helper can remember an earlier visit; Reset chat starts clean.
        reset = page.get_by_role("button", name="Reset chat")
        if await reset.count() and await reset.is_visible():
            await reset.click()
            await page.get_by_text("I'm Sprig").first.wait_for(state="visible", timeout=10000)
```

When one click is all it takes, `click_selector("#help-button")` is a ready
`prepare`: it clicks the control, and carries on with a warning when the
control is not there.

`cleanup(page)` is its counterpart: it runs at the end of each conversation,
before the page closes. A failure in it is logged; the page closes anyway.

## Signing in once per Run

A site behind a sign-in signs in once per Run, not once per conversation:
override `sign_in(page)` and `sign_out(page)`, and run the simulation inside a
`with` block.

```python
class Portal(VisionTarget):
    url = "https://portal.example.com/support"

    async def sign_in(self, page: Page) -> None:
        await page.goto("https://portal.example.com/login")
        await page.fill("#username", os.environ["PORTAL_USER"])
        await page.fill("#password", os.environ["PORTAL_PASS"])
        await page.click("#sign-in")
        await page.get_by_text("Your account").wait_for()    # raises if it did not land

    async def sign_out(self, page: Page) -> None:
        await page.goto("https://portal.example.com/logout")
        await page.get_by_text("You are signed out").wait_for()

with Portal() as target:     # sign_in runs here, before anything is submitted
    okareo.run_simulation(..., target=Target(name="Portal", target=target), ...)
# leaving the block runs sign_out, even when the Run failed
```

- `sign_in` runs when the block opens, in a short-lived browser launched with
  the Target's own settings (`headless`, `channel`, `browser_args`), on a
  thread of its own. The session it leaves (cookies and local storage, and
  IndexedDB on Playwright 1.51 or later) is held in memory, never written to
  disk, and every conversation of the Run starts from it. If the sign-in
  raises, `sign_out` runs on the same page, best effort, and the error is
  raised: nothing has been submitted. Ctrl-C during the sign-in lets it
  finish, signs out, then stops.
- `sign_out` runs when the block closes (`target.close()`), in a short-lived
  browser holding the session, even when the Run failed. Raise in it when the
  sign-out did not land. Leaving the block never raises from the sign-out:
  `target.signed_out` is `True` when it landed and `False` when it raised (the
  error is logged). Ctrl-C during the sign-out lets it finish, then stops. A
  Target that does not override `sign_out` launches nothing when the block
  closes, and `signed_out` stays `None`.
- Okareo's SDK does not tell a Target that its Run is over; the end of the
  block says it. On Ctrl-C during the Run, stop the Run's listener
  (`okareo.model_under_test.stop_listener_for_run(run_id)`) and let the block
  sign out.
- Nothing signs in again during the Run. A conversation that lands signed out
  ends with its error.
- The sign-in, the conversations and the sign-out present the same Chrome: each
  context's user agent is read from the browser it is opened in.

## Changing how a command acts

A Target may override the commands Okareo sends, for a page the defaults
cannot work (a chat painted on a canvas, say): `click(st, x, y, label=None,
near=None)`, `type_text(st, text)`, `press(st, key)`, `scroll(st, x, y, dy)`,
`compose(st, text, x, y)`, and `settle(st, *, quiet_s, timeout_s, poll_s,
until_change)`, the wait. `settle` returns a `quiet.Done`: the last capture
(`shot`), its `sha1`, whether the turn `settled`, and the `record` of how the
wait ended; an override returns one too. Call `super()` for the default. `st`
is the conversation's state: `st["page"]` is its Playwright page, and nothing
else in `st` is part of the package's surface. Points arrive in the page's
pixels; with a `screen_selector`, Okareo's points have been moved onto the page
already.

`controls(st)` reads the screen's control list as it is now, the one sent to
Okareo with each frame. Each entry has `ref`, `role`, `names` (each a `text`
and the `source` it came from), `display` (the first name), `value`, `states`,
`within` (the list, menu or dialog it sits in, as `role` and `id`, or none) and
`bbox` (`x`, `y`, `w`, `h`). At most 80 entries, then one whose `role` is
`_dropped`: its `dropped` lists the controls off screen or covered, each with
its `bbox` and a `why`. Boxes are in the pixels of the screen Okareo is shown,
the `screen_selector` element's when there is one; `screen_origin(st)` is where
that screen sits on the page, `(0, 0)` without one. Add it to a box to press or
turn the wheel there with the page's mouse.

## The Driver is yours

The package ships no Driver. A Driver is a prompt you write and register with
the Okareo SDK: who the customer is, what they want, how they behave, and when
they are done. Okareo adds what every Driver in a web Run needs to read a
screen and make moves, so your text is about the customer only.
`{scenario_input.KEY}` is filled from the row's input. The start of the
example's `driver.md`:

```markdown
## Persona

-   **Identity:** You are role-playing a shopper on a store's website, chatting with the store's shopping helper to get one thing done.
-   **Mindset:** Friendly and to the point. You know what you want and how much you will spend; you answer the helper's questions from the Scenario Details and stop as soon as your request is done.

## Scenario Details

What you want: {scenario_input.task}
Budget: {scenario_input.budget}
Notes: {scenario_input.notes}

## Objectives

1. Open by telling the helper what you are shopping for, in one short sentence in your own words.
2. Answer the helper's questions from the facts above, and pick from what it shows you.
3. Stop once the helper has answered everything your request asks for. A question from the helper is something to answer, not the end.
```

It goes on with tactics, hard rules and when to end. A row that fills it:

```json
{"input": {"test_case": "SHOP-02", "task": "Find out which bag sizes the lump charcoal comes in.", "budget": "under $40", "notes": "Ask for lump charcoal first, then press the 'Show bag sizes' suggestion under Sprig's reply."}, "result": "Sprig suggests lump charcoal under $40 and says its bags come in 15 lb and 20 lb."}
```

And the SDK calls that run it, shortened from the example (the example reads
the server, the project and the names from its flags and environment):

```python
import json, os
from okareo import Okareo
from okareo.model_under_test import Driver, Target
from okareo_api_client.models import ScenarioSetCreate

okareo = Okareo(api_key=os.environ["OKAREO_API_KEY"], base_path="https://api.okareo.com")
rows = [json.loads(line) for line in open("scenarios.jsonl") if line.strip()]
scenario = okareo.create_scenario_set(ScenarioSetCreate(
    name="Shopwise shopper", seed_data=Okareo.seed_data_from_list(rows)))
driver = Driver(name="Shopwise shopper", prompt_template=open("driver.md").read(), temperature=0.0)

ShopwiseHelper.preflight()
run = okareo.run_simulation(
    name="Shopwise shopper",
    scenario=scenario,
    target=Target(name="Shopwise shopper", target=ShopwiseHelper()),
    driver=driver,
    checks=["result_completed"],
    max_turns=20,
    first_turn="driver",
)
print(run.app_link)
```

`run_simulation` serves the browser side from this process until the Run
finishes. The browser runs where this process runs, so the page can be on your
own network.

## The run record

With `--debug` in the example, `debug=True` on the Target, or
`OKAREO_WEB_TARGET_DEBUG=1`, the Target writes a record of its conversations to
a directory under `okareo_runs/` in the working directory (or under
`$OKAREO_RUN_DIR`), named for the time the first one started, in UTC. A
`RunDir` passed as `run_dir=` writes the same record without the video and the
HAR.

```
okareo_runs/20261006-063959/
  turns.jsonl            one line per turn, and one per conversation's video
  shots/                 turn00-9d0efd23.jpg: each frame Okareo saw
  ax/                    turn00-9d0efd23.json: the control list read from the page
  dom/                   turn00-9d0efd23.html.gz: the page's HTML at that frame
  video/                 9d0efd23.webm per conversation (and .mp4 with ffmpeg); debug mode
  9d0efd23.har           the page's network traffic per conversation; debug mode
```

`9d0efd23` is a conversation, `turn00` its first frame, taken before any turn
and so with no line of its own; turn `N`'s files are `turnNN-9d0efd23.*`. The
example's first turn in that conversation, one line of `turns.jsonl` shown
across several:

```json
{
  "turn": 1,
  "session": "9d0efd23",
  "commands": [
    {"op": "compose", "text": "I’m looking to buy a bag of lump charcoal for grilling, under $40.", "x": 1073, "y": 867},
    {"op": "wait", "quiet_s": 1.2, "timeout_s": 30.0}
  ],
  "results": [
    {"op": "compose", "ok": true},
    {"op": "wait", "ok": true, "settled": true}
  ],
  "frame_sha1": "7fb9cafdec473e0df1dfc424887afe59c2349b44",
  "video_offset_s": 10.29,
  "browser_ms": 1354,
  "at": "2026-10-06T06:40:09+00:00"
}
```

| Field | What it holds |
|---|---|
| `turn`, `session` | the turn's number in its conversation, and the conversation |
| `commands` | the commands as Okareo sent them |
| `results` | one per command: `op`, `ok`, `error` when it failed, and what the command reports (for `click`: `how`, which is `name`, `point` or `typed`, with `name`, `at` and `scrolled`; `landed` and `value` for `type`, with `typed_as`, `retried` and `note` when they apply; `settled` for `wait`; a command after a failed `type` is not run and says so) |
| `frame_sha1` | the SHA-1 of the capture the settle wait compared, the screenshot before it is encoded as the JPEG in `shots/`; it is not the saved file's hash |
| `video_offset_s` | where this turn is in the conversation's video, in seconds |
| `browser_ms` | the time this turn spent in the browser |
| `at` | when the line was written, UTC |

When a conversation ends in debug mode, one more line names its video:
`{"session": "9d0efd23", "video": ".../video/9d0efd23.webm", "at": "..."}`.

Three more files are written when your runner asks for them, with the helpers
in `okareo_vision.diagnostics` and a `RunDir` passed to the Target:
`setup(run_dir)` logs to `run.log`; `preflight(run_dir, okareo_base=...,
app_url=...)` writes `preflight.json` (Python, Playwright, proxy settings, and
whether Okareo and the page can be reached); `report(run_dir, run_link=...)`
writes `report.txt`, a per-turn summary with no credentials in it. The three
are `Diagnostics(run_dir)`'s `setup()`, `preflight(okareo_base=..., app_url=...)`
and `report(run_link=...)`, the same calls.

The record holds whatever the page showed and sent: the screenshots, the HTML
and the video show all of it. The HAR is recorded in Playwright's minimal mode
(no cookies, headers or security details) unless `OKAREO_WEB_TARGET_HAR=full`.
The video is written when its conversation ends.

## The server it needs

okareo-vision speaks version 1 of Okareo's web simulation protocol, and says so
when the Target registers. Okareo's cloud (`https://api.okareo.com`) speaks it,
and a local Okareo server works the same; the example talks to
`http://localhost:8000` unless `OKAREO_BASE_PATH` says otherwise. The oldest
Okareo SDK it works with is 0.0.151, the floor of the `okareo` it installs.

The traffic goes two ways. To Okareo: HTTPS to the API, and NATS over a
WebSocket to `wss://connect.ngs.global:443`, which every Run needs, a local
server included, so the machine needs that outbound connection. To the page:
the browser, directly. Chromium does
not read `HTTPS_PROXY`, so a page that must be reached through a proxy needs
it as a browser flag, `browser_args=["--proxy-server=http://proxy.example.com:8080"]`.

## When it goes wrong

### The page never shows a chat

- Watch it: `headless=False` (the example's `--headed`), or `--debug` and open
  `shots/turn00-*.jpg`, the first frame Okareo saw.
- `prepare` is where the chat gets opened. End it by waiting for the message
  box, as the example does, so the first frame has the chat in it; when
  `prepare` raises, the error is on the row.
- A chat in a panel over a busy page: `screen_selector` shows Okareo the panel
  alone.
- A chat in an iframe needs nothing set: typing, sending, pressing and the
  visible text reach into the frame. One served from another site entirely
  does not list its controls yet; press it at a point.
- A Flutter app needs nothing set: the library turns on its accessibility
  layer in every page and frame, so its controls are listed.
- When Okareo cannot find the message box: `composer_selector` and
  `send_selector` name it.

### The site refuses headless Chrome

The browser already presents itself as a normal Chrome: the automation marker
(`navigator.webdriver`) is off and the user agent has no "HeadlessChrome" in
it. When the site still refuses, show the browser and use the real Chrome:
`headless=False` and `channel="chrome"` (or `OKAREO_WEB_BROWSER=chrome`; run
`playwright install chrome` if it is not on the machine). Sites with a bot gate
commonly admit only the real Chrome, headed. A headed browser needs a display;
on a Linux server, run under `xvfb-run`.

### Sign-in needs MFA

A sign-in whose code can be scripted (a test account's fixed code, say)
belongs in `sign_in`, above: once per Run, held in memory. One that needs a
person cannot be scripted. Sign in by hand once, save the session, and start
every conversation from it:

```python
from okareo_vision import VisionTarget
from okareo_vision.auth import ensure_session

with ensure_session("https://portal.example.com/", storage_state="portal_session.json",
                    login_timeout=300) as (session, page):
    print("signed in at", session["landed_url"])

class Portal(VisionTarget):
    url = "https://portal.example.com/"
    storage_state = "portal_session.json"
```

With no `recipe`, `ensure_session` opens a visible browser at the url and waits
up to `login_timeout` seconds (600 unless you say) for you to sign in, MFA
included, then saves the session to `storage_state` with file mode 0600. Run
it before each Run: it reuses the saved session and asks you to sign in again
only when it no longer works (`verify=`, a function of `(page, api_base)`
returning a bool, says what "works" means for your site). The file is a
signed-in session; keep it out of version control. `form_login(...)` and
`entra_login(...)` in the same module are scripted sign-ins for sites without
MFA.

## What is not in the package

- **A Driver.** Yours, registered with the Okareo SDK.
- **Model calls.** This process holds no model key; Okareo reads the screens
  and picks the moves.
- **Scenarios and Checks.** Made with the Okareo SDK, as in the example.
- **The example and test pages.** They are in the
  [repository](https://github.com/okareo-ai/okareo-vision), not in the wheel.
- **Anything about a particular site.** Its url, sign-in and selectors belong
  in your subclass.

Documentation: <https://docs.okareo.com/simulation/web-ui>. Changes:
[CHANGELOG.md](https://github.com/okareo-ai/okareo-vision/blob/main/CHANGELOG.md).
License: Apache-2.0.
