Metadata-Version: 2.5
Name: newmeasure-cli
Version: 0.2.0
Summary: NewMeasure eval CLI
Requires-Python: >=3.11
Requires-Dist: httpx>=0.28
Requires-Dist: python-dotenv>=1.0
Requires-Dist: rich>=13.0
Requires-Dist: typer>=0.16
Description-Content-Type: text/markdown

<!-- prettier-ignore -->
<div align="center">

# NewMeasure CLI

*Upload eval tasks, start cloud runs, and download results with `newm`*

[Install](#install) • [Usage](#usage) • [Browser tasks](#browser-tasks) • [Commands](#commands) • [Development](#development)

</div>

`newm` is the customer-facing client for NewMeasure. It uploads task directories, starts runs in the cloud, shows results, and downloads stored artifacts. Requires Python 3.11 or later and a NewMeasure customer API key.

## Install

```bash
uv tool install newmeasure-cli
```

<details>
<summary>Other install options</summary>

```bash
pipx install newmeasure-cli
# or, inside a virtual environment
pip install newmeasure-cli
```

</details>

## Usage

Sign in with your customer API key:

```bash
newm login
```

The default API is `https://api.newmeasure.cc`. Configuration is stored in `~/.newm/config.json` with owner-only permissions.

Run a task:

```bash
# Upload and run a local task directory
newm run ./my-task --no-wait

# Or run an already uploaded task, several times
newm run t_abc123 --n-trials 5 --no-wait
```

`--no-wait` returns the run ID immediately; the run continues in the cloud. Without it, the CLI waits for completion. Check status with:

```bash
newm result <run-id>
```

> [!NOTE]
> Task IDs persist across uploads (the first upload writes `task_id` into `task.toml`). Each execution gets a fresh trial ID.

## Browser tasks

Browser tasks test a web product: an AI agent uses it in a cloud browser. Put the application login in a local env file, using the field names the task declares in `credential_fields`:

```dotenv
username=YOUR_APPLICATION_EMAIL
password=YOUR_APPLICATION_PASSWORD
```

Keep that file out of version control, and submit it with the run:

```bash
newm run ./my-task --credentials-file credentials.env --no-wait
```

Only the declared fields are sent, over HTTPS (plain HTTP is allowed for `localhost`). Cloud and model credentials are managed by NewMeasure.

Download a trial's artifacts using the trial ID from `newm result`:

```bash
newm artifacts <run-id> <trial-id> --output ./artifacts/<trial-id>
```

Downloads include `outputs/` (the files the product generated, such as PDF and HTML exports), `score.json` (the reward and per-check breakdown), and `manifest.json`. The CLI checks each file's size and SHA-256.

A browser task bundle contains `task.toml`, `instruction.md`, and, when present, `scenario.yaml`, `product.md` (or `../../shared/product.md`), and the `environment/` and `tests/` folders.

## Commands

```bash
newm login                      # store API token
newm upload ./my-task           # validate and upload a task
newm run <task-id | dir>        # run an eval
newm result <run-id>            # run details and trial results
newm artifacts <run-id> <trial-id> [--output DIR]
newm tasks                      # list uploaded tasks
newm runs [<task-id>]           # list runs
newm harbor ...                 # pass through to the Harbor CLI
newm --help
```

## Development

From this directory:

```bash
uv sync
uv run newm --help
uv run pytest
```
