Metadata-Version: 2.5
Name: newmeasure-cli
Version: 0.1.0
Summary: NewMeasure eval CLI
Requires-Python: >=3.11
Requires-Dist: httpx>=0.28
Requires-Dist: python-dotenv>=1.0
Requires-Dist: rich>=13.0
Requires-Dist: typer>=0.16
Description-Content-Type: text/markdown

# NewMeasure CLI

Upload evaluation tasks, start cloud runs, check results, and download artifacts
from NewMeasure. Requires Python 3.11 or later and a NewMeasure customer API key.

## Install

```sh
uv tool install newmeasure-cli
```

Alternatively, use `pipx install newmeasure-cli`, or install with
`pip install newmeasure-cli` inside a virtual environment. The executable is `newm`.

## Sign in

```sh
newm login
```

Enter your customer API key when prompted. The default API is
`https://api.newmeasure.cc`. Local configuration is stored in
`~/.newm/config.json` with owner-only permissions.

## Run a task

```sh
# Upload and run a local task directory
newm run ./my-task --no-wait

# Or run an already uploaded task
newm run t_abc123 --no-wait
```

`--no-wait` returns the run ID immediately; execution continues in the cloud.
Without it, the CLI waits for completion. Check status with:

```sh
newm result <run-id>
```

For browser tasks, provide application credentials in a local environment file
using the field names declared by the task, for example:

```dotenv
username=YOUR_APPLICATION_EMAIL
password=YOUR_APPLICATION_PASSWORD
```

Keep that file out of version control. Submit it for the run:

```sh
newm run ./my-task --credentials-file credentials.env --no-wait
```

The CLI sends the declared credential values over HTTPS for that run. Cloud
provider and model credentials are managed by the NewMeasure service.

## Download browser artifacts

Use the trial ID printed by `newm result`:

```sh
newm artifacts <run-id> <trial-id> --output ./artifacts/<trial-id>
```

Downloads include the available exports, evidence, and manifest. The CLI verifies
file sizes and SHA-256 checksums. Browser smoke completion is an operational
result, not a content-quality score. Inspect the manifest for cleanup status
before repeating a failed task.

## Other commands

```sh
newm upload ./my-task
newm tasks
newm runs
newm runs <task-id>
newm --help
```

Task IDs persist across uploads; each execution gets a fresh trial ID. Browser
task bundles include the instruction, task settings, and optional scenario and
product context. HTTP-agent tasks also require their seed and verifier files.

## Development

From the CLI source directory:

```sh
uv sync
uv run newm --help
uv run pytest
```
