Metadata-Version: 2.1
Name: zhanla
Version: 0.1.2.19
Summary: CLI from Zhanla for running and uploading AI component evaluations
Project-URL: Homepage, https://github.com/zhanla-ai
Project-URL: Repository, https://github.com/zhanla-ai
Author-email: Zhanla <chouweimin@berkeley.edu>
License: Proprietary
Requires-Python: >=3.10
Requires-Dist: httpx>=0.28.0
Requires-Dist: rich>=13.0.0
Requires-Dist: supabase>=2.0.0
Requires-Dist: tree-sitter-python>=0.25.0
Requires-Dist: tree-sitter-typescript>=0.23.0
Requires-Dist: tree-sitter>=0.25.0
Requires-Dist: typer>=0.12.0
Description-Content-Type: text/markdown

# zhanla

Command-line interface for validating, uploading, running, and evaluating Zhanla SDK components.

The CLI can be installed on its own for auth, listing web resources, dataset/component upload, and fully web-backed runs. Local component execution requires the matching language SDK.

## Installation

```bash
pip install zhanla
```

Requires Python `>=3.10`.

Optional local execution dependencies:

```bash
pip install zhanla-sdk-py
npm install @zhanla/sdk-ts
```

Check the install:

```bash
zhanla --version
zhanla status
```

## Authentication

Authenticate with an SDK API key:

```bash
zhanla login
```

The key format is:

```text
bm_kid_XXXX.bm_sec_XXXX
```

Credentials are stored at `.zhanla/credentials.json` in the current repo root. The saved login is repo-local, not global machine state.

By default the CLI targets the production control plane. For local development, set `ZHANLA_API_URL` to a **loopback** host to repoint the CLI without re-logging-in, e.g. `ZHANLA_API_URL=http://localhost:3001 zhanla serve --repo acme/app`. If `ZHANLA_API_URL` is unset, a loopback `NEXT_PUBLIC_APP_URL` from `.env.local` is also honored. Non-loopback hosts are ignored because the CLI sends a bearer token. Only `localhost`/`127.0.0.1`/`::1` are accepted.

Remove saved credentials:

```bash
zhanla logout
```

Authenticated commands exchange the saved API key for a fresh short-lived token. If you are not logged in, they exit with:

```text
Not logged in. Run `zhanla login` first.
```

## Environment

The CLI loads env files from the current directory or nearest ancestor containing one of:

1. `.env.local`
2. `.env`
3. `.env.example`

Existing shell variables are not overwritten. Provider API keys such as `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_API_KEY`, and `OPENROUTER_API_KEY` should live there for local component execution.

The CLI always connects to the hosted Zhanla app at `https://benchmark-black.vercel.app`. Base URL env overrides are ignored for CLI backend calls.

## Command overview

```bash
zhanla validate COMPONENT.py:component-key
zhanla upload COMPONENT.py:component-key
zhanla upload --dataset DATASET.json --name "Dataset name"
zhanla run COMPONENT.py:component-key --dataset DATASET.json --eval EVAL.py:eval-key
zhanla list datasets
zhanla list autoraters
zhanla list components
```

Advanced local-runner/refinement commands:

```bash
zhanla serve --repo ORG/REPO
zhanla apply-patch --run RUN_ID
```

## Terminal output

The CLI prints progress and a compact completion summary for run, upload, daemon,
patch, and output-file flows. Run summaries show how many rows completed, how
many were scored, failure counts, upload status, result links, and output files
when applicable.

Errors redact common token/key shapes and secret-like environment values before
printing. Command failures include next steps such as the validation, login,
list, or retry command to run next. Empty list results also include next steps to
upload resources or relax filters.

## `zhanla validate`

Validate component definitions offline, without auth or network:

```bash
zhanla validate components.py
zhanla validate components.ts:priority-tool
```

Targets resolve by SDK component `key`. If a file contains multiple runnable components or evals, include the `:key` suffix.

## `zhanla upload`

Upload a component or eval definition:

```bash
zhanla upload components.py:priority-tool
zhanla upload evals.ts:priority-eval --yes
```

Upload a standalone dataset:

```bash
zhanla upload --dataset tickets.json
zhanla upload --dataset tickets.csv --name "Support Tickets" --yes
```

Component/eval upload syncs the full discoverable closure. Dataset upload creates or updates rows idempotently.

## `zhanla run`

`zhanla run` supports three common execution modes:

1. Local component + local eval.
2. Local component + web autorater.
3. Fully web-backed component + dataset + autorater.

### Local component + local eval

```bash
zhanla run components.py:priority-tool \
  --dataset tickets.json \
  --eval evals.py:priority-eval \
  --dry-run \
  --yes
```

For TypeScript:

```bash
zhanla run components.ts:priority-tool \
  --dataset tickets.json \
  --eval evals.ts:priority-eval \
  --dry-run \
  --yes
```

`--dry-run` keeps both component and eval local and does not require login.

Without `--dry-run`, the component still runs locally, definitions and rows/results are synced, then the eval runs as a managed evaluate-only run. That path requires `zhanla login`.

### Local component + web autorater

```bash
zhanla run components.py:priority-tool \
  --dataset tickets.json \
  --web-eval support-quality \
  --yes
```

You can also use a web dataset:

```bash
zhanla run components.py:priority-tool \
  --web-dataset support-tickets \
  --web-eval support-quality \
  --yes
```

This flow runs the local component, uploads or reuses rows and outputs, starts the managed autorater, and polls until completion.

### Fully web-backed run

```bash
zhanla run \
  --web-config support-agent \
  --web-dataset support-tickets \
  --web-eval support-quality \
  --yes
```

The server generates component responses using the web component definition and org provider keys, then runs the managed autorater.

### Run flags

| Flag | Short | Description |
|---|---|---|
| `--eval <spec>` | `-e` | Local eval target in `file.py:key` or `file.ts:key` form |
| `--dataset <path>` | `-d` | Local `.json` or `.csv` dataset |
| `--web-eval <key-or-id>` |  | Managed autorater key or ID |
| `--web-dataset <key-or-id>` |  | Web dataset key or ID |
| `--web-config <key-or-id>` |  | Web component key or ID for fully web-backed runs |
| `--dry-run` |  | Execute locally and skip sync/upload |
| `--yes` | `-y` | Skip confirmation prompts |
| `--max-rows <N>` |  | Run only the first `N` dataset rows |
| `--fail-on-row-error` |  | Exit non-zero when any row has component or eval execution errors |
| `--output <path>` | `-o` | Write machine-readable per-item result JSON |

Removed or unsupported `zhanla run` flags include `--model-endpoint`, `--parallel`, and `--verbose`.

### Valid combinations

- Provide exactly one component source: local target or `--web-config`.
- Provide exactly one dataset source: `--dataset` or `--web-dataset`.
- Do not combine `--eval` and `--web-eval`.
- `--web-config` only supports `--web-config + --web-dataset + --web-eval`.
- `--dry-run` cannot combine with `--web-config` or `--web-eval`.
- `--max-rows` must be positive.
- Local component and local eval files must use the same language.
- `Skill` is prompt-only configuration and cannot be run directly.

## Target syntax and discovery

Use `file.py[:component-key]` for Python and `file.ts[:component-key]` for TypeScript. The suffix is the SDK component `key`, not the Python variable name, TypeScript export name, or display `name`.

Python discovery imports the file and scans module-level `zhanla` component instances.

TypeScript discovery uses the SDK helper binary internally. Components must be exported, but normal users should run the main `zhanla` CLI.

If a target accidentally uses a display name or variable/export name, the CLI rejects it and suggests matching keys when possible.

## Dataset formats

### JSON

JSON datasets must be a top-level array whose first item contains `_schema`:

```json
[
  {
    "_schema": {
      "message": "string",
      "expected_output": "object"
    }
  },
  {
    "_config": {
      "name": "support tickets",
      "description": "Priority classification cases"
    }
  },
  {
    "message": "Reset my password",
    "expected_output": { "priority": "normal" }
  }
]
```

Leading `_schema` and `_config` rows are metadata and are not executed.

Object-shaped JSON datasets with `schema` and `rows` are not accepted.

### CSV

CSV datasets use the header row as field names:

```csv
message,expected_output
Reset my password,normal
```

CSV values are loaded as strings.

### Empty datasets

If the resolved dataset has no rows, the CLI exits with:

```text
Dataset is empty.
```

## Eval input contract

Local evals and CLI-managed eval-only runs receive canonical text kwargs:

- `model_input`: serialized dataset row, or serialized row `input` when present.
- `model_response`: serialized component output.
- `expected_output`: serialized row `expected_output`, row `output`, or the full row.

Python `CodeEval` functions receive keyword arguments:

```python
def score(model_response, expected_output=None, model_input=None, **_):
    response = zhanla.parse_json_response(model_response)
    expected = zhanla.parse_json_response(expected_output or "{}")
    return {"score": 1.0 if response == expected else 0.0}
```

TypeScript `CodeEval` functions receive one kwargs object:

```ts
fn: ({ model_response, expected_output }) => {
  const response = model_response ? JSON.parse(model_response) : {};
  const expected = expected_output ? JSON.parse(expected_output) : {};
  return { score: response.priority === expected.priority ? 1.0 : 0.0 };
}
```

`model_response_format` / `modelResponseFormat` is synced as eval metadata for platform compatibility. Current local and CLI-managed eval-only execution still passes strings, so parse structured data inside the eval body.

## Local execution semantics

- `Tool`: runs local code and normalizes non-object output to `{"result": value}`.
- `CodeEval`: runs local scoring code and normalizes non-object output to `{"score": value}`.
- `Skill`: raises an error when executed directly.
- `Agent`: requires `model` plus `client` or `runner`.
- `LLMProcessor`: requires `model` plus `client` or `runner`.
- `LLMEval`: requires `model` plus `client` or `runner`.
- `Checklist`: runs child evals and computes a weighted score.
- `EvalTree`: routes through branches and aggregates leaf scores.
- `Orchestration`: executes the DAG and returns the final executed step output.

The CLI validates the first component output against `output_schema` when present.

## List commands

```bash
zhanla list datasets
zhanla list autoraters
zhanla list components
```

Useful filters:

```bash
zhanla list datasets --component-type tool --name support
zhanla list autoraters --component-type agent --component-id component-uuid
zhanla list components --component-type orchestration --name support
zhanla list components --all --verbose
```

Supported component filters are `agent`, `skill`, `tool`, and `orchestration`.

## Output JSON

`--output FILE.json` writes:

```json
{
  "mean_score": 0.9,
  "run_id": "run-id",
  "items": [
    {
      "item_id": "0",
      "model_input": "...",
      "model_response": "...",
      "expected_output": "...",
      "score": 1.0,
      "reason": "..."
    }
  ]
}
```

For managed runs, output is written after remote results are available.

## File locations

| Path | Purpose |
|---|---|
| `.zhanla/credentials.json` | Repo-local SDK credentials and org metadata |
| `.zhanla/version_cache.json` | Repo-local update-notifier cache |

## Running tests

```bash
python -m pytest packages/cli/tests -v
```

Broaden to SDK tests when discovery, execution, manifests, eval contracts, or helper integration change.
