Metadata-Version: 2.4
Name: rubric-load-tester
Version: 0.3.1
Summary: Zero-config, LLM-powered API load tester.
Author: Rubric Team
License: MIT License
        
        Copyright (c) 2026
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/adetu-siyan/rubric_lt
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Testing :: Traffic Generation
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click
Requires-Dist: rich
Requires-Dist: aiohttp
Requires-Dist: numpy
Requires-Dist: groq
Requires-Dist: python-dotenv
Requires-Dist: pyyaml
Requires-Dist: pydantic
Requires-Dist: prompt_toolkit
Dynamic: license-file

# Rubric

Rubric is a CLI tool for testing APIs using their OpenAPI schema.

It reads the API's `/openapi.json`, generates request payloads with an LLM, sends the requests concurrently, and records response times and status codes. It can also use the failed requests to generate a short diagnostic report.

## How it works

1. **Read the OpenAPI schema**
   Rubric loads `/openapi.json` and identifies the available routes, methods, parameters, and request bodies.

2. **Generate test payloads**
   An LLM generates payloads for each endpoint. The payloads are split into valid inputs, edge cases, and malformed inputs.

3. **Run the requests**
   Requests are sent asynchronously with a configurable concurrency limit. Progress and response statistics are shown in the terminal.

4. **Generate the report**
   Rubric records latency percentiles, response status codes, error rates, and endpoint results. Failed requests can also be passed to the LLM for a short explanation.

## Route buckets

| Bucket | Methods          | Description                                             |
| ------ | ---------------- | ------------------------------------------------------- |
| A      | GET, DELETE      | Endpoints without a request body                        |
| B      | POST, PUT, PATCH | Endpoints that accept a request body                    |
| C      | Any              | Endpoints that need an ID created by an earlier request |

## Payload tiers

Rubric automatically alters its grading logic depending on the tier you are testing:

| Tier         | Count | Description / Grading Rule                                                               |
| ------------ | ----- | ---------------------------------------------------------------------------------------- |
| `happy_path` | 17    | Valid data. Expects a `2xx` success code. `4xx` and `500` are failures.                  |
| `edge_cases` | 17    | Boundary limits (e.g. max-length strings). Expects `2xx` or `4xx`. `500` is a failure.   |
| `malformed`  | 16    | Invalid data. Expects a `4xx` rejection. `2xx` (data accepted) or `500` are failures.    |

## Installation

Requires Python 3.11+.

```bash
pip install rubric-load-tester
```

**Windows users:** To avoid Windows Defender interfering with the `.exe` wrapper pip creates, you can run the CLI directly via Python:

```bash
python -m rubric run <url>
```

Copy the environment file:

```bash
cp .env.example .env
```

Then configure the LLM provider:

```env
LLM_PROVIDER=groq

GROQ_API_KEY=your_groq_key_here
GROQ_MODEL=meta-llama/llama-4-scout-17b-16e-instruct

OPENAI_API_KEY=your_openai_key_here
OPENAI_MODEL=gpt-4o
OPENAI_BASE_URL=https://api.openai.com/v1
```

## Usage

Run against a local API:

```bash
rubric run http://localhost:8000
```

Specify the request count, concurrency, payload tier, and thresholds:

```bash
rubric run http://localhost:8000 \
  --users 50 \
  --tier happy_path \
```

Run all payload tiers:

```bash
rubric run http://localhost:8000 --tier all --users 100
```

Inspect the API without running the load test:

```bash
rubric inspect http://localhost:8000
```

Disable the LLM diagnostic step:

```bash
rubric run http://localhost:8000 --no-diagnostics
```

Choose a provider for a single run:

```bash
rubric run http://localhost:8000 --provider groq
rubric run http://localhost:8000 --provider openai
```

### Authentication

Rubric supports interactive and automated authentication flows for APIs requiring authorization. During the first run against a protected API, Rubric will interview you to acquire credentials and test them before starting the load test.

Supported schemes:
* **Bearer / JWT**
* **Basic Auth**
* **API Keys** (Header-based)
* **Cookie-based sessions**
* **OAuth2** (Client flows)

Tokens are cached locally in `.rubric/auth.json` (for local dev) and refreshed silently when they expire. You can also bypass the interview by providing a token directly:
```bash
rubric run http://localhost:8000 --token "your-jwt-here"
```

### Self-signed certificates

For local development servers using a self-signed certificate:

```bash
rubric run https://localhost:8443 --allow-self-signed
```

This should only be used against a server you control. The option allows Rubric to continue when the target certificate cannot be verified.


## Configuration (rubric.json)

While Rubric is designed to be zero-config via CLI flags, you can store your settings in a `rubric.json` file in your project root to standardize tests across your team.

```json
{
  "target": "http://localhost:8000",
  "tier": "all",
  "users": 50,
  "assertions": [
    "status < 500",
    "p95 < 300"
  ],
  "fail_on_degraded": true
}
```

Rubric will automatically pick up this file when you run:

```bash
rubric run
```

If your project does not use an OpenAPI spec, you can use `rubric.json` to manually define your endpoints (or generate it automatically using `rubric scan`).

## Command options

| Flag                  | Default      | Description                                       |
| --------------------- | ------------ | ------------------------------------------------- |
| `--users`             | `10`         | Target concurrent users (Closed Workload Model)   |
| `--rps`               | `-`          | Target requests per second (Open Workload Model)  |
| `--tier`              | `happy_path` | `happy_path`, `edge_cases`, `malformed`, or `all` |
| `--assert`            | `-`          | Custom assertions (e.g., `"status == 200"`)       |
| `--no-diagnostics`    | `False`      | Disable LLM failure diagnostics                   |
| `--save`              | `True`       | Save the JSON report                              |
| `--provider`          | `.env` value | Override the configured LLM provider              |
| `--allow-self-signed` | `False`      | Allow unverified TLS certificates                 |

## Output

During a run, Rubric displays the current request and response statistics:

```text
RUBRIC · 847 requests · 142.3 RPS · 5.9s

┌────────────────────────┬──────┬──────┬──────┬──────┬──────┬───────┐
│ Endpoint               │ RPS  │ P95  │ 2xx  │ 4xx  │ 5xx  │ Total │
├────────────────────────┼──────┼──────┼──────┼──────┼──────┼───────┤
│ POST /users            │ 142  │ 84ms │ 891  │ 23   │ 6    │ 920   │
│ GET /users/{user_id}   │ 89   │ 31ms │ 412  │ 0    │ 1    │ 413   │
│ POST /products         │ 201  │ 19ms │ 1204 │ 11   │ 0    │ 1215  │
└────────────────────────┴──────┴──────┴──────┴──────┴──────┴───────┘
```

At the end of the run, each endpoint is classified as `PASSED`, `DEGRADED`, or `FAILED` based on the configured thresholds.

```text
┌────────────────────────┬────────────┬─────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┐
│ Endpoint               │ Status     │ RPS │ P50  │ P95  │ P99  │ 2xx  │ 4xx  │ 5xx  │ Err% │
├────────────────────────┼────────────┼─────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┤
│ POST /users            │ PASSED     │ 142 │ 34ms │ 84ms │201ms │ 891  │ 23   │ 6    │0.67% │
│ GET /users/{user_id}   │ DEGRADED  │ 89  │ 12ms │312ms │ 67ms │ 412  │ 0    │ 1    │0.24% │
│ POST /products         │ FAILED     │ 201 │ 18ms │ 44ms │ 98ms │ 980  │ 11   │ 24   │2.10% │
└────────────────────────┴────────────┴─────┴──────┴──────┴──────┴──────┴──────┴──────┴──────┘

SUITE FAILED

Passed: 1
Degraded: 1
Failed: 1

Total: 2507 requests
Duration: 17.6s
RPS: 142.4

Thresholds:
P95 < 200ms
Error rate < 1%
```

With diagnostics enabled, failed requests can also include an explanation:

```text
1. POST /users

   Payload: {"name": "", "email": "x@y.com", "role": ""}

   Cause: empty 'role' string reaches the database layer.

   Fix: validate the field before sending the request.
```

## Sample API

The repository includes a small FastAPI application for testing Rubric locally.

Install FastAPI and Uvicorn:

```bash
pip install fastapi uvicorn
```

Start the sample API:

```bash
uvicorn sample_api:app --reload
```

Then run Rubric in another terminal:

```bash
rubric run http://localhost:8000
```

Or inspect the API first:

```bash
rubric inspect http://localhost:8000
```

## JSON reports

Rubric saves a JSON report after each run:

```text
rubric_report_<url>_<timestamp>.json
```

Example:

```json
{
  "suite_result": "SUITE FAILED",
  "thresholds": {
    "p95_ms": 200,
    "error_rate_pct": 1.0
  },
  "summary": {
    "total_requests": 2507,
    "duration_seconds": 17.6,
    "overall_rps": 142.4,
    "connection_drops": 0,
    "passed": 1,
    "degraded": 1,
    "failed": 1
  },
  "endpoints": [],
  "failure_diagnostics": []
}
```

`SUITE FAILED` is also used as the CI result. Rubric exits with status code `1` when the configured thresholds are exceeded.

## Project structure

```text
rubric/
├── core/
│   ├── schema.py
│   ├── synthesizer.py
│   └── llm.py
├── engine/
│   └── runner.py
├── report/
│   └── reporter.py
├── cli/
│   └── main.py
├── sample_api.py
├── requirements.txt
├── setup.py
└── .env.example
```

### Main components

* `core/schema.py` - reads the OpenAPI schema and categorizes routes
* `core/synthesizer.py` - generates request payloads
* `core/llm.py` - LLM provider handling
* `engine/runner.py` - sends requests and collects metrics
* `report/reporter.py` - builds reports and diagnostics
* `cli/main.py` - CLI entry point
* `sample_api.py` - local API used for testing

## Security notes

### Data sent to the LLM

When diagnostics are enabled, Rubric sends information about failed requests to the configured LLM provider.

This includes:

* the endpoint
* a redacted request payload
* the first 300 characters of the response

Rubric attempts to remove values from fields such as `password`, `token`, and similar secret-looking fields. It also looks for JWTs, bearer tokens, API keys, and email addresses.

This redaction is not guaranteed to catch every sensitive value. If the target API can return sensitive information in error responses, use:

```bash
rubric run http://localhost:8000 --no-diagnostics
```

### Target reset

Rubric does not call `/reset` by default.

The reset endpoint is included for the sample API. If you need it:

```bash
rubric run http://localhost:8000 --reset-target
```

or:

```env
RUBRIC_ALLOW_RESET=1
```

Only enable this when testing an environment where resetting the data is acceptable.

### Credentials

Rubric stores its local configuration and authentication files with owner-only permissions:

```text
~/.rubric/config.json
.rubric/auth.json
```

For CI or other non-interactive runs, stored passwords are removed after the run.

For authentication, prefer:

```env
RUBRIC_AUTH_TOKEN=...
```

over passing a token directly on the command line.

### Reports

Values originating from the target API or the LLM are HTML-escaped before being included in reports.

The generated report also includes a restrictive Content Security Policy.
