Metadata-Version: 2.4
Name: jleechanorg-testing-utils
Version: 0.1.2
Summary: Generic server/MCP/browser testing utilities: HTTP client with request capture, evidence bundles, local server management, Playwright base, and raw request/response logging
Author-email: jleechan <jlee@jleechan.org>
License-Expression: MIT
Project-URL: Homepage, https://github.com/jleechanorg/worldarchitect.ai
Project-URL: Repository, https://github.com/jleechanorg/worldarchitect.ai
Project-URL: Issues, https://github.com/jleechanorg/worldarchitect.ai/issues
Keywords: testing,mcp,http,browser,playwright,evidence,request-capture,llm,server
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.25.0
Provides-Extra: browser
Requires-Dist: playwright>=1.40.0; extra == "browser"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Dynamic: license-file

# jleechanorg-testing-utils

Generic server/MCP/browser testing utilities for Python projects.

## Features

- **HTTP client** with request/response capture, credential redaction, and retry logic
- **MCP client** for Model Context Protocol (JSON-RPC 2.0) servers
- **Evidence bundles** – structured `/tmp/<project>/<branch>/iteration_NNN/` directories with git provenance
- **Local server management** – subprocess server lifecycle with port reservation
- **Generic test base** – abstract `BaseTestRunner` for server integration tests
- **Browser test base** – Playwright-based `BrowserTestBase` for UI tests
- **HTTP test base** – `HttpTestBase` using `requests`/`httpie` for web server tests
- **Server-side logging** – `LLMCallLogger` and `HttpRequestLogger` that servers can import to capture raw request/response payloads

## Installation

```bash
pip install jleechanorg-testing-utils

# For browser testing:
pip install "jleechanorg-testing-utils[browser]"
```

## Usage

### MCP Server Testing

```python
from testing_utils import MCPClient, BaseTestRunner, TestResult

class SmokeTest(BaseTestRunner):
    TEST_NAME = "mcp_smoke"
    PROJECT_NAME = "myproject"
    DEFAULT_BASE_URL = "http://localhost:8081"

    def run_scenarios(self) -> list[TestResult]:
        tools = self.client.tools_list()
        return [TestResult("tools_list", passed=bool(tools), detail=f"{len(tools)} tools")]

if __name__ == "__main__":
    SmokeTest().run()
```

### HTTP Web Server Testing

```python
from testing_utils.http_test import HttpTestBase

class ApiTest(HttpTestBase):
    TEST_NAME = "api_smoke"
    BASE_URL = "http://localhost:9000"
    PROJECT_NAME = "myproject"

    def run_scenarios(self):
        r = self.get("/health")
        r.assert_ok()
        r.assert_json_key("status", "healthy")

        # httpie-style subprocess
        rc, out, err = self.httpie("GET", "/api/users")
        return [{"name": "health_ok", "passed": r.ok}]

if __name__ == "__main__":
    test = ApiTest()
    results = test.run()
```

### Browser Testing (requires `[browser]` extra)

```python
from testing_utils.browser import BrowserTestBase

class HomepageTest(BrowserTestBase):
    BASE_URL = "http://localhost:3000"
    TEST_NAME = "homepage_smoke"
    PROJECT_NAME = "myproject"
    HEADLESS = True

    def run_scenarios(self):
        self.page.goto(self.BASE_URL)
        title = self.page.title()
        self.take_screenshot("homepage")
        return [{"name": "page_loads", "passed": bool(title), "detail": title}]

if __name__ == "__main__":
    HomepageTest().run()
```

### Server-Side LLM Logging

Import in your server to capture raw LLM request/response payloads for evidence bundles:

```python
from testing_utils.logging_capture import LLMCallLogger

# Use default env var names:
_llm_logger = LLMCallLogger(
    capture_path_env="RAW_LLM_CAPTURE_PATH",
    enabled_env="CAPTURE_RAW_LLM",
    max_chars_env="CAPTURE_RAW_LLM_MAX_CHARS",
)

def call_llm(provider, model, messages, system_instruction=None):
    _llm_logger.log_request(
        provider=provider,
        model=model,
        request_payload={"messages": messages},
        system_instruction=system_instruction,
    )
    response_text = _actual_llm_call(provider, model, messages)
    _llm_logger.log_response(
        provider=provider,
        model=model,
        response_text=response_text,
    )
    return response_text
```

Set `RAW_LLM_CAPTURE_PATH=/tmp/myproject/llm_captures.jsonl` before running tests to enable capture.

### Evidence Bundles

```python
from testing_utils.evidence import (
    get_evidence_dir,
    create_evidence_bundle,
    write_with_checksum,
    capture_provenance,
)

# Get branch-scoped evidence directory
evidence_dir = get_evidence_dir("my_test", project_name="myproject")
# → /tmp/myproject/<branch>/my_test/

# Create a structured bundle after running tests
create_evidence_bundle(
    evidence_dir,
    results={"passed": 5, "failed": 0},
    test_name="my_test",
    project_name="myproject",
    http_captures=client.get_captures_as_dict(),
)
# → /tmp/myproject/<branch>/my_test/iteration_001/
#   ├── README.md
#   ├── metadata.json
#   ├── results.json
#   └── request_responses.jsonl
```

## Environment Variables

| Variable | Module | Default | Description |
|---|---|---|---|
| `RAW_LLM_CAPTURE_PATH` | `logging_capture` | `` (disabled) | Path to JSONL file for LLM captures |
| `RAW_HTTP_CAPTURE_PATH` | `logging_capture` | `` (disabled) | Path to JSONL file for HTTP captures |
| `CAPTURE_RAW_LLM` | `logging_capture` | `true` | Enable/disable LLM capture |
| `CAPTURE_RAW_LLM_MAX_CHARS` | `logging_capture` | `20000` | Max chars per LLM payload field |
| `TEST_SCREENSHOT_DIR` | `browser` | branch-scoped `/tmp` | Screenshot output directory |
| `TEST_VIDEO_DIR` | `browser` | branch-scoped `/tmp` | Video output directory |
| `TEST_RECORD_VIDEO` | `browser` | `false` | Enable Playwright video recording |
| `TEST_BASE_URL` | `browser`, `http_test` | `http://localhost:8081` | Server URL override |

## Design Principles

- **No hardcoded project names** – everything is parameterized
- **No hardcoded URLs or credentials** – all configurable via constructor or env vars
- **Thread-safe** – logging capture uses file locks
- **Credential redaction** – API keys, JWTs, and auth tokens are automatically redacted in captures
- **Evidence-first** – every test run creates a structured evidence bundle with git provenance
