Metadata-Version: 2.4
Name: posthog-llm-doctor
Version: 0.1.0
Summary: Diagnose and repair why your LLM traces aren't showing up in PostHog.
Project-URL: Homepage, https://github.com/Ref34t/llm-doctor
Project-URL: Repository, https://github.com/Ref34t/llm-doctor
Project-URL: Issues, https://github.com/Ref34t/llm-doctor/issues
Project-URL: Changelog, https://github.com/Ref34t/llm-doctor/blob/main/CHANGELOG.md
Author-email: Mohamed Khaled <mohamed.khaled@9hdigital.com>
License: MIT License
        
        Copyright (c) 2026 Mohamed Khaled
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: cli,diagnostics,gen-ai,llm,observability,openinference,opentelemetry,otel,posthog,tracing
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Debuggers
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: httpx>=0.28.1
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.43.0
Requires-Dist: opentelemetry-sdk>=1.43.0
Requires-Dist: typer>=0.12
Description-Content-Type: text/markdown

# PostHog LLM Doctor

**Your LLM traces aren't showing up in PostHog? Find out why — in one command.**

`doctor check` sends a controlled test span to your PostHog project, reads it back, and tells you *whether* your LLM telemetry is landing and, if not, *exactly why* — with a plain-English fix. No guessing, no digging through ingestion logs.

The bug it was built for: frameworks like **OpenInference** (Arize, LlamaIndex) send spans PostHog *accepts* (HTTP 2xx) but never classifies as generations — so your dashboard stays empty with no error. The Doctor names that (and nine other failure modes) with `doctor check`, and repairs it in flight with `doctor fix` ([below](#repair-mode--doctor-fix)).

---

## Quickstart (under 5 minutes)

You need a PostHog project and its **project API key** (Project Settings → API key, starts with `phc_`).

```bash
# 1. Install the CLI (requires Python 3.11+) — the command is `doctor`
pipx install posthog-llm-doctor  # or:  uv tool install posthog-llm-doctor  /  pip install posthog-llm-doctor
# …bleeding edge from git:        uv tool install git+https://github.com/Ref34t/llm-doctor

# 2. Point it at your project (env var keeps the key out of your shell history)
export POSTHOG_PROJECT_KEY=phc_your_project_key

# 3. Diagnose
doctor check
```

That's it. You'll get a verdict like:

```
✗ FAIL — missing_gen_ai_operation_name

  endpoint   https://us.i.posthog.com/i/v0/ai/otel
  control    surfaced ✓
  repro      dropped ✗
  cause      The pipe works, but your framework's span shape is dropped: it has no
             gen_ai.operation.name, the attribute PostHog uses to classify a generation.
  remedy     Set gen_ai.operation.name (e.g. "chat") on your spans so PostHog recognises
             them as generations…
```

> **EU region?** Add `--endpoint https://eu.i.posthog.com/i/v0/ai/otel` (or set `$POSTHOG_OTEL_ENDPOINT`).

### Confirm ingestion, not just acceptance

A 2xx from PostHog only means *accepted* — not *ingested* (a bogus key still returns 2xx). To have the Doctor **verify the span actually surfaced**, give it read-back credentials too:

```bash
export POSTHOG_PERSONAL_API_KEY=phx_your_personal_key   # Personal API Keys, scope: query:read
export POSTHOG_PROJECT_ID=12345                          # numeric project id
doctor check
```

Without these, the Doctor still runs but reports `⚠ WARN — readback_not_configured` (it won't claim "healthy" off a 2xx it can't confirm).

---

## Repair mode — `doctor fix`

Diagnosis tells you *why* traces vanish. **`doctor fix` repairs them in flight.** It runs a local
OTLP receiver; point your app's exporter at it, and every span flows on to PostHog — broken
OpenInference spans get the missing `gen_ai.operation.name` (and their `llm.*` fields remapped to
`gen_ai.*`) added on the way, correctly-shaped spans pass through untouched. Trace ids are
preserved, so the repaired spans stay stitched into the right trace.

```bash
# 1. Start the Doctor (same project key as `doctor check`)
export POSTHOG_PROJECT_KEY=phc_your_project_key
doctor fix                      # listens on http://localhost:4318, forwards to PostHog

# 2. Point your app's OpenTelemetry exporter at the Doctor instead of PostHog
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
#    …then run your app as usual.
```

Each received span prints a one-line verdict so you can see the repair happen:

```
received openinference.repro → missing_gen_ai_operation_name → translated → forwarded (accepted)
received gen_ai.control       → healthy                      → forwarded unchanged (accepted)
```

> **EU region?** `doctor fix --endpoint https://eu.i.posthog.com/i/v0/ai/otel` (or `$POSTHOG_OTEL_ENDPOINT`).
> **Port in use?** `doctor fix --port 4319` (or `$LLM_DOCTOR_PORT`).

Fix mode reports *accepted* (2xx) per span; to prove a repaired span actually **surfaced**, run
`doctor check` (with read-back creds) — or see the opt-in live test in `tests/test_e2e.py`.

---

## What it checks

`doctor check` compares a **control** span (correctly shaped, should surface) against a **repro** span (shaped like a real framework, may not). The comparison isolates *"your pipe is broken"* from *"your span shape is wrong"*. Every run ends in one of three states:

| | Verdict | Means |
|--|---------|-------|
| ✓ | **PASS** | `healthy` — telemetry is reaching PostHog. |
| ✗ | **FAIL** | A confirmed problem: your spans are the wrong shape (`missing_gen_ai_operation_name`, the OpenInference case), PostHog rejected the send (`ingest_rejected`), or even correct spans aren't surfacing (`control_not_surfaced`). |
| ⚠ | **WARN** | Couldn't verify — no read-back credentials, a network/query read-back error, or an unexpected result. *Not* claimed as healthy or broken. |

The exit code drops it into CI:

- **`0`** — healthy, *or* accepted-but-unverified (no read-back credentials configured)
- **`2`** — a read-back credential you must fix (`readback_auth_failed`)
- **`1`** — everything else: a confirmed problem (wrong shape / ingest rejected / not surfacing), *or* a run that couldn't be verified (network or query read-back error, unexpected result)

So `doctor check && deploy` proceeds on a clean or gracefully-degraded run, and stops on a confirmed failure *or* one it couldn't verify.

---

## Why traces go missing

PostHog classifies a span as an LLM generation from one attribute: **`gen_ai.operation.name`**. OpenInference (Arize) spans carry `openinference.span.kind` + `llm.*` instead and omit `gen_ai.operation.name`, so they're accepted but silently dropped before classification.

- Root cause, with a runnable repro: **[PostHog/posthog#66614](https://github.com/PostHog/posthog/issues/66614)**
- OpenInference semantic conventions: <https://github.com/Arize-ai/openinference/blob/main/spec/semantic_conventions.md>
- PostHog OTel ingestion: <https://posthog.com/docs/ai-observability/installation/opentelemetry>

Repairing these spans in flight (so you don't have to change your app) is the Doctor's next version — *fix mode*.

---

## Configuration

Every value resolves in this order: **CLI flag > environment variable > default**.

| Flag | Env var | Required | Default |
|------|---------|----------|---------|
| `--project-key` | `POSTHOG_PROJECT_KEY` | yes — to send | — |
| `--personal-api-key` | `POSTHOG_PERSONAL_API_KEY` | for read-back | — |
| `--project-id` | `POSTHOG_PROJECT_ID` | for read-back | — |
| `--endpoint` | `POSTHOG_OTEL_ENDPOINT` | no | `https://us.i.posthog.com/i/v0/ai/otel` |

Prefer env vars over flags so secrets stay out of your shell history. Run `doctor check --help` for the full list.

---

## Development

```bash
git clone https://github.com/Ref34t/llm-doctor && cd llm-doctor
uv sync                 # install deps + dev tools
uv run pytest -q        # tests
uv run ruff check .     # lint
uv run doctor check     # run the CLI from source
```

---

*Part of the PostHog LLM Doctor project — diagnose with `doctor check`, repair in flight with `doctor fix`.*

> **Not affiliated with PostHog.** This is an independent, community tool built *for* PostHog users. "PostHog" is a trademark of PostHog, Inc.
