Metadata-Version: 2.5
Name: mavpose
Version: 0.2.1
Summary: Clean, unit-correct telemetry from drone flight logs (MAVLink .tlog, ArduPilot .bin, PX4 .ulg), with an optional plain-English plot assistant
License: MIT
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: pandas>=2.0.0
Requires-Dist: pyarrow>=14.0.0
Requires-Dist: pymavlink>=2.4.48
Requires-Dist: pyulog>=1.1.0
Provides-Extra: chat
Requires-Dist: chromadb>=0.5.0; extra == 'chat'
Requires-Dist: langchain-chroma>=0.1.0; extra == 'chat'
Requires-Dist: langchain-core>=0.2.0; extra == 'chat'
Requires-Dist: langchain-openai>=0.1.0; extra == 'chat'
Requires-Dist: matplotlib>=3.8; extra == 'chat'
Requires-Dist: openai>=1.0.0; extra == 'chat'
Requires-Dist: python-dotenv>=1.0.0; extra == 'chat'
Requires-Dist: tiktoken>=0.5.0; extra == 'chat'
Provides-Extra: dev
Requires-Dist: chromadb>=0.5.0; extra == 'dev'
Requires-Dist: langchain-chroma>=0.1.0; extra == 'dev'
Requires-Dist: langchain-core>=0.2.0; extra == 'dev'
Requires-Dist: langchain-openai>=0.1.0; extra == 'dev'
Requires-Dist: matplotlib>=3.8; extra == 'dev'
Requires-Dist: mcp<3,>=2.3; extra == 'dev'
Requires-Dist: openai>=1.0.0; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: python-dotenv>=1.0.0; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: tiktoken>=0.5.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9; extra == 'docs'
Requires-Dist: mkdocs<2,>=1.6; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.25; extra == 'docs'
Provides-Extra: mcp
Requires-Dist: mcp<3,>=2.3; extra == 'mcp'
Description-Content-Type: text/markdown

<div align="center">

# 🧭 MAVPose

**Ask questions about your drone flight in plain English. Get plots.**

*Inspired by the Log Pose from One Piece — it locks onto your flight data and charts a course through it.*

[![Python](https://img.shields.io/badge/Python-3.10%2B-blue?logo=python&logoColor=white)](https://www.python.org/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
[![CI](https://github.com/AyushMaria/MAVPose/actions/workflows/ci.yml/badge.svg)](https://github.com/AyushMaria/MAVPose/actions/workflows/ci.yml)
[![PyPI version](https://badge.fury.io/py/mavpose.svg)](https://pypi.org/project/mavpose/)

</div>

---

MAVPose is a headless CLI tool that turns a natural language prompt into a matplotlib plot of your MAVLink flight log. It runs a **two-phase pipeline**: the parent process first extracts clean, time-aligned telemetry into a Parquet file (like a headless database query), then hands the LLM a precise column schema and a `pd.read_parquet()` call — no raw binary data, no pymavlink in the generated script. This drastically reduces hallucinations and self-healing loops.

```
$ mavpose flight.tlog --prompt "Plot altitude over time"

📂 Parsing log schema: flight.tlog
✅ Schema indexed.

🔍 Finding relevant message types for: 'Plot altitude over time'
   → ['GLOBAL_POSITION_INT', 'VFR_HUD']

🗃️  Extracting telemetry to Parquet...
┌──────────────────────────────────────────────────────────┐
│  🗃️  Extracted Parquet schema                               │
├──────────────────────────────────────────────────────────┤
│  [GLOBAL_POSITION_INT]  1842 rows                          │
│      time_s: float64 s  [0.0 … 312.4]                     │
│      alt: float64 m  [487.32 … 512.1]                     │
│  [VFR_HUD]  1842 rows                                      │
│      time_s: float64 s  [0.0 … 312.4]                     │
│      alt: float64 m  [476.1 … 501.3]                      │
└──────────────────────────────────────────────────────────┘
   Saved → /path/to/telemetry.parquet

✍️  Generating plot script with z-ai/glm-5.1...
⚙️  Running script...
✅ Plot saved to: /path/to/plot.png
```

---

## Architecture

```
┌──────────────────────────────────────────────────────────┐
│  PHASE 1 — Extraction  (parent process, no LLM)             │
├──────────────────────────────────────────────────────────┤
│                                                             │
│  LogExtractor.schema_only()                                 │
│    └─ pymavlink fast scan → {msg_type: {fields, count}}     │
│    └─ ChromaDB embeddings (semantic field search)           │
│                                                             │
│  find_relevant_data_types(prompt)                           │
│    └─ vector similarity → ["GLOBAL_POSITION_INT", ...]      │
│                                                             │
│  LogExtractor.extract_all()  ←  full row materialisation    │
│    └─ per-msg-type DataFrames, time_s column, cast numerics │
│                                                             │
│  LogExtractor.export_parquet(msg_types, path)               │
│    └─ clean telemetry.parquet + schema summary dict         │
│                                                             │
├──────────────────────────────────────────────────────────┤
│  PHASE 2 — LLM Plot Generation                              │
├──────────────────────────────────────────────────────────┤
│                                                             │
│  LLM receives:                                              │
│    └─ parquet_file path                                    │
│    └─ schema: {msg_type: {rows, columns: {col: dtype+range}}} │
│    └─ output_file path (.png)                              │
│                                                             │
│  LLM writes pandas + matplotlib script                      │
│    df = pd.read_parquet(parquet_file)                       │
│    df_x = df[df['msg_type'] == 'X']                         │
│    ...                                                      │
│                                                             │
│  run_script() → subprocess exec                            │
│    └─ on failure: attempt_to_fix_script() up to N×         │
│                                                             │
│  → plot.png saved next to log file                          │
└──────────────────────────────────────────────────────────┘
```

**Why this matters:** In the old design, the LLM had to write `pymavlink` code to parse a binary `.tlog` — a notoriously tricky API with blocking/non-blocking subtleties, timestamp inconsistencies, and message-type guesswork. By the time the LLM touched the data, it was flying blind. Now it receives a typed, time-aligned Parquet file with exact column names, dtypes, and value ranges. Writing `pd.read_parquet()` + `matplotlib` code against a known schema is trivially reliable.

---

## Features

- **Plain-English plots** — describe any flight data; MAVPose figures out which MAVLink fields to use
- **ArduPilot and PX4** — one API for MAVLink `.tlog`, ArduPilot DataFlash `.bin`/`.log` and PX4 ULog `.ulg` logs
- **Headless extraction layer** — `LogExtractor` parses every supported format into per-message-type DataFrames before the LLM is ever invoked:
  - **one clock** — every message type shares a single `time_s` axis (tlog receive time, DataFlash `TimeUS`, or ULog `timestamp` from logging start); native time fields are kept as columns
  - **real units** — scaled fields are converted using unit metadata from the log or the autopilot's own message definitions (`alt` mm → m, `lat` degE7 → deg, `voltage_battery` mV → V, …) and every column's unit is reported; pass `convert_units=False` for raw values
  - **unknown values removed** — "not provided" markers (MAVLink `current_battery = -1`, `eph = UINT16_MAX`; PX4 `battery_status.current_a = -1`, …) become NaN instead of fake readings, using the `invalid` markers from the official MAVLink and PX4 definitions; counts appear in `extractor.unknown_counts` and the LLM schema. Pass `filter_unknown=False` to keep them
  - **corruption-tolerant** — bad packets are skipped and counted (`extractor.stats`) instead of silently ending the parse
- **Clean Parquet handoff** — only the relevant message types are exported; the LLM sees exact column names, dtypes, min/max ranges — no binary guesswork
- **Semantic field search** — ChromaDB vector embeddings surface the most relevant message types for your query
- **Self-healing scripts** — LLM debugs and rewrites failing scripts up to N times; fix prompt also includes the full schema
- **Restricted execution** — generated code runs in a clean subprocess (no API keys in its environment) under a PEP 578 audit hook that blocks process spawning, network access and file writes outside the output folder, with CPU/memory limits and a timeout. This is defence in depth, not a hard security boundary: for untrusted logs or prompts, also run MAVPose in a container
- **Interactive REPL mode** — omit `--prompt` to enter a live loop; Parquet is re-extracted per query with the relevant types
- **Configurable model** — drop in any OpenRouter model with a one-line `.env` change
- **Persisted vector store** — ChromaDB is saved to disk; re-running on the same log skips re-embedding
- **CI-tested** — lint (ruff) and pytest run on every push

---

## Tech Stack

| Layer | Library / Service |
|---|---|
| LLM | [OpenRouter](https://openrouter.ai) → Z.ai GLM-5.1 (default) |
| Embeddings | OpenRouter → `openai/text-embedding-3-small` |
| LLM framework | LangChain (LCEL) — `langchain-openai`, `langchain-chroma` |
| Vector store | ChromaDB ≥ 0.5 (in memory, one index per log) |
| Extraction layer | pandas ≥ 2.0 + pyarrow ≥ 14.0 |
| Drone log parsing | pymavlink ≥ 2.4.48 |
| Plotting | matplotlib ≥ 3.8 (in generated script) |
| Sandbox | Subprocess + PEP 578 audit hook, stripped env, resource limits, timeout |
| Config | python-dotenv |
| Lint / CI | ruff + pytest + GitHub Actions |

---

## Prerequisites

- **Python 3.10+**
- For the chat assistant only: an **OpenRouter API key** — [get one here](https://openrouter.ai/keys) (free tier available)

---

## Installation

MAVPose has two layers. Install only what you need:

| You want… | Install | Footprint |
|---|---|---|
| Clean, unit-correct telemetry in pandas / Parquet (no AI) — ArduPilot and PX4 logs | `pip install mavpose` | 10 packages |
| …plus the plain-English plot assistant (`mavpose` command) | `pip install "mavpose[chat]"` | ~114 packages |
| …or an MCP server so AI agents can read your logs (`mavpose-mcp`) | `pip install "mavpose[mcp]"` | 38 packages |

The core never imports the chat layer, so `pip install mavpose` has no LLM, vector-store or plotting dependencies.

To work on MAVPose itself:

```bash
git clone https://github.com/AyushMaria/MAVPose.git
cd MAVPose
python -m venv venv
source venv/bin/activate      # Windows: venv\Scripts\activate
pip install -e ".[dev]"       # core + chat + test tools
cp template.env .env          # chat assistant only
```

Open `.env` and fill in your key. If you installed from PyPI, there's no `template.env`: just create a file named `.env` in the folder you'll run `mavpose` from (MAVPose looks there and in parent folders), or set the variables in your shell.

```env
OPENROUTER_API_KEY=your-key-here
OPENROUTER_MODEL=z-ai/glm-5.1   # default — change to any OpenRouter model
```

> ⚠️ Never commit `.env` — it is already listed in `.gitignore`.

---

## Usage

The `mavpose` command is the chat assistant and needs `pip install "mavpose[chat]"`; without it, the command explains what to install.

### Single prompt

```bash
mavpose flight.tlog --prompt "Plot altitude over time"
```

### Interactive REPL

Omit `--prompt` to enter a live loop:

```bash
mavpose flight.tlog

📊 Plot request > Plot battery voltage and current
📊 Plot request > Show GPS latitude and longitude
📊 Plot request > quit
```

### All CLI flags

```
usage: mavpose [-h] [--prompt PROMPT] [--retries N] [--verbose] log_file

positional arguments:
  log_file              Path to a flight log: MAVLink .tlog, ArduPilot .bin/.log, or PX4 .ulg

options:
  -p, --prompt TEXT     Plot request. Omit for interactive mode.
  -r, --retries N       Max self-healing retries on script failure (default: 3)
  -v, --verbose         Enable debug logging
```

The output plot is saved as `plot.png`, and intermediate telemetry as `telemetry.parquet`, both in the same directory as your log file. Each run replaces the previous files. `mavpose` only reports success when the script saved a new plot; with `--prompt` it exits with status 1 otherwise.

> **Need a sample log file?**  
> [Download here](https://drive.google.com/file/d/1BKv-NbSvYQz9XqqmyOyOhe3o4PAFDyZa/view?usp=sharing)

---

## Output Files

| File | Description |
|---|---|
| `telemetry.parquet` | Extracted, time-aligned telemetry for the relevant message types |
| `plot.py` | The LLM-generated pandas + matplotlib script |
| `plot.png` | The final plot at 400 dpi |

---

## Switching Models

MAVPose routes through [OpenRouter](https://openrouter.ai/models), so any model on the platform works. Just update `.env`:

```env
# Z.ai options
OPENROUTER_MODEL=z-ai/glm-5.1          # default — long-horizon agentic coding
OPENROUTER_MODEL=z-ai/glm-5            # flagship, complex systems
OPENROUTER_MODEL=z-ai/glm-5-turbo      # fast inference
OPENROUTER_MODEL=z-ai/glm-4.5          # MoE, 355B params, switchable reasoning

# OpenAI fallback
OPENROUTER_MODEL=openai/gpt-4o
```

---

## Project Structure

```
MAVPose/
├── mavpose/                       # pip install mavpose — core data layer
│   ├── flightlog.py               # mavpose.load() and FlightLog: the public API
│   ├── mcp_server.py              # mavpose-mcp: MCP tools for AI agents ([mcp] extra)
│   ├── log_extractor.py           # LogExtractor: one clock, real units, unknown markers
│   ├── ulog_reader.py             # PX4 ULog (.ulg) support via pyulog
│   ├── file_validator.py          # File validation (extension, size, symlink)
│   ├── data/
│   │   ├── mavlink_unknown_markers.json   # generated from the MAVLink spec
│   │   └── px4_fields.json                # units + markers from PX4 msg definitions
│   └── chat/                      # pip install "mavpose[chat]" — plot assistant
│       ├── plot_creator.py        # PlotCreator: LLM writes the plot script
│       ├── safe_executor.py       # Restricted subprocess executor (audit hook + limits)
│       └── cli.py                 # `mavpose` command
├── docs/                          # guide, format notes, API reference (MkDocs)
├── CHANGELOG.md
├── tools/
│   ├── gen_unknown_markers.py     # regenerates MAVLink markers from mavlink/mavlink
│   └── gen_px4_field_info.py      # regenerates PX4 units/markers from PX4-Autopilot/msg
├── tests/
│   ├── log_builders.py            # builds real .tlog / .bin files for tests
│   ├── test_real_logs.py          # end-to-end extraction tests
│   ├── test_core_boundary.py      # core must not depend on chat
│   └── …
├── .github/workflows/ci.yml       # ruff + pytest on 3.10–3.13, core-only job, strict docs build
├── template.env
└── LICENSE
```

---

## Library API

```python
import mavpose

log = mavpose.load("flight.tlog")          # .tlog, .bin, .log or .ulg
log
# <FlightLog 'flight.tlog': mavlink, 38 message types, 2443.8 s>

alt = log.series("GLOBAL_POSITION_INT", "relative_alt")   # Series indexed by time_s
alt.attrs["unit"]                                         # 'm'

log.unknown_counts        # {'GPS_RAW_INT': {'epv': 6338}}  "not provided" values now NaN
log.to_parquet("telemetry.parquet")
```

`mavpose.load`, `FlightLog`, `LogExtractor`, `validate_mavlink_file` and
`FileValidationError` are the public API, covered by the
[stability policy](docs/api-stability.md). Full documentation:

- [Concepts](docs/guide/concepts.md): time axis, units, unknown values, damaged logs
- [Log formats](docs/guide/formats.md): message names and quirks per format
- [API reference](docs/api/flightlog.md) and [changelog](CHANGELOG.md)

To preview the docs locally: `pip install -e ".[docs]" && mkdocs serve`.

## MCP server for AI agents

```bash
pip install "mavpose[mcp]"
mavpose-mcp --root ~/flight-logs
```

Any MCP-capable assistant or agent framework can then call `list_logs`,
`summarize_flight`, `list_messages`, `describe_message`, `get_series`,
`get_log_messages` and `export_parquet`. The server is limited to the
`--root` folders, read-only except export, and never runs code. Long series
are downsampled for the agent's context, but min/max/mean always cover every
sample. See [docs/guide/mcp.md](docs/guide/mcp.md) for client setup.

## Plot assistant API — `PlotCreator`

```python
from mavpose.chat import PlotCreator   # needs mavpose[chat]

creator = PlotCreator(max_retries=3)
creator.set_logfile_name("flight.tlog")

# Phase 1a: schema scan + embeddings
creator.parse_mavlink_log()

# Phase 1b: semantic search + Parquet extraction
msg_types = creator.find_relevant_data_types("Plot altitude over time")
schema_summary = creator.extract_dataframes(msg_types)

# Phase 2: LLM writes + executes the script
creator.create_plot("Plot altitude over time", schema_summary)
result, code = creator.run_script()
# → plot.png and telemetry.parquet saved next to the log file
```

---

## Running Tests

```bash
pip install -e ".[dev]"
pytest tests/ -v --cov=mavpose
```

To lint:
```bash
ruff check mavpose tests tools
```

In a core-only install (`pip install -e .`), the chat tests skip themselves.

---

## Troubleshooting

| Issue | Cause | Fix |
|---|---|---|
| `OPENROUTER_API_KEY is not set` | No key in the shell or in a `.env` file | Create `.env` in the folder you run `mavpose` from with `OPENROUTER_API_KEY=your-key`, or export the variable |
| `ModuleNotFoundError: pandas` | Missing dependency | Run `pip install mavpose` |
| `ModuleNotFoundError: pyarrow` | Missing dependency | Run `pip install mavpose` |
| "The MAVPose chat assistant needs extra packages" | Core-only install | Run `pip install "mavpose[chat]"` |
| Plot not generated after N retries | LLM script failed repeatedly | Try a more specific prompt; use `--verbose` to inspect errors |
| `FileValidationError` | Wrong file type or empty file | Only `.tlog`, `.bin`, `.log` files ≤ 200 MB are accepted |
| `ValueError: None of the requested message types were found` | Semantic search returned types not in log | Use `--verbose` to see what types the log actually contains |
| ChromaDB version conflict | Stale venv | Delete `venv/` and reinstall with `pip install "mavpose[chat]"` |

---

## License

MIT — see [LICENSE](LICENSE) for details.
