Metadata-Version: 2.4
Name: ingero-annotate
Version: 0.1.1
Summary: Annotate live Ingero GPU traces with your application's semantics (training step, epoch, inference request_id, model).
Project-URL: Homepage, https://github.com/ingero-io/ingero
Project-URL: Repository, https://github.com/ingero-io/ingero
Project-URL: Documentation, https://github.com/ingero-io/ingero/blob/main/docs/commands.md
Project-URL: Issues, https://github.com/ingero-io/ingero/issues
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: annotation,ebpf,gpu,ingero,observability
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.8
Description-Content-Type: text/markdown

# ingero-annotate

Add semantic labels to GPU traces captured by [Ingero](https://github.com/ingero-io/ingero).

## What is Ingero?

Ingero is an open source eBPF agent that traces what your GPU is doing
in production. It attaches to `libcudart`, `libcuda`, and Linux kernel
tracepoints, then answers questions like "which CUDA kernels stalled
and why," "which process held VRAM," and "which rank in a multi-GPU
run was the slow one." No source-code changes, no rebuild. You run
`ingero trace` next to your training or inference job.

What `ingero trace` cannot know on its own is *your* application's
semantics. It sees kernel launches, memcpy timings, scheduler events.
It does not see "this is epoch 3, step 42" or "this is request
abc-123 for the llama-3-8b model."

That is what `ingero-annotate` is for.

## What does annotate do?

It is a small Python client that pushes semantic labels into the live
trace over a Unix-domain socket. The agent attaches those labels to
the trace timeline, so when you later run `ingero explain` or open the
trace in Perfetto, you see kernel events keyed to your concepts.

**Training example.** Emit a `step` and `phase` label at each
iteration boundary. A question like "which optimizer step had the 4x
backward-pass spike" becomes a one-line query against the trace,
instead of squinting at unlabeled kernel timings.

**Inference example.** Emit `request_id` and `model` per request. You
can then ask "which incoming request caused the GPU saturation at
14:23:07" and get an answer that names the request, instead of just
"something heavy happened."

## Install

```
pip install ingero-annotate
```

## Quick start

Run the agent with the annotation socket bound:

```
sudo ingero trace --record --annotate
```

Then from your Python code:

```python
import os
from ingero_annotate import AnnotationWriter

writer = AnnotationWriter()                       # connects, or no-ops silently
writer.write({"step": "42"}, pid=os.getpid())     # one instant annotation
writer.write({"phase": "backward"}, pid=os.getpid())
writer.close()
```

`AnnotationWriter` is also a context manager for connection lifecycle:

```python
with AnnotationWriter() as w:
    for step in range(1000):
        w.write({"step": str(step)}, pid=os.getpid())
        train_one_step()
```

If the agent is not running, or the socket is missing, the writer
drops silently. Your training or inference code keeps working
unchanged.

## Don't want to wire it by hand?

Drop-in adapters for common frameworks live in the agent repo, under
[`examples/integrations/`](https://github.com/ingero-io/ingero/tree/main/examples/integrations):

- PyTorch Lightning callback
- HuggingFace Trainer callback
- DeepSpeed wrapper
- Accelerate hook
- Ray task hook
- vLLM per-request emitter (inference, emits time-bounded spans)

Each one pulls `step` / `epoch` / `request_id` out of its framework
and emits annotations for you.

## Wire protocol

Each annotation is one line of NDJSON written to the socket:

```json
{"labels": {"step": "42"}, "pid": 1234, "ts": 1700000000000000000}
```

`labels` is required. `pid` scopes the annotation to one process;
without it the label applies trace-wide. `ts` is optional unix
nanoseconds (absent means the agent stamps receive time). The
contract also supports time-bounded **span** annotations (`span_start`
/ `span_end`), used today by the vLLM per-request emitter; this
library exposes instant annotations and lets specialized adapters
write spans directly.

Label keys and values are validated against the agent contract
(charsets, length limits) before being sent.

## Per-request inference correlation

For per-request inference workloads, `ingero explain --by-request` and
`ingero query --by-request` group analyzed events by `request_id`. The
output is a **time-overlap** slice, not exclusive kernel ownership:
under continuous batching a single kernel can belong to many in-flight
requests. The agent prints that caveat with every per-request report.
Full semantics, trust-domain scope, and example output:
[`docs/commands.md`](https://github.com/ingero-io/ingero/blob/main/docs/commands.md).

## Links

- Agent: https://github.com/ingero-io/ingero
- Docs: https://github.com/ingero-io/ingero/blob/main/docs/commands.md
- Issues: https://github.com/ingero-io/ingero/issues

## License

Apache-2.0.
