When to stream
- Remote or embedded targets — the interesting workload runs on a device you reach over the network, not on your workstation.
- No disk, or no spare disk — nothing is written to the device's filesystem; events live in RAM until they leave over TCP.
- Long-running workloads — file mode grows unboundedly. Streaming bounds device-side memory to one fixed-size ring, however long the program runs.
How it works
- Traced process. With
TRACE_SHM=/nameset, the hooks flush per-thread event batches into a POSIX shared-memory ring instead of files. A flush is a lockedmemcpyof a buffered batch — short critical sections, serialized by a spinlock, never held across I/O, because there is no I/O. - On-device client.
trace_streammaps the same ring, drains it in batches, ZSTD-compresses each batch, and sends it over a raw TCP connection. It exits once the traced process detaches and the ring is drained. - Analysis host.
callsight servedecompresses each chunk and appends it to a standardtrace.stream.<id>.binfile — one per connection. Events from all threads arrive interleaved in one stream and are demultiplexed by the analyzer, since every event carries its tid.
The never-stall guarantee
Profiling must never slow down the thing it measures.
- No disk or network I/O in the traced process — only shared-memory copies.
- Ring full → drop and count. If the ring fills faster than the client
drains it (slow network, small ring), the tracer drops events and increments a
droppedcounter in the ring header. The client forwards the count as a notice chunk and the server prints it, so data loss is always visible, never silent. - Size the ring to taste with
TRACE_SHM_SIZE(bytes, default 16 MiB).
unmatched_exits=0, and drop counting confirmed under deliberate ring
overflow — one of the repository's CI smoke jobs.
Quick start
# analysis host (needs the stream extra):
$ uv tool install 'callsight[stream]'
$ callsight serve # 0.0.0.0:9001, writes traces/# adopt with streaming support — copies trace_stream.c + the vendored zstd:
$ callsight init --stream /path/to/project# on the device:
$ cc -O2 -o callsight/trace_stream callsight/trace_stream.c callsight/zstd.c
$ ./callsight/trace_stream /callsight0 <server-ip> 9001 &
$ TRACE_ENABLE=1 TRACE_SHM=/callsight0 ./yourapp.instrThen analyze on the host exactly as with local traces:
$ callsight analyze traces/ --exe ./yourapp.instr --top 20--exe must point at it, and
addr2line must match its architecture.
Ring layout and wire protocol
Both live in src/callsight/runtime/trace_shm.h
(TRACE_SHM_VERSION / TRACE_STREAM_VERSION are bumped on any
layout change). All integers little-endian.
Ring header
trace_shm_header_t, followed by capacity bytes of event
storage. The header is 64 bytes.
| Field | Type | Meaning |
|---|---|---|
magic | char[8] | TKSHM |
version | u32 | TRACE_SHM_VERSION (1) |
capacity | u32 | Ring bytes after the header |
writers | u32 | Tracer processes attached |
lock | u32 | Spinlock: 0 = free |
head | u64 | Monotonic write offset (bytes) |
tail | u64 | Monotonic read offset (bytes) |
dropped | u64 | Events dropped because the ring was full |
_pad | u8[16] | Reserved |
head and tail are monotonic byte counters; bytes in use is
head - tail.
Wire protocol
Client → server over TCP: an 8-byte TKSTREAM magic, u32 version
and u32 event_size, then a sequence of chunks. Each chunk is
u32 type, u32 raw_len, u32 zstd_len followed by zstd_len bytes of
ZSTD-compressed payload.
| Type | Payload | Server action |
|---|---|---|
0 — events | Decompresses to raw_len bytes of raw events | Appended to the output file as-is |
1 — notice | Decompresses to a u64 dropped-event count | Printed, so loss is visible |
The on-device client
trace_stream.c is self-contained C built against a vendored single-file
zstd v1.5.7 (generated from the official repository's build/single_file_libs;
BSD licensed — see zstd.LICENSE). One cc line, and it
cross-compiles like any other C file:
$ aarch64-linux-gnu-gcc -O2 -o trace_stream trace_stream.c zstd.c