Remote streaming — callsight

When to stream

Remote or embedded targets — the interesting workload runs on a device you reach over the network, not on your workstation.
No disk, or no spare disk — nothing is written to the device's filesystem; events live in RAM until they leave over TCP.
Long-running workloads — file mode grows unboundedly. Streaming bounds device-side memory to one fixed-size ring, however long the program runs.

How it works

Traced process. With TRACE_SHM=/name set, the hooks flush per-thread event batches into a POSIX shared-memory ring instead of files. A flush is a locked memcpy of a buffered batch — short critical sections, serialized by a spinlock, never held across I/O, because there is no I/O.
On-device client. trace_stream maps the same ring, drains it in batches, ZSTD-compresses each batch, and sends it over a raw TCP connection. It exits once the traced process detaches and the ring is drained.
Analysis host. callsight serve decompresses each chunk and appends it to a standard trace.stream.<id>.bin file — one per connection. Events from all threads arrive interleaved in one stream and are demultiplexed by the analyzer, since every event carries its tid.

The never-stall guarantee

Profiling must never slow down the thing it measures.

No disk or network I/O in the traced process — only shared-memory copies.
Ring full → drop and count. If the ring fills faster than the client drains it (slow network, small ring), the tracer drops events and increments a dropped counter in the ring header. The client forwards the count as a notice chunk and the server prints it, so data loss is always visible, never silent.
Size the ring to taste with TRACE_SHM_SIZE (bytes, default 16 MiB).

Verified end to end. 500,000 events across 26 threads over localhost with unmatched_exits=0, and drop counting confirmed under deliberate ring overflow — one of the repository's CI smoke jobs.

Quick start

# analysis host (needs the stream extra):
$ uv tool install 'callsight[stream]'
$ callsight serve                    # 0.0.0.0:9001, writes traces/
$ callsight serve --max-mb 8192      # raise the per-connection budget

# adopt with streaming support — copies trace_stream.c + the vendored zstd:
$ callsight init --stream /path/to/project

# on the device:
$ cc -O2 -o callsight/trace_stream callsight/trace_stream.c callsight/zstd.c
$ ./callsight/trace_stream /callsight0 <server-ip> 9001 &
$ TRACE_ENABLE=1 TRACE_SHM=/callsight0 ./yourapp.instr

Then analyze on the host exactly as with local traces:

$ callsight analyze traces/ --exe ./yourapp.instr --top 20

Keep the binary. Symbols are resolved on the host from the same instrumented binary you shipped to the device — --exe must point at it, and addr2line must match its architecture. For a cross-compiled target that means the toolchain's own copy: --addr2line aarch64-linux-gnu-addr2line.

The host has limits too. serve rotates and caps its output per connection (4 GB by default). A device streaming for an hour must not fill the machine you are analyzing on either — the same reasoning as the on-device budget.

Ring layout and wire protocol

Both live in src/callsight/runtime/trace_shm.h (TRACE_SHM_VERSION / TRACE_STREAM_VERSION are bumped on any layout change). All integers little-endian.

Ring header

trace_shm_header_t, followed by capacity bytes of event storage. The header is 96 bytes.

Field	Type	Meaning
`magic`	`char[8]`	`TKSHM`
`version`	`u32`	`TRACE_SHM_VERSION` (2)
`capacity`	`u32`	Ring bytes after the header
`writers`	`u32`	Tracer processes attached
`lock`	`u32`	Spinlock: 0 = free
`head`	`u64`	Monotonic write offset (bytes)
`tail`	`u64`	Monotonic read offset (bytes)
`dropped`	`u64`	Events dropped because the ring was full
`flags`	`u32`	Whether timestamps are raw ticks; written by the tracer
`load_bias`	`u64`	PIE relocation offset of the traced binary
`tick_hz`, `t0_ticks`, `t0_ns`	`u64`	Clock calibration
`hook_ns`	`u64`	Measured per-hook cost

head and tail are monotonic byte counters; bytes in use is head - tail.

The lower half of the header is written once by the traced process, read by the drain client, and forwarded to the server. Without it the server would have to guess: raw cycle counts and nanoseconds are indistinguishable once they reach the wire, and guessing wrong produces a trace that looks perfectly normal and is wrong by the clock ratio. The client therefore waits for a tracer to attach before sending its handshake — it starts first, and the values it needs do not exist yet.

The spinlock is bounded and yields. A tracer killed inside its critical section leaves the lock held forever, and on a single-core device an unbounded spin would prevent the holder from ever being scheduled to release it. A participant that gives up drops its batch, because the ring's whole contract is that profiling never stalls the workload.

Wire protocol

Client → server over TCP: a 64-byte handshake — TKSTREAM magic, u32 version, u32 event_size, then the same clock calibration and load bias the ring carries — followed by a sequence of chunks. Each chunk is u32 type, u32 raw_len, u32 zstd_len followed by zstd_len bytes of ZSTD-compressed payload.

Type	Payload	Server action
`0` — events	Decompresses to `raw_len` bytes of raw events	Appended to the output file as-is
`1` — notice	Decompresses to a `u64` dropped-event count	Printed, so loss is visible

The on-device client

trace_stream.c is self-contained C built against a vendored single-file zstd v1.5.7 (generated from the official repository's build/single_file_libs; BSD licensed — see zstd.LICENSE). One cc line, and it cross-compiles like any other C file:

$ aarch64-linux-gnu-gcc -O2 -o trace_stream trace_stream.c zstd.c

Architecture →The runtime, the event record, and how the hot path stays lock-free. Analysis →Streamed traces analyze exactly like local ones.