Internals

Architecture

What happens between "you edited trace.config" and "here is your hotspot table" — and why each piece is built the way it is.

The four stages

1 · SELECT trace.config → flags.py patterns, call graph, exclude lists callsight flags 2 · COMPILE GCC emits hook calls __cyg_profile_func_enter excluded code: nothing emitted 3 · RECORD trace.c runtime TLS buffer · 8192 events no locks · no malloc · no I/O flush on full / at exit trace.<pid>.<tid>.bin 4 · ANALYZE streaming matcher → addr2line → report events matched as they are read; memory stays flat text tables · JSON · folded stacks web UI · flame graph

1 · Selection becomes compiler flags

callsight flags reads trace.config and the source list your build system passes it, and prints the flag string the compiler should use. Both build integrations call it on every build, so the selection is always current.

The substring-collision guard. Because the compiler's function exclude list is a substring match, an auto-generated exclusion that happens to be a substring of a selected function would silently disable the function you asked for. callsight detects that case, drops the offending entry, and warns instead.

The static call graph

include-func needs to know what a function calls. callsight uses a deliberately lightweight heuristic parser (comments and string literals blanked, definitions matched at line starts, call sites matched inside the body) rather than pulling in libclang — adoption stays a single uv tool install with no toolchain dependency.

The trade-off is documented rather than hidden: function pointers, macro-generated calls and C++ dynamic dispatch are not followed. The expansion is breadth-first, so a depth limit always measures the shortest path to each function.

2 · The hook runtime

trace.c is self-contained C with no dependencies beyond pthreads, compiled without -finstrument-functions and with no_instrument_function on every function, so the hooks can never trigger themselves.

The event record

32 bytes, fixed layout, little-endian — the same on disk and on the wire.

FieldTypeMeaning
ts_nsu64CLOCK_MONOTONIC_RAW nanoseconds
func_addru64Address of the entered/exited function
caller_addru64Return address in the caller
tidu32Kernel thread id
kindu80 = enter, 1 = exit
_padu8[3]Padding to 32 bytes

Each trace file starts with a 16-byte header: an MLTRACE magic, a format version, and the event size — so a mismatched or truncated file is detected and skipped rather than misread.

3 · The analyzer

Analysis is a single streaming pass. Events are read in blocks and matched as they arrive, so analyzer memory tracks the number of functions and threads, not the number of events — a multi-million-event trace costs a few MB.

  1. Match per thread. Each thread has its own stack. An exit pops back to the nearest matching enter, closing any frames left dangling above it (which happens after a longjmp, or when a buffer tail is lost).
  2. Attribute time. Inclusive time is exit minus enter; the duration is also credited to the parent frame's "children" total, and self time is inclusive minus children.
  3. Resolve symbols. All distinct addresses go to addr2line in batches, once, at the end — static functions included.
  4. Emit. Text tables, JSON, or folded stacks. Folded output is accumulated during the same pass, keyed by the tuple of addresses on the live stack.

Because everything is derived from the same pass, the flame graph and the self_ms column are guaranteed to agree.

Compiler mechanisms surveyed

What each compile-time instrumentation mechanism can do, what it costs, and how it maps to callsight. Overhead figures are order-of-magnitude, per event, assuming a lean hook.

MechanismCompilersGranularityRuntime toggleOverhead/eventVerdict
-finstrument-functionsGCC, ClangFunction entry/exitNo (compile-time)~30–60 nsDefault backend
-finstrument-functions-after-inliningClangPost-inline entry/exitNo~30–60 nsClang enhancement
-pg (mcount/gprof)GCC, ClangFunction entryNo~50–100 nsLegacy, rejected
-fpatchable-function-entryGCC ≥ 8, Clang ≥ 11NOP sleds at entryYes (patch sleds)~0 off, few ns onRoadmap candidate
-fsanitize-coverageClang, GCC ≥ 12Edge / PC guardNo~5–20 nsAlternative backend, evaluating
XRay (-fxray-instrument)ClangEntry/exit sledsYes (official API)~0 when offDesign reference
-fprofile-arcs (gcov)GCC, ClangEdge countersNoCounter incOut of scope (no timing)
GCC plugin APIGCC (version-locked)Arbitrary (GIMPLE)PossibleVariesNon-goal

-finstrument-functions — the current backend

Emits a call to __cyg_profile_func_enter(this_fn, call_site) and __cyg_profile_func_exit(...) at every function boundary, static functions included. Selection happens at compile time through the two exclude lists, both substring-matched, with the file list matched against the file a function is defined in. Excluded code emits no hook at all, so selection is free at runtime.

Caveats: inlined functions emit no hooks, because no call boundary exists; hooks fire even when you don't want data, so runtime gating costs a flag check per event; and -no-pie (or offset bookkeeping) is needed to map addresses back to symbols.

The exclude lists are GCC-only. Clang implements -finstrument-functions but has never taken the exclusion patches (LLVM #15627), and its driver rejects unknown arguments outright. So a selective config requires GCC; under Clang only an unfiltered "instrument everything" config compiles. callsight flags detects the toolchain (--compiler-cmd, passed by both build integrations) and reports this before the build rather than letting it fail once per translation unit. Giving Clang real file-level selection would mean applying -finstrument-functions per translation unit instead of globally — a build-integration change, not a flag change.

-finstrument-functions-after-inlining

Same hooks, inserted after inlining, so functions that survived inlining get hooked even when they were inlined at some call sites, and you see the real optimized call graph. A candidate extra flag on Clang toolchains; it needs an analyzer-side note that the call graph differs from the source-level one.

-pg / mcount (gprof)

The original: an entry-only hook into mcount, plus flat-profile sampling. Call-graph arcs only — no per-call exit timing — with PLT and shared-library blind spots and effectively unmaintained semantics under modern optimization. Documented for completeness; callsight does not use it.

-fpatchable-function-entry=N,M

Emits N NOPs at function entry (M of them before the prologue) plus a __patchable_function_entries section listing their addresses — the mechanism behind the Linux kernel's ftrace and uftrace's dynamic mode. A runtime can patch those sleds into hook calls and back, giving genuinely zero-cost-when-off, toggleable tracing. The strongest candidate for "trace a live production process for five seconds". Costs: code-size growth, patching machinery, and entry-only hooks — exit timing needs a return-address trampoline, which is the hard part.

-fsanitize-coverage

compiler-rt coverage callbacks (__sanitizer_cov_trace_pc_guard per edge, with a per-guard toggle word in the guard variant), aimed at fuzzers. Leaner than function hooks and gives basic-block-transition granularity, but there is no caller address and no exit event, so timing must be reconstructed, and guard tables need the compiler-rt runtime. Clang has the full menu; GCC ≥ 12 has trace-pc and trace-cmp only. Worth an experiment as a low-overhead "which edges ran" backend, not a replacement for call timing.

LLVM XRay

Clang-only, and the most production-grade design in this list: sleds at function entry/exit patched at runtime through a supported API (__xray_patch()), per-function selection at compile time, and a logging library writing binary flight-recorder traces with tooling (llvm-xray). The closest existing model for callsight's streaming design — its sled layout and FDR buffering are the reference. Not usable as a default, being Clang-only.

gcov / -fprofile-arcs

Edge counters for coverage, not timing. It answers "did this line run, how often", not "how long did it take". A possible complement if a coverage view is ever added; out of scope for tracing.

GCC plugin API

Loadable modules running custom GIMPLE/RTL passes: arbitrary injection, maximum control. But plugins are locked to the exact GCC version, the C++ API is fragile, Clang has no equivalent, and distribution is a nightmare. A deliberate non-goal.

Not compile-time (context only)

Repository layout

PathWhat lives there
src/callsight/flags.pyConfig parsing, pattern matching, exclude-list generation, compiler detection
src/callsight/callgraph.pyThe heuristic static call graph behind include-func
src/callsight/analyze.pyStreaming trace reader, enter/exit matcher, report formats
src/callsight/runtime/trace.c, trace.h, trace_shm.h — copied into adopted projects
src/callsight/stream/trace_stream.c plus vendored single-file zstd
src/callsight/share/, cmake/The Make fragment and the CMake module
src/callsight/ui/The optional FastAPI web UI
tests/matrixlab/Multi-threaded C11 demo workload; the end-to-end fixture