Telemetry boundaries and realistic capacity tests

Determine whether a counter covers a physical port, virtual interface, queue, priority, function, or entire adapter. Hardware and software statistics may count at different points, and offloads can change the relationship between packets observed by software and frames on the link. A counter named bytes may include headers, exclude some framing, or count only successful traffic. Use the driver or vendor definition instead of assuming every similarly named metric can be added or directly compared.

Compute differences only across a compatible measurement epoch. Device reset, process restart, counter wrap, or collector changes can break a simple subtraction. A negative delta is not negative traffic. Align clocks and intervals when correlating endpoints and switches, but remember that imperfect synchronization can shift a short event between samples. Preserve the raw totals, collection timestamps, definitions, and topology labels so a derived percentage can be audited after an incident.

A one-second rate cannot reveal every microburst within that second. Queue watermarks, device-supported latency histograms, and carefully chosen higher-resolution signals can expose short congestion intervals, each with its own scope and sampling limits. A watermark gives a maximum since a defined reset, not the duration of congestion. Mark, drop, pause, and retry counters explain different mechanisms; a low value in one does not establish spare capacity in another resource.

Collective tests add another normalization layer. Algorithm bandwidth divides a defined input size by operation duration; normalized bus bandwidth applies an operation-specific factor. Neither is automatically a physical port's exact wire rate. Compare them only after mapping ranks, local links, NIC sharing, and actual algorithm behavior. A standalone point-to-point transfer can validate one path, but a realistic training step also contains synchronized communication, compute overlap, storage activity, and straggler effects.

Build a workload matrix covering message sizes, fan-in, all-reduce, all-gather, and skewed all-to-all where relevant. Record correctness, mean and tail completion, rank spread, and per-boundary telemetry. Increase load gradually to identify where useful throughput flattens or tail latency changes sharply. Repeat under a removed or degraded path and representative background work. Keep configuration and placement reproducible so a regression can be separated from a different hash assignment or traffic distribution.

The Roofline lab provides a deliberately separate accounting check. Reset and set both I=4 FLOP/byte, B=80 GB/s, predict 320 GFLOP/s and ridge 12.5 FLOP/byte, then Step. Reset and re-set I=16, B=80, predicting the fixed 1,000 GFLOP/s ceiling. This is a hypothetical memory-boundary model, not fabric telemetry: playback traces a roof without measuring traffic. State exactly which measured bytes and work would justify a comparable bound before using any reported bandwidth as a substitute.