Logs are versioned event records with a data boundary
Design typed event fields, timestamps, identity, and schema evolution while controlling sensitive data before it spreads through the telemetry pipeline.
On this page
The question for today
What you’ll understand
- Separate an event's stable identity/schema from its human-readable message.
- Distinguish event time, observation time, and collection delay assumptions.
- Version fields and constrain sensitive payloads before indexing or exporting logs.
A useful log record can be read by a person and queried by software without guessing meaning from a sentence. That requires a stable event name, typed attributes, explicit units, and a clear boundary around data allowed to leave the application. Structured syntax alone does not supply these properties: a JSON object full of inconsistent strings can still be difficult to interpret. Treat operational events as an evolving interface with producers, consumers, retention, and failure behavior.
Make fields mean one thing across records
Choose a stable event class such as checkpoint.publish_failed and keep variable details in typed fields rather than generating a different event name for every path. A duration field should have one documented unit, an outcome should use a bounded vocabulary where appropriate, and identifiers should have defined scope. Preserve a human-readable body for context without forcing every query to parse its punctuation. Resource identity and instrumentation scope help distinguish which component emitted the event from the operation or tenant the event describes.
OpenTelemetry's log model separates fields including body, severity, attributes, event name, and optional trace/span context. A severity label is a classification, not an error-rate denominator or a guarantee that an event was delivered once. A retry can produce several similar records for one user operation. If deduplication is required, design an event identity and retention scope that distinguish retransmission from a genuinely repeated occurrence. Do not use a timestamp or a whole rendered message as a reliable universal uniqueness key.
Preserve time and schema meaning during evolution
Event time describes when the source says the event happened; observation time describes when a collector observed it. OpenTelemetry carries these separately because buffered or externally collected events can arrive later. Their difference can include clock offset and conversion errors as well as transport delay, so it is not automatically a precise one-way network latency. Use appropriate timing sources for local elapsed durations, retain timezone/epoch conventions for absolute timestamps, and test units explicitly. A millisecond/nanosecond mix-up can look like an extraordinary incident while only being a schema defect.
Version a schema when changing a field's meaning, type, or unit would break existing consumers. Prefer additive compatible changes when possible, document missing versus null versus empty values, and test old/new producers with queries and parsers. An event's schema version should not become an uncontrolled metrics label merely because it is useful in logs. Keep migration fixtures for historical records and handle malformed events observably rather than dropping them silently. A successful parse is weaker than correctly interpreting the intended event; consumers need both syntactic and semantic compatibility.
Apply the data boundary before fan-out
Define which fields may be collected, transformed, retained, and accessed. Raw authorization headers, credentials, unrestricted request bodies, or sensitive model inputs should not flow into an event merely because debugging would be convenient. Prefer explicit allowed fields and early redaction or omission at the intended boundary, then test nested attributes and exception paths. Hashing an identifier does not automatically make it anonymous or reduce its cardinality. Once a sensitive value reaches several exporters, indexes, and backups, removing only the source record does not undo that propagation.
The queue lab illustrates buffering of equal-sized log batches, not a schema validator. Reset, set both arrival λ=4 and service μ=2, predict backlog growth from four batches, then Play or Step. Reset and re-enter both λ=1 and μ=3 for a separate drainage case. Slider edits preserve elapsed time and recalculate from the initial four rather than continuing an accumulated queue at new rates. The instrument cannot parse JSON, redact data, enforce access, or guarantee delivery. It simply shows why buffered logs can be observed later than their source event.
Work it through
Budget event volume and interpret apparent lag
A service emits 2,000 records/s averaging 600 serialized bytes for one hour. One record's source event time is 12:00:00.000 and collector observation time is 12:00:00.180. Assume the collector clock is 30 ms ahead of the source.
Raw event bandwidth is 2,000 × 600 = 1,200,000 bytes/s. Over 3,600 seconds, emitted payload is 4,320,000,000 bytes, or 4.32 decimal GB.
The apparent timestamp difference is 180 ms. Under the stated constant clock-offset model, real collection delay is 180 − 30 = 150 ms.
If the clock offset were unknown, the timestamps alone would not establish that 150 ms delay. Separately budget compression, indexes, replication, and schema-dependent size variation rather than calling 4.32 GB the final storage bill.
The volume estimate and lag calculation are valid only with their declared byte and clock assumptions; typed fields should preserve those assumptions for consumers.
The living notebook
Explore the model
Interactive lab · queue
Buffered records can arrive later
Reset, set both λ=4 and μ=2 for equal-sized log batches, predict growth from four, then Play or Step. Reset and re-enter λ=1 and μ=3 to compare fresh drainage. Edits retain elapsed time and recompute from initial four. This fluid queue does not parse schemas, synchronize clocks, redact fields, or guarantee log delivery.
- Fluid backlog
- 4
- Net rate · λ − μ
- −1 /s
- M/M/1 mean residence
- 1 s
Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forEvent time, observation time, and buffer pressure describe different parts of an event's path.
Pause & reason
Check your understanding
Your turn
Try it yourself
A source timestamp is 10:00:00.000 and its collector observation is 10:00:00.100. The collector clock is known to be twenty milliseconds behind. Estimate collection delay and explain what changes if that offset is unknown.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Use apparent lag equals real delay plus collector-minus-source clock offset.
Compare with the solution
The apparent lag is 100 ms and the relative offset is −20 ms, so real collection delay is 100 − (−20) = 120 ms in this simple model. Without a known clock relationship, the timestamps alone do not isolate real delay; preserve that uncertainty rather than reporting a precise transport latency.
Keep the idea
Take it with you
- Stable event classes and typed, unit-bearing fields make logs an interpretable interface.
- Schema versions and separate event/observation timestamps preserve meaning across collection and evolution.
- Apply an explicit sensitive-data boundary before records fan out into retention and indexing systems.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.