Message boundaries come from parser state, not socket reads
Design incremental framing with explicit size limits, completeness rules, and consistent interpretation across intermediaries.
On this page
The question for today
What you’ll understand
- Trace a length-prefixed parser across arbitrary read boundaries.
- Separate incomplete input from invalid framing and resource exhaustion.
- Explain why intermediaries must agree on request boundaries.
A network read can contain half a header, several complete messages, or the end of one message followed by the beginning of another. The application therefore needs a grammar and remembered parsing state. Treat the parser as a machine that consumes bytes only when its current rule permits it. This approach explains both ordinary partial reads and dangerous cases in which two network participants disagree about where one request ends.
Write a framing contract before a receive loop
Consider a fictional protocol with a four-byte unsigned big-endian payload length followed by that many payload bytes. Its parser begins in a prefix state, retains partial prefix bytes across reads, checks the decoded length against a configured maximum, then enters a body state. Once the body is complete it emits exactly one message and returns to the prefix state. Any bytes remaining in the same read belong to subsequent parsing, not automatically to the emitted message. A zero-length body can be valid if the protocol explicitly permits it; an implementation should not invent a different rule silently.
A delimiter-based protocol needs equally precise rules for escaping, character encoding, and delimiters inside payloads. Closing the connection can delimit certain responses, but that makes reuse and distinguishing truncation more difficult. Different framing choices move complexity rather than eliminate it. Record the grammar, integer width, byte order, permitted lengths, and end-of-input behavior. The parser should be correct for every fragmentation of the same valid byte sequence: splitting a prefix into four reads must not change the messages produced compared with receiving the entire sequence in one read.
Incomplete input and invalid input require different decisions
If only two prefix bytes have arrived, waiting for the remaining two is normal incremental parsing. If a complete prefix announces a body larger than the permitted maximum, the input violates the contract immediately; waiting for the entire announced body wastes resources. Validate lengths before allocating, account for aggregate buffered bytes across connections, and avoid arithmetic overflow when checking header-plus-body bounds. A compressed or nested payload can consume far more memory after decoding than its framed length, so decode limits and nesting limits belong beside wire-size limits rather than being assumed to follow from them.
End-of-input in the middle of a declared body is truncation, not successful completion of a shorter message. A peer that sends prefixes very slowly can also retain parser and connection resources without exceeding a size limit. Use appropriate progress and lifetime policies, while distinguishing a documented timeout from malformed syntax. Keep unconsumed bytes and parser state together so cancellation cannot accidentally reuse a buffer as the next message. Error responses should not leave an intermediary and backend interpreting the remaining connection differently; rejecting one request may require terminating that ambiguous connection safely.
Framing agreement is a boundary between participants
HTTP/1.1 defines precedence rules for body length, including special response cases and transfer coding. A message containing both Transfer-Encoding and Content-Length can create request-smuggling risk if participants choose different boundaries. The specification defines handling and forwarding requirements; a gateway can adopt a stricter documented rejection policy. Do not repair ambiguous input by guessing which parser the next hop probably uses. HTTP/2 and HTTP/3 have different frame grammars, yet translating versions still requires preserving message semantics and rejecting invalid constructions consistently. Test the whole intermediary chain, not only a standalone parser fed neatly segmented examples.
The probability lab examines a smaller completeness assumption rather than implementing a parser. Reset, explicitly set BOTH delivery probability p=0.9 and trial count n=4, and predict that all four hypothetical independent pieces arrive with probability 0.9⁴ before Play or Step. Reset and re-set BOTH p=0.9 and n=10 for comparison. Playback highlights an exact distribution; it does not sample packets, split headers, or determine valid messages. A count of received pieces cannot establish byte order or framing. Use the worked state trace, not the bars, to decide when this protocol can emit a message.
Work it through
Two partial bodies and one following frame
The fictional four-byte-prefix protocol receives a first read containing a complete prefix L=8 and three body bytes. A second read contains the remaining five body bytes plus a complete prefix L=2 and its two body bytes.
After the first read, seven wire bytes have arrived: four prefix bytes and three of the required eight body bytes. Retain body state with five bytes still needed; emit nothing.
The second read's first five bytes complete the eight-byte body. Emit the first message and return to prefix state rather than discarding the rest of that read.
Six bytes remain in the second read: a four-byte prefix and two body bytes. Emit the second message too. Total consumed wire size is (4+8)+(4+2)=18 bytes for two messages.
Two socket reads produced two frames with different boundaries. The parser's state, not read count, determined when each message became complete.
The living notebook
Explore the model
Interactive lab · probability
Completeness needs all required pieces
Reset, set BOTH p=0.9 and n=4, predict the n-success probability, then Play or Step. Reset and re-set BOTH p=0.9 and n=10 for a fresh comparison. This is exact independent binomial counting, not random delivery, byte ordering, a parser, or framing validation.
- Mean · np
- 5
- Variance · np(1−p)
- 2.5
- Total probability
- 1
These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forReceiving a collection of pieces and parsing a complete valid message are different claims.
Pause & reason
Check your understanding
Your turn
Try it yourself
A complete read contains 25 bytes of a four-byte-prefix frame announcing L=30. Assume no earlier bytes and a maximum payload of 64. How many more body bytes are required, and what changes if the connection closes now?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Subtract prefix bytes before counting body progress; distinguish a valid size from a complete body.
Compare with the solution
Twenty-one body bytes have arrived, so nine more are required. The announced size is within the maximum, but the frame is still incomplete. Closing now truncates the frame; it does not convert the 21 received body bytes into a successful shorter message.
Keep the idea
Take it with you
- Incremental parsing must be invariant to arbitrary socket-read boundaries.
- Length, aggregate-memory, decoding, and progress limits protect different resources.
- Intermediaries need consistent framing interpretation to avoid ambiguous request boundaries.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.