Multipart upload separates staging from publication
Track upload IDs, numbered parts, completion results, ambiguous responses, and cleanup races without mistaking staged bytes for a completed object.
On this page
The question for today
What you’ll understand
- Model initiation, part upload, completion, and abort as separate states.
- Explain why an initial completion HTTP 200 is insufficient by itself.
- Budget retries and verify cleanup of abandoned parts.
Large-object upload is easier to retry when independent parts can be transferred separately. The tradeoff is a protocol with durable intermediate state: an upload ID, part numbers, returned validators, and an explicit completion request. Staged parts consume storage without necessarily creating the current object readers expect. A correct client must distinguish acknowledged parts from successful publication and must clean up work it abandons. The following details describe Amazon S3's documented multipart API rather than every service that uses similar endpoint names.
One upload ID coordinates many part operations
Initiating an upload returns an upload ID associated with the destination bucket and key. Parts are uploaded under that ID with part numbers, and the client records the returned information required for completion. A later upload of the same part number replaces that part's staged content for the upload rather than appending a second logical part. Keep the intended part list and its validators in durable client state when recovery needs to survive a process restart.
Completion assembles the specified parts in the API's part-number ordering and creates the object after successful processing. The client must supply the correct complete list rather than assuming every previously attempted transfer is present. Part sizes and counts have API limits, including special handling of the final part, so choose chunking within the documented contract. Parallelism improves transfer utilization only up to the actual network, memory, and service limits; allocating an unbounded number of part buffers creates a new failure mode.
Completion has a body-level outcome
Amazon S3 documents that CompleteMultipartUpload can send an initial HTTP 200 header while processing continues, then return an error inside the response body. A raw client that treats the header as final success can publish a manifest referring to an object whose upload did not complete. Consume and parse the full response according to the API, or use a supported SDK whose error handling covers that case. Whitespace while waiting is not a completed object result.
A lost connection after sending completion creates an ambiguous observation: the client may not know whether publication happened. Reconcile using the API's documented state and object evidence, preserving the upload identity and intended object properties. Blindly starting a new unconditional upload can replace another writer's object or create an extra version. Conditional completion, version-aware checks, and application operation identifiers can constrain that recovery, but the exact error-specific retry behavior must follow the service contract rather than a universal retry-everything rule.
Aborted work needs verified cleanup
Incomplete uploads can retain parts and incur storage usage until they are completed or removed. Stop or join concurrent part transfers before cleanup where the client design permits it. S3 documents that in-flight part operations may still succeed around an abort and that repeated abort action can be needed to release all retained parts. Use the documented listing and cleanup checks rather than assuming a single returned abort header proves no staged bytes remain. Lifecycle cleanup of incomplete uploads is a separate backstop, not proof that the client recovered its own state.
The queue lab visualizes a normalized backlog of abandoned-upload cleanup work. Reset, set both λ=3 and μ=2, predict growth from four initial items, then Play or Step. For a separate case, Reset and set both λ=2,μ=3 before predicting drainage. Editing pauses but retains elapsed time and recomputes the fixed-rate model from its initial four. This is not a multipart state machine or a byte-occupancy estimator: uploads have different sizes, part races, and completion outcomes that the lab does not represent.
Work it through
Budget a part retry
Partition a 130 MiB object into target parts of 32 MiB, with the final smaller part allowed by the API. Assume the initial payload is fully sent once and exactly one 32 MiB part is then retransmitted in full.
Compute ceil(130/32) = 5 parts. Four full parts contain 128 MiB and the final part contains 2 MiB, giving the intended 130 MiB object payload.
The modeled wire payload is 130 + 32 = 162 MiB after one full-part retry. Re-uploading the entire payload instead would require 260 MiB under the same full-attempt counting assumption.
The difference is 98 MiB of modeled payload transfer. Completion still needs the selected part list and a parsed success result; the smaller retry budget does not certify that publication or cleanup succeeded.
Five parts and one full-part retry use 162 MiB of modeled transfer, while the completed object's intended payload remains 130 MiB.
The living notebook
Explore the model
Interactive lab · queue
Abandoned staging work can outgrow cleanup
For each independent run, Reset, set both λ and μ, predict the backlog trend from four cleanup items, then Play or Step. The fluid model does not track upload IDs, part sizes, completion responses, or actual retained bytes.
- Fluid backlog
- 4
- Net rate · λ − μ
- −1 /s
- M/M/1 mean residence
- 1 s
Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forMultipart reliability includes publication evidence and cleanup of intermediate state, not merely successful part transfers.
Pause & reason
Check your understanding
Your turn
Try it yourself
A process abandons an upload while two part transfers are still in flight, sends one abort, then exits without checking state. Why can that leave cleanup unproven, and what evidence should its recovery design collect?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
The abort races with operations already accepted for processing. Separate stopping new work from verifying removal of existing staged parts.
Compare with the solution
In-flight parts can finish around the abort according to the documented API behavior, so one abort response alone does not establish that all staged storage has been released. Recovery should retain the bucket, key, and upload ID, stop or reconcile transfers, perform the required abort/list checks, and repeat cleanup where the contract requires it. A lifecycle rule can provide a backstop, but it does not replace observing the intended upload's terminal cleanup state.
Keep the idea
Take it with you
- Multipart parts are staged state until the completion operation succeeds.
- Completion HTTP headers can precede the final body-level success or error.
- Retries and cleanup need upload identity, part evidence, and race-aware recovery.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.