advanced25 min studyLesson 312 in Engineering

Multipart upload separates staging from publication

Track upload IDs, numbered parts, completion results, ambiguous responses, and cleanup races without mistaking staged bytes for a completed object.

On this page

What you’ll understand

  • Model initiation, part upload, completion, and abort as separate states.
  • Explain why an initial completion HTTP 200 is insufficient by itself.
  • Budget retries and verify cleanup of abandoned parts.

Large-object upload is easier to retry when independent parts can be transferred separately. The tradeoff is a protocol with durable intermediate state: an upload ID, part numbers, returned validators, and an explicit completion request. Staged parts consume storage without necessarily creating the current object readers expect. A correct client must distinguish acknowledged parts from successful publication and must clean up work it abandons. The following details describe Amazon S3's documented multipart API rather than every service that uses similar endpoint names.

One upload ID coordinates many part operations

Initiating an upload returns an upload ID associated with the destination bucket and key. Parts are uploaded under that ID with part numbers, and the client records the returned information required for completion. A later upload of the same part number replaces that part's staged content for the upload rather than appending a second logical part. Keep the intended part list and its validators in durable client state when recovery needs to survive a process restart.

Completion assembles the specified parts in the API's part-number ordering and creates the object after successful processing. The client must supply the correct complete list rather than assuming every previously attempted transfer is present. Part sizes and counts have API limits, including special handling of the final part, so choose chunking within the documented contract. Parallelism improves transfer utilization only up to the actual network, memory, and service limits; allocating an unbounded number of part buffers creates a new failure mode.

n=⌈SP⌉,S=(n−1)P+Plastn=\left\lceil\frac{S}{P}\right\rceil,\qquad S=(n-1)P+P_{\rm last}
S and target part size P use the same byte unit, n counts parts, and the final part is positive and at most P for this equal-prefix partition. Provider size/count constraints still apply. This formula partitions payload; it does not describe internal erasure-code shards.↔ Scroll wide equations horizontally.

Completion has a body-level outcome

Amazon S3 documents that CompleteMultipartUpload can send an initial HTTP 200 header while processing continues, then return an error inside the response body. A raw client that treats the header as final success can publish a manifest referring to an object whose upload did not complete. Consume and parse the full response according to the API, or use a supported SDK whose error handling covers that case. Whitespace while waiting is not a completed object result.

A lost connection after sending completion creates an ambiguous observation: the client may not know whether publication happened. Reconcile using the API's documented state and object evidence, preserving the upload identity and intended object properties. Blindly starting a new unconditional upload can replace another writer's object or create an extra version. Conditional completion, version-aware checks, and application operation identifiers can constrain that recovery, but the exact error-specific retry behavior must follow the service contract rather than a universal retry-everything rule.

Bsent=S+∑j∈RPjB_{\rm sent}=S+\sum_{j\in\mathcal{R}}P_j
This simplified transfer budget counts one initial complete payload attempt plus full retransmissions of parts in retry multiset R, using bytes throughout. It excludes protocol overhead and partially sent attempts. Repeated part numbers count again on the wire even though only their selected staged content is completed.↔ Scroll wide equations horizontally.

Aborted work needs verified cleanup

Incomplete uploads can retain parts and incur storage usage until they are completed or removed. Stop or join concurrent part transfers before cleanup where the client design permits it. S3 documents that in-flight part operations may still succeed around an abort and that repeated abort action can be needed to release all retained parts. Use the documented listing and cleanup checks rather than assuming a single returned abort header proves no staged bytes remain. Lifecycle cleanup of incomplete uploads is a separate backstop, not proof that the client recovered its own state.

The queue lab visualizes a normalized backlog of abandoned-upload cleanup work. Reset, set both λ=3 and μ=2, predict growth from four initial items, then Play or Step. For a separate case, Reset and set both λ=2,μ=3 before predicting drainage. Editing pauses but retains elapsed time and recomputes the fixed-rate model from its initial four. This is not a multipart state machine or a byte-occupancy estimator: uploads have different sizes, part races, and completion outcomes that the lab does not represent.

Budget a part retry

Partition a 130 MiB object into target parts of 32 MiB, with the final smaller part allowed by the API. Assume the initial payload is fully sent once and exactly one 32 MiB part is then retransmitted in full.

  1. Compute ceil(130/32) = 5 parts. Four full parts contain 128 MiB and the final part contains 2 MiB, giving the intended 130 MiB object payload.

  2. The modeled wire payload is 130 + 32 = 162 MiB after one full-part retry. Re-uploading the entire payload instead would require 260 MiB under the same full-attempt counting assumption.

  3. The difference is 98 MiB of modeled payload transfer. Completion still needs the selected part list and a parsed success result; the smaller retry budget does not certify that publication or cleanup succeeded.

The result

Five parts and one full-part retry use 162 MiB of modeled transfer, while the completed object's intended payload remains 130 MiB.

Explore the model

Interactive lab

Interactive lab · queue

Abandoned staging work can outgrow cleanup

For each independent run, Reset, set both λ and μ, predict the backlog trend from four cleanup items, then Play or Step. The fluid model does not track upload IDs, part sizes, completion responses, or actual retained bytes.

Abandoned staging work can outgrow cleanupFluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Current readings: Fluid backlog: 4; Net rate · λ − μ: −1 /s; M/M/1 mean residence: 1 s. All parameters are adjustable with the labeled controls below.02.557.51001.534.56Elapsed time · sFluid backlogλ < μ
Fluid backlogCurrent time
Fluid backlog
4
Net rate · λ − μ
−1 /s
M/M/1 mean residence
1 s
t = 0 s

Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forMultipart reliability includes publication evidence and cleanup of intermediate state, not merely successful part transfers.

Check your understanding

A raw CompleteMultipartUpload client receives an initial HTTP 200 header. What must it still do?

Try it yourself

A process abandons an upload while two part transfers are still in flight, sends one abort, then exits without checking state. Why can that leave cleanup unproven, and what evidence should its recovery design collect?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

The abort races with operations already accepted for processing. Separate stopping new work from verifying removal of existing staged parts.

Compare with the solution

In-flight parts can finish around the abort according to the documented API behavior, so one abort response alone does not establish that all staged storage has been released. Recovery should retain the bucket, key, and upload ID, stop or reconcile transfers, perform the required abort/list checks, and repeat cleanup where the contract requires it. A lifecycle rule can provide a backstop, but it does not replace observing the intended upload's terminal cleanup state.

Take it with you

  • Multipart parts are staged state until the completion operation succeeds.
  • Completion HTTP headers can precede the final body-level success or error.
  • Retries and cleanup need upload identity, part evidence, and race-aware recovery.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.