Latent decoding and memory peaks

A stipulated spatial compression factor f maps an H-by-W image to a smaller latent grid, with a model-specific latent channel count. This factor and the latent scaling convention come from the autoencoder and training configuration; they are not universal constants for every image or video model. A latent denoiser expects the representation it was trained on. Applying a decoder from another revision or forgetting a configured scale reversal can produce artifacts without any memory error or tensor-shape exception.

The latent-diffusion paper motivates perceptual compression and its fidelity tradeoffs. Diffusers v0.35.1’s AutoencoderKL exposes latent channels, scaling-related configuration, slicing, tiling, and precision behavior. Inspect the actual model rather than hardcoding a familiar scaling constant. Some pipelines upcast parts of VAE decoding for numerical stability, changing both element widths and memory pressure. Output conversion to arrays, images, encoders, or response buffers can retain additional full-resolution copies after the GPU computation appears complete.

List resident weights, conditioning, current latents, scheduler histories, denoiser activations, decoder workspaces, decoded outputs, and transfer buffers. Peak live allocated memory is the maximum sum of allocations live at the same instant, not the sum of every stage’s independent maximum or the largest tensor alone. If model weights remain resident across stages, include them at each relevant instant; if offloading changes residency, include transfer and staging lifetimes. Concurrent denoise and decode requests can overlap peaks that a serial single-request benchmark never exposes.

VAE slicing processes a multi-image batch in smaller pieces, reducing the batch-dependent activation component while preserving other resident allocations. Spatial tiling addresses a different dimension and can introduce tile-dependent output differences. Neither mechanism automatically shrinks parameters, all scheduler history, or full outputs retained by the caller. Model offload can reduce residency but introduces transfers and scheduling costs. Measure the deployed configuration’s peak allocated and reserved memory separately and verify headroom under the actual concurrency mix.

Compare the final output under slicing, tiling, offloading, or precision changes using controlled latents and suitable numerical or perceptual criteria. A lower peak that changes colors, seams, or saturation is a quality tradeoff needing evidence. Record the decode and response stages separately from iterative denoising so an optimization cannot hide a new bottleneck. Admission must reserve enough capacity for later stages or use an explicit decode queue and bounded output storage; otherwise a large completed latent backlog can create a delayed memory incident.

The roofline instrument illustrates a resource ceiling at a declared local memory boundary. Reset, set both I=0.5 and B=100, predict a fifty-GFLOP/s ceiling, then Play or Step. Separately Reset, set both I=2 and B=100, predict two hundred GFLOP/s, then Play or Step. Interpret I relative to a stipulated last-level-cache/main-memory boundary only. The model’s compute ceiling is fixed at one thousand GFLOP/s. It contains no VAE graph, activation allocator, output format, or GPU product measurements and cannot estimate a real decoder’s throughput or peak memory.