Generation request batching and cancellation

Compatible image shape, model revision, precision, and conditioning layout are basic grouping requirements. The current noise coordinate, prediction parameterization, guidance branches, and solver state also matter. Some implementations can evaluate different timesteps per batch element; others assume one shared step or maintain a single scheduler index. Verify the actual kernel and pipeline support rather than replacing independent trajectories with a shared loop counter. A batch assembled from newly arriving requests needs an explicit map from slots to request-owned states.

For requests with different evaluation budgets, a fixed batch running until the longest trajectory ends may waste slot work on already finished entries. Compacting the active set can avoid those evaluations in a suitable implementation, but it changes batch shapes, launch behavior, and sometimes numerical kernel selection. A smaller active batch may be less hardware-efficient per image. Count useful evaluations and measure end-to-end completion time, including compaction and decode queues, before concluding that removing padded slots produces a proportional latency reduction.

When a client cancels, mark the request so no new unnecessary work is scheduled under the policy. Work already submitted to a GPU stream may still access its buffers. A callback that stops future denoising steps is not a general primitive for interrupting a running kernel. Keep the slot’s storage and identity protected until the relevant operations complete, then release histories, conditioning references, latents, and outputs safely. Generation or epoch tags can prevent a late completion from being mistaken for a newly admitted request in a recycled slot.

Diffusers v0.35.1 documents step-end interruption callbacks for specific pipelines, and the classic implementation checks interruption between denoising iterations. That behavior does not automatically provide per-element cancellation in a batch, asynchronous multi-request isolation, or removal of later decode work. Inspect the returned-output and postprocessing path too. A production scheduler may need its own cancellation result and cleanup state machine rather than exposing a partially denoised image as a normal completed response. Preserve request-specific random streams so canceling one request does not shift another’s future draws.

Admission should account for trajectory state and eventual decode/output memory, not just current denoiser occupancy. A busy decode stage can make completed latents accumulate even when denoising slots turn over quickly. Apply backpressure at a known boundary and report queue delay, denoise service, decode service, canceled work, and time to usable output. Test mixed step counts, canceled long requests, slow clients, and interleaved scheduler histories. A throughput number from identical uncanceled requests does not characterize that lifecycle.

The queue instrument illustrates admission pressure only. Reset, set both λ=4 and μ=3, predict backlog six at t=2 from four, then Step four times. Separately Reset, set both λ=2 and μ=3, predict drainage at t=4, then Step eight times. These fresh runs compare offered and service rates in a fixed fluid model. It has no batch slots, solver coordinates, cancellation events, GPU completion, or decode memory. Lower offered rate in the diagram is not evidence that an actual cancellation implementation safely reclaimed device buffers or improved generation goodput.