Admit and test multimodal requests against representation-aware resource and failure contracts

For a simple full-attention autoregressive decoder with uncompressed keys and values, cache bytes depend on batch, cached positions, layer count, KV-head count, head dimension, and bytes per scalar. Use KV heads rather than assuming they equal query heads in grouped-query attention. Visual prefix positions count if the architecture stores them in this decoder sequence. Separate visual cross-attention memory or encoder caches require additional accounting. Sliding, chunked, quantized, offloaded, and statically allocated caches have different shapes or residency behavior, so this formula is an explicit model rather than a universal serving rule.

Reserve generation according to the service's policy and compatible cache implementation. A static cache can allocate its maximum capacity before every position is used; a dynamic cache may grow and create transient allocation effects. Beam or candidate expansion can increase effective batch. Include processor output length rather than only raw prompt text. Versioned Transformers 4.57.1 cache documentation describes different cache strategies, but availability and limits remain model-specific. Measure real allocation and support rather than selecting a strategy from its name alone.

Weights, decoded media, encoder intermediates, projected features, decoder cache, and temporary workspaces can overlap in memory. Peak usage is a simultaneous sum, not the sum of each component's separate maximum. Decode and resize may run on the host while features and caches reside on a device, so keep memory domains separate. A compressed upload's small byte count says little about its decoded dimensions or resulting patch count. Enforce declared pixel, duration, crop, and context limits before an oversized request reaches an expensive late stage.

Latency likewise depends on required dependencies and queues. Independent modality encoders may overlap, while a shared device can serialize them or cause contention. Prefill, generation, transfer, and feature-cache behavior vary by representation length and cohort. Measure completed request latency and time to useful output under stated failure handling; adding stage tail percentiles does not produce an end-to-end percentile. Report invalid and timed-out attempts rather than improving a successful-only metric by excluding the hardest media or longest contexts.

Test missing media, corrupt decode, unsupported formats, excessive processed length, feature-count mismatch, encoder errors, cancellation, and stage deadlines. Include mixed-validity batches to check isolation and resource cleanup. A documented text-only fallback must not claim to have inspected failed visual evidence, and a retry must not reuse another request's feature state. Record model, processor, media, connector, and serialization versions with the outcome. Pair resource tests with grounding and task-quality cohorts so fitting the budget does not hide information loss or a changed conditioning contract.

The queue lab illustrates a separate capacity-warning mechanism. Reset, set both λ=2 and μ=3, predict the stable M/M/1 mean residence one second, then Play or Step. Reset and set both λ=2.5 and μ=3 for two seconds. The visible fluid backlog starts at four and follows max(0,4+(λ−μ)t); the separate mean assumes Poisson arrivals and exponential single-server service. There are no encoders, cache bytes, media shapes, failure paths, or measured tail percentiles. Use actual completed-request and memory measurements to validate a multimodal serving contract.