Define useful output before dividing energy by tokens
Keep accepted output, processing volume, quality, and completed throughput aligned with the measured interval.
On this page
The question for today
What you’ll understand
- Compute energy per useful output and useful throughput from a matched interval.
- Distinguish accepted output from draft, padding, retry, and processing tokens.
- Explain why lower power or higher raw throughput does not alone prove service efficiency.
Energy per token is a ratio whose denominator can change meaning quietly. A system might count input tokens, generated candidates, accepted output, or padded batch positions. A speculative decoder can process many candidates while delivering fewer useful tokens, and a retry can repeat work without adding useful service. Define the output unit, tokenizer, quality requirement, and electrical window first. Then an efficiency comparison can describe the same delivered service rather than reward an implementation for inflating its own processing count.
Give the denominator a service contract
A useful-output definition should identify which responses were completed, delivered, and accepted under the evaluation policy. If an application compares quality-constrained answers, a shorter inaccurate answer cannot automatically earn an efficiency win through fewer tokens. Match input lengths, output limits, model or quality criteria, and request mix, or stratify them transparently. Token counts from different tokenizers are not a common physical unit of service without an explicit mapping or task-level comparison.
Maintain separate counters for inputs, accepted outputs, speculative drafts, padding, retries, and failed requests where relevant. Their sum is not generally useful output, but their computation belongs in the energy boundary if it was incurred to serve that output. Excluding the energy of rejected drafts while counting their token processing would distort both sides of the ratio. A complete job account includes wasted work; a component microbenchmark needs a narrower name.
Connect energy, power, and achieved service
At a matched boundary, energy per useful token equals average power divided by useful-token throughput. Both numerator and denominator matter. Batching can increase power while increasing useful throughput even faster, reducing energy per token. However, a throughput improvement that increases queueing latency beyond the service objective is not a feasible win for that objective. Measure completed service rather than assuming the hardware’s theoretical operation rate equals delivered tokens per second.
A fixed-duration load test and a fixed-work experiment answer related but different questions. The former should report arrivals, completions, failures, and backlog so deferred work is not silently dropped. The latter should run until the declared outputs finish and account for the complete interval. Saturation can improve one ratio while reducing interactive responsiveness. Present the achieved workload and service constraints with the ratio, instead of allowing an unconstrained maximum-throughput benchmark to stand in for production behavior.
Match lifecycle and concurrent-load coverage
A steady-state generation measurement may exclude loading weights, compiling kernels, building caches, or warming the service. Those costs can be amortized for a specific deployment frequency, but the assumed number of outputs must be declared. A per-token inference result is not automatically a full model-lifecycle result. Sustainable AI research emphasizes that development, data processing, training, and serving can require separate accounts before an overall lifecycle conclusion is drawn.
On a shared node, distinguish directly metered totals from per-service allocations. Changes to one tenant can alter batching, idle periods, or interference for others. An allocated ratio is still useful when its policy is clear and reconciles to the measured parent account, but it is not automatically the marginal energy of one additional token. Preserve absolute energy and useful output totals so readers can detect denominator changes and recompute the ratio under another legitimate service definition.
Work it through
Candidate processing makes a tempting denominator
Two fictional settings deliver the same 100 quality-matched accepted output tokens. A uses 200 W for 2 seconds. B uses 300 W for 1 second while processing 150 candidate tokens, of which only 100 become the accepted output. Powers cover the same complete declared boundary.
A consumes 200×2=400 J, or 400/100=4 J per useful token. Its useful throughput is 100/2=50 tokens/s. The shared useful-output count is fixed by the task, not by internal processing.
B consumes 300×1=300 J, or 300/100=3 J per useful token, at 100 useful tokens/s. Average power rose by 50%, but completed useful throughput doubled, giving a 25% energy-per-useful-token reduction.
Dividing B’s 300 J by its 150 candidates yields 2 J per candidate, a different metric. Comparing that number with A’s 4 J per useful token would incorrectly suggest a 50% reduction by mixing denominator definitions.
The matched useful-service comparison is 4 versus 3 J per token, a 25% reduction in this fixture. Candidate-processing efficiency cannot replace the useful-output ratio.
The living notebook
Explore the model
Interactive lab · queue
Completed capacity and arrivals remain separate
Reset, set Arrival rate λ to 2 and Service rate μ to 3; predict fluid backlog 3.5 after one Step (t=0.5 s). Reset, set λ to 4 and μ to 3; predict backlog 4.5 after one Step. Each run starts with backlog 4. This deterministic fluid model illustrates service capacity and unfinished work, not token generation, watts, latency tails, or a measured serving benchmark.
- Fluid backlog
- 4
- Net rate · λ − μ
- −1 /s
- M/M/1 mean residence
- 1 s
Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forA throughput claim should account for work still waiting, not merely submitted work.
Pause & reason
Check your understanding
Your turn
Try it yourself
A fictional system delivers 250 matched useful tokens using 300 W for 5 seconds. Another uses 250 W for 6 seconds to deliver the same 250. Compare energy per token and useful throughput.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Calculate total joules and completed tokens per second separately.
Compare with the solution
Both runs consume 1500 J, so both use 1500/250=6 J per useful token. Throughput is 250/5=50 tokens/s and 250/6≈41.667 tokens/s. Lower average power does not improve energy per useful token here; it extends runtime. Whether the slower setting is acceptable also depends on the declared service constraints.
Keep the idea
Take it with you
- Define useful output and retain every associated energy cost.
- Energy per useful token equals matched average power divided by useful throughput.
- Report quality, latency, absolute energy, and counting rules with efficiency ratios.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.