Clos fabrics and a declared bisection
Count ports and traffic across a precise cut before claiming full fabric bandwidth.
On this page
The question for today
What you’ll understand
- Count hosts and switches in a specified k-ary fat tree.
- Distinguish one-direction cut capacity from summed full-duplex capacity.
- Explain why topology capacity does not guarantee a workload's throughput.
A drawing with many parallel links can hide an oversubscribed boundary. Fabric reasoning begins with a traffic matrix and a clearly defined cut: which endpoints send to which others, at what rate, and through which links? Clos networks create path diversity using multiple switching stages. Their capacity depends on port allocation, routing, and placement, so count an explicit construction instead of treating the word Clos as a promise of unlimited bandwidth.
Count the construction before the throughput
Use the conventional three-stage k-ary fat-tree construction with even k and identical k-port switches. There are k pods, each containing k/2 edge switches and k/2 aggregation switches. Every edge switch uses k/2 ports for hosts and k/2 for aggregation links. The core contains (k/2)² switches. All links in this declared construction have the same full-duplex line rate; other designs called Clos may use different radix, speeds, or oversubscription.
Each aggregation switch connects to the appropriate core group, providing multiple equal-length paths between pods. Count endpoints separately from switch ports: every inter-switch cable consumes a port at both ends, while a host-facing cable consumes one switch port. Adding host ports without matching upstream capacity changes the design's oversubscription. Real deployments may reserve ports, combine switch roles, or use breakout links, so validate the bill of materials rather than extrapolating directly from the ideal construction.
A cut has a direction and a demand
Split the hosts into equal halves and consider traffic sent from one half to the other. With H hosts each attached at rate r, their total offered rate cannot exceed Hr/2 in that direction. A nonoversubscribed construction has enough ideal cut capacity to support full host injection for suitable admissible traffic and routing. Summing both directions gives a different number. Always specify whether a vendor's aggregate figure counts transmit and receive together.
Capacity is a necessary resource budget, not proof that a collection of flows will realize it. Flow hashing can concentrate large flows, individual destinations can receive more than their access links support, and failures remove paths. A permutation with each host sending to one distinct destination differs from many senders converging on one receiver. Define admissibility at ingress, egress, and internal cuts before comparing a workload with a full-bisection claim.
Test the boundary that the application crosses
Map training ranks and storage clients onto the topology, then estimate which stage carries their synchronized transfers. An intra-node operation may never use an external fabric link, while an all-to-all across pods can stress several cuts. Compare endpoint injection, rack uplinks, and cross-pod capacity separately. A test that reaches one host's line rate cannot establish bisection capacity, and a random traffic average can hide a consistently overloaded placement.
The queue lab is an abstract rate-budget exercise. Reset and set both λ=4 and μ=2, predict growth from four jobs, then Step. Reset again and set λ=2 and μ=4 before predicting decline and stepping. This illustrates offered demand exceeding a service budget, not Clos routing, packet buffers, or bit-rate conversion. Editing controls recomputes the whole trajectory at retained time; Reset makes the comparisons independent. The separately labeled stochastic waiting-time formula has additional assumptions.
Work it through
A four-port fat tree
Construct the stated k=4 network with 100 Gbit/s full-duplex links. Count hosts and switches, then calculate one-direction half-to-half host injection.
Each of four pods has two edge and two aggregation switches. There are eight edge switches, eight aggregation switches, and four core switches: twenty switches total.
Each edge has two host-facing ports, giving eight×two=16 hosts. Half the hosts can inject eight×100=800 Gbit/s toward the other half under the ideal admissible traffic assumptions.
Counting simultaneous opposite-direction injection as well gives 1,600 Gbit/s. Do not report that summed full-duplex number as 1,600 Gbit/s available to a one-way transfer; destination and routing constraints still apply.
The port count supports an explicit 800 Gbit/s one-direction injection budget, not a guaranteed measured collective rate.
The living notebook
Explore the model
Interactive lab · queue
Offered load against a service budget
For independent runs Reset, set both rates, predict, then Step: λ=4, μ=2 grows from four; λ=2, μ=4 declines. This fluid jobs model does not represent Clos links, packet buffers, or a fabric benchmark.
- Fluid backlog
- 4
- Net rate · λ − μ
- −1 /s
- M/M/1 mean residence
- 1 s
Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forName the constrained boundary and compare compatible demand and capacity units before drawing a throughput conclusion.
Pause & reason
Check your understanding
Your turn
Try it yourself
For the same conventional construction with k=8 and link rate 50 Gbit/s, calculate hosts, core switches, and one-direction half-host injection. State one condition that prevents treating the result as achieved throughput.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Use k³/4 hosts and count only half of their single-direction injection.
Compare with the solution
There are 8³/4=128 hosts and 8²/4=16 core switches. Half-host injection is 64×50=3,200 Gbit/s, or 3.2 Tbit/s in one direction. A many-to-one demand can exceed a destination's access rate even with sufficient aggregate fabric capacity; flow imbalance or routing constraints can also prevent attaining the ideal figure.
Keep the idea
Take it with you
- A Clos capacity claim needs port, speed, and routing assumptions.
- One-direction and summed full-duplex capacities differ.
- Application traffic matrices determine which cut matters.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.