advanced25 min studyLesson 221 in Engineering

Clos fabrics and a declared bisection

Count ports and traffic across a precise cut before claiming full fabric bandwidth.

On this page

What you’ll understand

  • Count hosts and switches in a specified k-ary fat tree.
  • Distinguish one-direction cut capacity from summed full-duplex capacity.
  • Explain why topology capacity does not guarantee a workload's throughput.

A drawing with many parallel links can hide an oversubscribed boundary. Fabric reasoning begins with a traffic matrix and a clearly defined cut: which endpoints send to which others, at what rate, and through which links? Clos networks create path diversity using multiple switching stages. Their capacity depends on port allocation, routing, and placement, so count an explicit construction instead of treating the word Clos as a promise of unlimited bandwidth.

Count the construction before the throughput

Use the conventional three-stage k-ary fat-tree construction with even k and identical k-port switches. There are k pods, each containing k/2 edge switches and k/2 aggregation switches. Every edge switch uses k/2 ports for hosts and k/2 for aggregation links. The core contains (k/2)² switches. All links in this declared construction have the same full-duplex line rate; other designs called Clos may use different radix, speeds, or oversubscription.

Each aggregation switch connects to the appropriate core group, providing multiple equal-length paths between pods. Count endpoints separately from switch ports: every inter-switch cable consumes a port at both ends, while a host-facing cable consumes one switch port. Adding host ports without matching upstream capacity changes the design's oversubscription. Real deployments may reserve ports, combine switch roles, or use breakout links, so validate the bill of materials rather than extrapolating directly from the ideal construction.

H=k34,Sedge=Sagg=k22,Score=k24H=\frac{k^3}{4},\qquad S_{\rm edge}=S_{\rm agg}=\frac{k^2}{2},\qquad S_{\rm core}=\frac{k^2}{4}
k is an even switch radix and all results count devices in the stated fat tree. H counts single-attached hosts, not bidirectional traffic or application ranks.↔ Scroll wide equations horizontally.

A cut has a direction and a demand

Split the hosts into equal halves and consider traffic sent from one half to the other. With H hosts each attached at rate r, their total offered rate cannot exceed Hr/2 in that direction. A nonoversubscribed construction has enough ideal cut capacity to support full host injection for suitable admissible traffic and routing. Summing both directions gives a different number. Always specify whether a vendor's aggregate figure counts transmit and receive together.

Capacity is a necessary resource budget, not proof that a collection of flows will realize it. Flow hashing can concentrate large flows, individual destinations can receive more than their access links support, and failures remove paths. A permutation with each host sending to one distinct destination differs from many senders converging on one receiver. Define admissibility at ingress, egress, and internal cuts before comparing a workload with a full-bisection claim.

Chalf→half=Hr2,O=Coffered downCavailable upC_{\rm half\to half}=\frac{H r}{2},\qquad O=\frac{C_{\rm offered\ down}}{C_{\rm available\ up}}
The first expression is a one-direction ideal half-host injection budget under the declared nonoversubscribed design. r and capacities share bit/s units. O is a dimensionless oversubscription ratio at a named boundary, not a universal fabric-wide scalar.↔ Scroll wide equations horizontally.

Test the boundary that the application crosses

Map training ranks and storage clients onto the topology, then estimate which stage carries their synchronized transfers. An intra-node operation may never use an external fabric link, while an all-to-all across pods can stress several cuts. Compare endpoint injection, rack uplinks, and cross-pod capacity separately. A test that reaches one host's line rate cannot establish bisection capacity, and a random traffic average can hide a consistently overloaded placement.

The queue lab is an abstract rate-budget exercise. Reset and set both λ=4 and μ=2, predict growth from four jobs, then Step. Reset again and set λ=2 and μ=4 before predicting decline and stepping. This illustrates offered demand exceeding a service budget, not Clos routing, packet buffers, or bit-rate conversion. Editing controls recomputes the whole trajectory at retained time; Reset makes the comparisons independent. The separately labeled stochastic waiting-time formula has additional assumptions.

A four-port fat tree

Construct the stated k=4 network with 100 Gbit/s full-duplex links. Count hosts and switches, then calculate one-direction half-to-half host injection.

  1. Each of four pods has two edge and two aggregation switches. There are eight edge switches, eight aggregation switches, and four core switches: twenty switches total.

  2. Each edge has two host-facing ports, giving eight×two=16 hosts. Half the hosts can inject eight×100=800 Gbit/s toward the other half under the ideal admissible traffic assumptions.

  3. Counting simultaneous opposite-direction injection as well gives 1,600 Gbit/s. Do not report that summed full-duplex number as 1,600 Gbit/s available to a one-way transfer; destination and routing constraints still apply.

The result

The port count supports an explicit 800 Gbit/s one-direction injection budget, not a guaranteed measured collective rate.

Explore the model

Interactive lab

Interactive lab · queue

Offered load against a service budget

For independent runs Reset, set both rates, predict, then Step: λ=4, μ=2 grows from four; λ=2, μ=4 declines. This fluid jobs model does not represent Clos links, packet buffers, or a fabric benchmark.

Offered load against a service budgetFluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Current readings: Fluid backlog: 4; Net rate · λ − μ: −1 /s; M/M/1 mean residence: 1 s. All parameters are adjustable with the labeled controls below.02.557.51001.534.56Elapsed time · sFluid backlogλ < μ
Fluid backlogCurrent time
Fluid backlog
4
Net rate · λ − μ
−1 /s
M/M/1 mean residence
1 s
t = 0 s

Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forName the constrained boundary and compare compatible demand and capacity units before drawing a throughput conclusion.

Check your understanding

A cut is advertised as 1.6 Tbit/s summed full-duplex capacity. What additional fact is needed for a one-way demand calculation?

Try it yourself

For the same conventional construction with k=8 and link rate 50 Gbit/s, calculate hosts, core switches, and one-direction half-host injection. State one condition that prevents treating the result as achieved throughput.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Use k³/4 hosts and count only half of their single-direction injection.

Compare with the solution

There are 8³/4=128 hosts and 8²/4=16 core switches. Half-host injection is 64×50=3,200 Gbit/s, or 3.2 Tbit/s in one direction. A many-to-one demand can exceed a destination's access rate even with sufficient aggregate fabric capacity; flow imbalance or routing constraints can also prevent attaining the ideal figure.

Take it with you

  • A Clos capacity claim needs port, speed, and routing assumptions.
  • One-direction and summed full-duplex capacities differ.
  • Application traffic matrices determine which cut matters.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.