Fragment placement determines which failures overlap

In Ceph, objects map into placement groups, and placement groups map to storage daemons using the cluster's placement configuration. Grouping lets the system track and manage sets of objects rather than maintaining a fully separate global placement record for each object. A placement group is a logical management unit, not a physical rack or a promise that all its fragments fail independently. The actual acting and desired locations depend on cluster maps, rules, and current daemon state.

CRUSH uses a topology hierarchy and placement rules to select locations, with weights and failure-domain choices influencing the result. A rule separating fragments by host protects against a different set of correlated failures than a rule separating them by rack. The topology must accurately describe where devices reside, and enough eligible domains must exist to satisfy the rule. Changing a label in a map without changing shared physical dependencies does not create isolation from those dependencies.

For a host-separated 4+2 stripe with six fragments distributed two per rack across three racks, losing one complete rack removes two fragments and leaves four. This is not a one-fragment-per-rack placement rule. Losing two racks removes four and leaves only two, below the code's general reconstruction threshold. Placing each fragment on a distinct host is still useful within that layout, but it does not turn the three racks into six independent rack domains. State the largest failure unit the design claims to tolerate and test the actual allocation against it.

Physical and administrative correlations extend beyond the declared hierarchy. Shared network uplinks, controller firmware, credentials, and maintenance batches can affect several nominal domains at once. An independent-disk probability multiplied across replicas ignores those mechanisms. Also inspect temporary placement during expansion, recovery, and maintenance: the steady-state rule can be satisfied before a failure while the remaining eligible space is insufficient to restore it afterward. Capacity and failure-domain feasibility must be evaluated in the degraded topology too.

A cluster can have aggregate free space yet lack a legal destination in the required failure domain. Rebalancing also consumes network and device bandwidth while clients continue using the surviving locations. Observe undersized, degraded, remapped, and recovery-related states according to the deployed Ceph version rather than flattening them into a single healthy/unhealthy badge. The states describe different relationships between desired redundancy, current placement, and ongoing work, and can call for different investigation.

Use the probability lab to challenge an independent-domain toy model. Reset, set both p=0.9 and n=3, interpret success as one rack being available, and predict the mass for at least two available racks. With the equal two-fragments-per-rack layout, that corresponds to retaining at least four fragments under the stated all-or-nothing rack model. Play or Step highlights exact outcomes only. It does not construct CRUSH placements, model shared power, or justify the assumed independence; compare the deterministic failure counts with the real topology first.