Rebuild trust after a builder compromise

Restrict the compromised builder's ability to issue new outputs, access secrets, publish artifacts, or create trusted assertions, using the incident's authorized response procedures. Preserve relevant logs, artifact identities, configuration, and access evidence so investigation remains possible. Credential rotation addresses future use of those credentials; artifact blocklisting or changed admission policy addresses acceptance of particular outputs. These are related but different controls. A previously signed artifact can remain deployed after key rotation, while an unaffected artifact might require a carefully justified policy path if an authority is broadly distrusted.

Build a candidate set from the earliest plausible compromise time, affected builder modes, accessible credentials, and downstream consumers. Detection time is not automatically the start of attacker access. Use independent sources such as registry records, deployment inventories, source-control history, and retained artifacts where available. Provenance generated by the compromised builder can help locate claims to investigate, but cannot by itself prove the claims true. Mark uncertain boundaries explicitly and expand investigation when evidence is incomplete. An absence of suspicious log entries is weak reassurance if the attacker could alter those logs.

Establish a clean build environment with reviewed source identities, approved dependency artifacts, and toolchain inputs whose trust is justified independently of the compromised path. Reusing an untrusted output cache or a base image produced by the attacker can carry the compromise into a new build. Reevaluate the controls that failed, including who can alter build definitions and which platform component generates provenance. A freshly created machine is not sufficient if it automatically downloads the same compromised inputs. Record the actual trust boundary and retained evidence for replacement artifacts.

Run the relevant integrity, provenance, reproducibility, and behavioral checks for the replacement release, recognizing that each answers a different question. A new signature on an old suspect binary changes attribution, not its behavior. Byte-identical rebuilding may help compare a known source and output, but matching a common compromised toolchain is not independent exoneration. Generate replacement attestations through the restored authority and update inventory bindings. Decide which existing deployments need replacement or containment using actual running identities, not merely the current value of a release tag or the success of new admission checks.

A rollback target must be a known approved artifact compatible with the current environment and data, not merely a tag called previous. Confirm that the target lies outside the investigated compromise path or has been independently cleared under the response policy. Track replacement and retirement across production, staging, mirrors, caches, and scheduled jobs as relevant. Monitor for unauthorized publishing or identity use after containment. Keep unresolved candidates visible rather than declaring success when one new release is signed. Recovery is complete only against a stated scope and acceptance criteria whose evidence can be reviewed.

The queue lab can illustrate capacity planning for the reconstruction workload. For each separate run, Reset, set both λ and μ, predict, then Play or Step. With λ=4,μ=2 the initial four-job backlog grows by two jobs per second; with λ=2,μ=4 a fresh four-job backlog drains in two seconds. Control changes are separate fixed-rate runs, not a live intervention on the previous backlog. This exact fluid calculation cannot identify compromised artifacts or clear replacements. It helps show that a response plan needs sufficient completion capacity alongside the independent checks that make each completed job meaningful.