Threat modeling AI systems
Turn an AI architecture into explicit assets, adversary capabilities, trust boundaries, and testable security claims.
On this page
The question for today
What you’ll understand
- Describe an adversary through capabilities rather than a vague threat label.
- Separate a security invariant from a model-quality target.
- Use conditional probabilities without assuming defenses fail independently.
A document assistant can answer accurately and still send a private attachment to the wrong recipient. Security asks which behavior must remain forbidden even when inputs, dependencies, or accounts are adversarial. Start with one concrete workflow: a user searches documents, a retriever supplies passages, a model proposes an action, and a service executes it. The threat model explains where authority comes from at every transition.
Name assets and authority
List what matters before listing controls: document confidentiality, integrity of approved actions, model availability, training-data rights, and the ability to reconstruct a decision. Each asset needs an owner and a failure statement. For example, a retrieved document must never expand the caller's document permissions. That is a security invariant; a target of ninety percent helpful answers is a different kind of requirement.
Draw a data-flow diagram with stores, services, external inputs, and privilege changes. A trust boundary appears when information enters a context with different authority, including a training-data import or a tool call. An internal network address does not settle trust. Record which component checks the user's identity and which component authorizes each resource, so an attractive interface cannot conceal a missing enforcement point.
Specify the adversary and the evidence
An adversary who edits one public document has different capabilities from an administrator who replaces model weights. State access, knowledge, budget, persistence, and desired outcome. Separate accidental malformed input from intentionally adaptive input, while testing both. A useful scenario describes the initial foothold, the boundary crossed, and an observable consequence without requiring a fully operational exploit to justify defensive design.
For each scenario, connect prevention, detection, containment, and recovery to evidence. A permission check can be tested with denied identities; an alert needs a known signal and an accountable responder. Record assumptions such as a trusted identity issuer or an uncompromised build service. Assumptions are not embarrassing footnotes: they identify which new evidence or architectural change should trigger a fresh assessment.
Quantify carefully and revisit
If compromise requires two stages, use the probability of the second stage conditional on the first. Multiplying two separately measured failure rates assumes independence that shared models, datasets, or operators may violate. Expected loss can compare scenarios when probabilities and consequences are meaningful, but uncertain estimates deserve ranges. Rare severe outcomes and contractual obligations may require controls even when a crude average appears small.
Threat modeling is an engineering artifact attached to a system version. Revisit it after adding retrieval sources, expanding tool privileges, changing tenancy, or importing new weights. Track unresolved risks with an owner and a decision date. A passed test suite only supports the tested scenarios under the stated setup; it does not establish that every possible adversary has been anticipated.
Work it through
Compare two containment changes
In an illustrative assessment, an untrusted passage influences a proposed action with probability 0.2. Given that event, an independent authorization service wrongly permits it with conditional probability 0.01. A resulting incident costs 50,000 accounting units. Compare reducing that conditional probability to 0.002.
Compute the joint probability from the conditional rate: 0.2 × 0.01 = 0.002 per assessed opportunity.
Multiply by the assumed consequence: 0.002 × 50,000 = 100 expected units per opportunity, under this deliberately simplified scenario.
With the improved authorization service, the joint probability becomes 0.2 × 0.002 = 0.0004 and expected loss becomes 20 units.
The modeled reduction is 80 units per opportunity. Validate the conditional rate with relevant tests; the example does not supply a real-world incident forecast.
Explicit conditioning makes the calculation inspectable. The architectural value is an enforcement boundary whose behavior can be tested independently of model helpfulness.
The living notebook
Explore the model
Interactive lab · probability
Explore repeated assessment outcomes
Use probability p and trial count n to inspect an exact binomial distribution. Interpret each trial as an independent, identical assessment with an assumed outcome probability. Adaptive attacks, shared defects, and changing conditions violate these assumptions; the lab does not discover attack likelihood.
- Mean · np
- 5
- Variance · np(1−p)
- 2.5
- Total probability
- 1
These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forRepeated tests increase evidence only when their sampling assumptions and coverage are understood.
Pause & reason
Check your understanding
Your turn
Try it yourself
An assessment estimates P(A)=0.1 and P(B given A)=0.04, with 200,000 units of loss if both occur. Calculate the joint probability and expected loss, then state one missing assumption needed for a deployment decision.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Use the conditional multiplication rule; distinguish the resulting arithmetic from confidence in its inputs.
Compare with the solution
The joint probability is 0.004 and modeled expected loss is 800 units per assessed opportunity. A deployment decision still needs the opportunity frequency, uncertainty in the conditional rate, consequence distribution, and whether scenarios overlap. Any one of these can materially change the comparison.
Keep the idea
Take it with you
- Threat models name assets, adversary capabilities, and explicit enforcement boundaries.
- Conditional failure rates are more informative than unjustified independence assumptions.
- Security evidence belongs to a stated architecture, version, and assessment scope.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.