advanced30 min studyLesson 917 in Engineering

Keep generated text and structured proposals outside interpreter authority

Validate output representations, authorize proposed operations, and use the correct boundary for HTML, SQL, shell arguments and other downstream consumers.

On this page

What you’ll understand

  • Treat model output as untrusted data even when it matches a schema.
  • Separate context-specific interpreter handling from resource authorization.
  • Bound structured output size and verify the actual downstream effect.

A model’s output can be fluent, valid JSON, and still unsafe to pass into a privileged interpreter. The application chooses whether that string becomes HTML, a database query, a command argument, a file path or a tool operation. Those transitions are enforcement boundaries outside the language model. Validate the expected representation, preserve data/code separation in the chosen API, and authorize the actual action and target. A model confidence score or instruction to behave carefully cannot replace these controls.

Separate well-formed data from permitted operations

Use a bounded schema for a structured proposal, including allowed operation names, field types and collection sizes. Reject unsupported fields and nonfinite numeric values under the actual parser/validator contract. Then resolve resources and check current permissions using trusted caller context. A valid integer document ID may refer to another tenant’s data; a well-formed URL may point to an unapproved destination. Syntactic validity is necessary for the selected interface but does not establish ownership, intent or allowed side effects.

Bound the complete output and each nested collection before expanding it into tasks. A model that emits a thousand small actions can exceed a work budget even if each action is individually valid. Keep a server-side maximum on operation count and cumulative cost, and require any task-specific approval at the actual execution boundary. These are proposed application controls; a structured-output API or JSON Schema implementation does not automatically implement the resource, destination or authorization policy.

execute(p,o)=V(o)∧A(p,o)∧B(o){\rm execute}(p,o)=V(o)\land A(p,o)\land B(o)
V checks the bounded output representation, A authorizes the resolved operation o for trusted principal p, and B checks the required work/resource reservation. This Boolean design separates validation, permission and budget. It does not assert that a schema or a model enforces the three predicates.↔ Scroll wide equations horizontally.

Use the consumer’s data interface and its exact context rules

For ordinary UI text, use a text-rendering path. If HTML is intentionally supported, use an appropriate sanitizer and context-specific handling rather than treating arbitrary model HTML as trusted. For SQL, parameterized value binding separates values from query structure, while dynamic identifiers or operation choices need an explicit allowed mapping. A read-only account can reduce authority but does not authorize reading every tenant’s rows. The database must still enforce the actual access policy.

For operating-system work, prefer a narrow library or fixed executable interface with validated arguments over building a shell command string. Avoiding a shell does not eliminate argument injection: a tool can interpret an untrusted argument as an option or another control value. Bound and validate each argument under the chosen program’s contract. Paths, URLs, templates and Markdown have their own interpretation rules, so one generic escaping function cannot establish safety across all downstream contexts.

BUTF8(s)=∑i=1nb(ci),1≤b(ci)≤4B_{\rm UTF8}(s)=\sum_{i=1}^{n}b(c_i),\qquad 1\leq b(c_i)\leq4
For a sequence of valid Unicode scalar values c_i encoded as UTF-8, b(c_i) is bytes per scalar and B_UTF8 is encoded bytes. This excludes JSON escaping/container overhead. Character/scalar counts and byte limits differ; runtime string code units require their own explicit counting convention.↔ Scroll wide equations horizontally.

Use harmless canaries to observe the actual consumer

Test known-safe data containing punctuation, markup-like text, option-like strings and unauthorized target IDs against the exact downstream interface in an isolated fixture. Verify the expected rendered text, bound query values, argument vector or denied operation, not just the model’s answer. Use a stubbed model output when testing deterministic enforcement so a model’s refusal cannot hide a missing application check. Separately measure model behavior on representative prompts without interpreting that behavior as proof of the consumer boundary.

The network lab is a single sigmoid scorer, not an authorization engine. Reset and set both x=3,w=3, predict σ(8.5)≈0.999797, then Play or Step. Reset and set both x=−3,w=3 for σ(−9.5)≈0.00007485. The fixed bias is −0.5. A high score does not grant a principal permission to execute a proposed operation. The instrument trains no model and implements no JSON parser, HTML/SQL/shell interpreter, credential or policy decision.

Enforce an encoded output budget

A fictional output field contains exactly 4,000 Unicode euro-sign scalars, each encoded as three UTF-8 bytes. The field’s encoded-byte policy is 8,192 bytes, exclusive of JSON overhead. Its schema separately permits up to 4,096 scalars. Evaluate both stated checks.

  1. The scalar count is 4,000, so this field passes the 4,096-scalar length condition under the specified counting convention.

  2. Its encoded payload is 4000×3=12,000 bytes. That exceeds the independent 8,192-byte field cap by 3,808 bytes; passing the scalar condition does not override it.

  3. Reject or apply the declared bounded-output policy before the field enters its downstream consumer. Even an output passing both size checks would still require context-specific interpretation and any action/resource authorization.

The result

The output passes one length contract and fails another. Neither size check makes generated text or a structured action trusted interpreter input.

Explore the model

Interactive lab

Interactive lab · network

Keep a toy model score separate from authority

Reset → set both x=3,w=3 predicts σ(8.5)≈0.999797; Reset → set both x=−3,w=3 predicts σ(−9.5)≈0.00007485. Play/Step displays one fixed-bias sigmoid. It is not a trained authorization model, output validator, downstream interpreter or permission check.

Keep a toy model score separate from authorityOne inspectable sigmoid unit, σ(wx − 0.5). Playback traces the computation graph. There is no training, attention mechanism or hidden deep network. Current readings: Product · wx: 1.2; Logit · wx − 0.5: 0.7; Sigmoid output: 0.6682. All parameters are adjustable with the labeled controls below.One input. One weight. One sigmoid.x1.20× w1.20+ b0.70σ0.668output0.668w = 1.00b = −0.50σ(z) = 1 / (1 + exp(−z))
Computation graphSignal trace
Product · wx
1.2
Logit · wx − 0.5
0.7
Sigmoid output
0.6682
t = 0 s

One inspectable sigmoid unit, σ(wx − 0.5). Playback traces the computation graph. There is no training, attention mechanism or hidden deep network. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA confident numerical output cannot confer authority to a generated operation.

Check your understanding

A generated SQL request is valid JSON and all user values use bound parameters. Which statement remains necessary?

Try it yourself

A field has 3,000 valid Unicode scalars requiring four UTF-8 bytes each, while its encoded-byte cap is 10,000. What is the payload and overage? Would merely escaping it for HTML make it acceptable to a SQL or shell consumer?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Count bytes before any additional representation overhead, then keep consumer contexts distinct.

Compare with the solution

The payload is 3000×4=12,000 bytes, exceeding the cap by 2,000. HTML escaping neither fixes that byte-budget failure nor creates a general SQL or shell data boundary. Each consumer needs its own correct API and validation, with authorization for any action and resource. Extra escaping can also change encoded length, so its representation needs separate bounded handling.

Take it with you

  • Well-formed model output remains untrusted at the next interpreter boundary.
  • Use consumer-specific data interfaces and independent operation/resource authorization.
  • Measure output units and effects directly instead of trusting model scores or refusals.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.