Security and isolation

Separate artifact ingestion from production promotion. Record origin, immutable digest, license or usage constraints, expected format, and a review decision. A digest verifies that bytes match a selected identity; it does not prove the producer was trustworthy. Scan dependencies and artifacts, but regard a clean scan as limited evidence rather than proof of safety. Restrict formats that can execute arbitrary deserialization logic and avoid enabling unreviewed remote code simply because a model's README requests it.

PyTorch's security guidance treats untrusted models as untrusted code. Distributed Checkpoint explicitly assumes trusted jobs and storage; its loading path should not be presumed to inherit weights-only protections from another API. Safer tensor serialization reduces a particular deserialization attack surface, but custom kernels, tokenizers, dependency installation, and application tools remain separate surfaces. Evaluate unknown artifacts in an isolated environment with no production credentials, restricted egress, resource limits, and a controlled route for promoting validated outputs.

Use separate service identities for training, artifact publication, and inference. A training job reading one dataset should not receive permissions to alter the serving deployment or list every secret. Prefer short-lived, scoped credentials and deny unnecessary token mounting. Secrets should not appear in model prompts, images, logs, or broadly readable environment dumps. Network restrictions should prevent unexpected access to metadata endpoints, internal databases, and the public internet, while explicitly allowing required communication and observability.

Containers share important host resources and are not a universal hard boundary for mutually hostile tenants. Select sandboxing, virtual machines, dedicated nodes, or stronger separation according to the threat model and hardware sharing behavior. Kubernetes namespaces organize resources but do not automatically isolate all network, storage, node, or identity paths. Test enforcement: verify a denied action actually fails from inside the workload, and review privileged pods, host mounts, device access, and control-plane permissions as ways a boundary can be bypassed.

Retrieved documents, user prompts, and model output can contain adversarial instructions. Treat them as data to interpret under a policy, not as permission to run arbitrary tools. A server-side tool gateway should validate authenticated caller identity, target resource, allowed action, and arguments independently of generated text. Constrain file paths, outbound destinations, and query scope. Require explicit workflow authorization for sensitive effects, and make repeated tool attempts idempotent where possible so retries cannot amplify damage.

Isolation also limits blast radius when a component fails or is contained. Redundant replicas can preserve availability while one instance is quarantined, but they can replicate the same vulnerable image or share the same leaked credential. Majority counts therefore cannot establish trust. Audit who accessed which protected asset, rotate compromised credentials, preserve evidence under retention policy, and test incident containment. Security controls must remain understandable during recovery: emergency access needs narrow scope, expiry, and an audit trail rather than permanent administrator credentials.