Abuse detection and rate controls

Define charging units that track the constrained resource: input and output tokens, accelerator time, tool invocations, or a documented combination. A request-count limit can be a first boundary, but equal requests need not have equal cost. Apply limits at appropriate layers such as credential, tenant, endpoint, and overall service. Shared IP addresses can represent many legitimate users, while one actor may control many identities, so no single key solves attribution.

A token bucket accumulates credit at rate r up to capacity b; an admitted request consumes credit. Its burst allowance lets short demand spikes pass without permitting an unlimited long-run rate. Distributed replicas need a clear consistency and overshoot policy for shared limits. Independent local buckets multiply aggregate allowance unless deliberately partitioned. Admission should occur before expensive work where possible, with cancellation and maximum output budgets constraining work already in flight.

A classifier score is not an authenticated identity or a universal abuse probability. Validate it on relevant populations, inspect calibration, and measure false positives alongside recall. Base rates matter: even a low false-positive rate can create substantial review work when legitimate traffic dominates. Use proportionate actions such as additional review or a constrained path where appropriate, instead of treating every uncertain score as proof of misconduct.

Keep a way to investigate and correct erroneous decisions, and monitor effects across user groups and changing traffic. Minimize retained sensitive content and define access and deletion policies for review evidence. Adaptive behavior can shift the distribution after deployment, so a fixed benchmark result has a limited shelf life. Rate controls can slow repeated probing, but they do not establish confidentiality, prevent all misuse, or replace authorization.

HTTP 429 communicates rate limiting and can include Retry-After. Clients should respect server guidance and use bounded retries with jitter where retries are appropriate. A rejected generation request must not automatically launch multiple parallel replacements. Distinguish temporary capacity rejection from a persistent policy denial so client libraries do not repeatedly retry an action that will remain forbidden.

Track admitted work, rejected work, enforcement latency, budget overshoot, and completed useful responses. Under severe overload, even generating detailed rejection responses costs resources, so infrastructure may need earlier connection-level protection. Test limiter-store failure policy deliberately: failing open protects availability but may exhaust compute; failing closed preserves a budget while denying legitimate service. State the chosen tradeoff and provide a constrained recovery path rather than letting it emerge accidentally.