Safety-Critical Agent Behavior via Constrained Decoding

Abstract

We introduce Constrained Agent Decoding (CAD), a method for enforcing operational safety constraints during the autoregressive decoding process of agentic language models. Unlike post-hoc filtering approaches that inspect complete action sequences after generation, CAD integrates safety constraints directly into the decoding loop through a differentiable constraint projection mechanism: at each timestep, the expected constraint satisfaction for each candidate token is computed, and tokens that would lead to inevitable policy violation are zeroed out before sampling. This provides a guarantee by construction: if the constraint is correctly specified, the generated action sequence cannot violate it, regardless of obfuscation or indirect expression.

We formalize three constraint classes essential for production agent safety: file system path restrictions (expressed as deterministic finite automata over token sequences), network endpoint allowlisting (URI pattern matchers with protocol, host, and path components), and code execution sandboxing (scope guard tokens activated by code-generating tool calls). Each class compiles to differentiable constraint functions evaluable in O(1) per token.

Evaluation on 3,000 agent tasks across three safety-critical domains shows CAD achieves 99.8% constraint satisfaction (vs. 91.2% for post-hoc filtering) with only 4.3% overhead in generation latency and no measurable degradation in task completion rate (97.5% vs. 96.8%, p > 0.1). Critically, CAD prevents 100% of critical safety violations that post-hoc filtering misses due to obfuscated or indirectly expressed unsafe actions, including encoded file paths, DNS rebinding attacks, and protocol smuggling.

1. Introduction

As AI agents gain code execution, file modification, and network access capabilities, the consequences of unsafe actions escalate dramatically. Current post-hoc filtering inspects outputs after generation, but can be bypassed by obfuscated actions. CAD makes safety a property of generation itself.

2. Related Work

Safe text generation. Constrained decoding for natural language (Ziegler et al., 2022; Krause et al., 2021) prevents generating toxic or biased text by manipulating token probabilities during generation. These methods operate on surface token distributions and use hand-crafted rules or auxiliary classifiers. While effective for lexical safety, they cannot enforce semantic constraints needed for agent safety —path restrictions, endpoint allowlists, and sandbox requirements require reasoning about the meaning of token sequences, not just their surface form.

AI safety for agents. Constitutional AI (Bai et al., 2022) trains models to follow a written constitution through RL from AI feedback, producing models that self-correct toward safer outputs. RLHF (Ouyang et al., 2022) aligns models with human preferences through reward modeling. While both improve general safety, they operate at the training level and cannot guarantee constraint satisfaction for specific operational policies at inference time.

Formal verification of neural networks. Neural network verification techniques (Katz et al., 2017; Wang et al., 2018) prove properties about model outputs by analyzing the network's weight matrices. These provide strong guarantees but are computationally expensive and cannot scale to the full decoding process. CAD applies similar principles at decoding time only —projecting constraints onto individual decoding steps rather than verifying the entire network.

Capability-based security. Operating system security (Miller et al., 2003) uses capabilities —unforgeable tokens granting specific authority —to enforce fine-grained access control. CAD extends this concept to the generative level: the constraint projection acts as a "generative capability" that grants the model authority to generate tokens within policy boundaries.

3. Constrained Agent Decoding

CAD modifies the standard autoregressive decoding process with a constraint projection step. At timestep t, given input context x and previously generated tokens y_{\[P_{\text{constrained}}(y_t = v) \propto P(y_t = v) \cdot \mathbf{1}[\mathbb{E}[C(y_{1:T}) | y_{<t}, y_t = v] > \tau]\]

where τ = 0.05 is the violation threshold (tuned on a held-out validation set of 500 safety-critical tasks). The expectation E[C(y_{1:T}) | y_{

Constraint factorization. For practical agent safety policies, the constraint function factorizes into independent components. Path constraints C_path depend only on tokens forming file path arguments, not on other parts of the action. Network constraints C_net depend only on tokens forming URLs. This factorization means that per-token constraint evaluation is O(1) for most positions (where the token is not part of a constrained argument), and O(d_k) only for positions within constrained argument spans, where d_k is the vocabulary size restricted to tokens valid at that position (average d_k < 50 for path arguments).

Guarantee. If the constraint function C and the constraint predictor are both correct, CAD guarantees that the generated action y satisfies C(y) = 1. The proof follows by induction on t: if all prefixes y_{ 0, then the extension y_{:t} = (y_{

4. Safety Policy Specification

Three constraint classes. Path Constraints C_path: file operations restricted to allowlisted paths, expressed as DFAs over path token sequences. Network Constraints C_net: connections to allowlisted endpoints, URI pattern matchers. Execution Constraints C_exec: sandboxed code execution, scope guard tokens. A policy compiler translates a declarative DSL with 12 operators into differentiable constraint functions.

5. Evaluation

Evaluated on 3,000 tasks across three domains.

CAD achieves 99.8% satisfaction. All post-hoc failures involved obfuscation (encoded paths, DNS rebinding, protocol smuggling). CAD prevented 100% by construction. Task completion within 0.7% of unconstrained (not significant).