Introducing inference hooks with Datadog AI Guard
Frank Aloia
Dust’s inference hooks connect AI Guard evaluations to agent execution, so teams can monitor or gate interactions at configured checkpoints.
Companies are increasingly using AI agents in critical workflows that depend on internal data and business systems. Teams need to evaluate those interactions and decide how to respond when one raises a security concern.
Dust is introducing inference hooks in private beta, with Datadog AI Guard as the first third-party integration. While an agent works, Dust sends Datadog AI Guard the agent content and proposed actions. This enables continuous identification of risks such as prompt injection, jailbreaking, tool misuse, and sensitive data exfiltration.
Administrators can enable this feature for their entire workspace. They can optionally choose to block the agent interaction entirely if a risk is identified.
Evaluate risks as an agent works
External factors could lead to unintended agent behavior. For example, a web search could introduce instructions that were not in the user's request. This can happen at any point in the agent conversation. In order to give administrators full control over their security evaluations, we introduce configurable evaluation checkpoints:
- Check the user's request before the agent starts working on it.
- Check which tools the agent plans to use and what information it plans to send before those calls run.
- Check the results returned by tools before the agent uses them to continue.
- Check the completed answer when the agent finishes responding.
More context can help Datadog AI Guard evaluate security risks. It also increases the amount of information being evaluated. We let administrators choose how much of the conversation is sent at each checkpoint:
- Send the full conversation including all earlier exchanges still available to the model
- Send only new content being checked without the surrounding conversation history
- Send only the user messages which can contain attachments and supporting context
- Send only messages from the current turn which includes the latest user request and all the steps the agent has taken to answer it
“A useful security signal needs to connect to the input that triggered it. Dust inference hooks let teams route selected agent inputs to AI Guard and review its findings as part of the workflow.” - Vijay George, AI Guard Product Leader
Customize the agent behavior
Administrators choose how Dust responds to each evaluation.
In monitor mode, the agent continues working while the check runs in the background. Dust records the result so teams can review findings without interrupting users.
In block mode, Dust waits for AI Guard's result before moving past the selected checkpoint. If AI Guard identifies a risk, Dust stops the agent interaction.
Start with a controlled rollout
As a prerequisite, you will need AI Guard access in your Datadog organization.
Once private beta access is enabled in Dust, an administrator can connect AI Guard in Dust's security settings by using Datadog API and application keys. Hooks apply across the workspace by default, with group exclusions available where needed. Enabling the integration sends conversation content to Datadog for evaluation. Review the data sent to Datadog and the applicable billing before enabling checks on sensitive workflows.
We recommend that you start in monitor mode to evaluate the impact to your current agent interactions. Dust's security view shows flagged, blocked, and error evaluations. In addition, Dust sends the trace of each evaluation to Datadog, so administrators can investigate in either platform.
“Customers bring different security requirements to their AI workflows. By connecting AI Guard to Dust, we’re giving joint customers a controlled way to evaluate selected inputs and learn how the integration fits before they expand its use.” - Uday Tennety, Head of Technology Partnerships
To enable the private beta, reach out to the Dust team directly or email [email protected]. We'll work with you on setup and a configuration suited to your security requirements and expected usage.