Enterprise-grade AI agents operate inside tightly integrated systems. They coordinate tasks, use tools, call APIs, manage memory, and interact with external data sources.
These capabilities let agents take on more useful workloads, yet they also create openings where errors or misuse can spread. A mistake in one component can influence others, sometimes without clear visibility, and the effects may show up much later.
Most safety tooling focuses on the model itself. It tests prompt responses and checks for harmful completions.
This helps at the model level, though it misses how agents behave once they’re part of a real system. Failures often happen during tool execution or memory updates, along with other dynamic steps that isolated prompt tests never touch.
To address this, NVIDIA and Lakera AI built a framework that treats the agent as a full system. It introduces ways to simulate threats inside workflows and apply safeguards in areas where failure would matter most.
The framework has been validated through AI-Q, NVIDIA’s internal research assistant, backed by a dataset of more than 10,000 annotated agent runs drawn from real-world usage.
Why It Matters: While model alignment matters, it doesn’t ensure safe behavior once agents work with tools, memory, and external inputs. These interactions introduce risks shaped by system design and task structure, many of which go unnoticed during testing and appear only after deployment. This system tackles those risks at their source by using live workflow monitoring alongside targeted threat simulation. By evaluating agents during execution, it helps teams spot weak points early and keep system behavior steady over time.
- Compositional Risk Taxonomy: The architecture introduces a system-level view of risk that looks at how failures emerge through interactions between components rather than within models alone. It groups issues such as tool misuse and memory corruption with longer failure patterns that unfold across multiple steps. Each risk is scored by severity and by how visible it is during execution, which helps teams focus on the threats that are both influential and hard to notice. This grounds mitigation in how agents behave in real production settings.
- Embedded Framework Architecture: Safety mechanisms sit directly inside the workflow. A global agent maintains policy awareness across the system, while local agents step in at critical points to probe behavior and apply defenses in real time. They run throughout development and deployment, giving teams a clear view of how agents handle pressure. The system adapts to context by considering factors like input type and execution history, which keeps overhead low while strengthening coverage where it counts.
- Agent Red Teaming via Probes (ARP): ARP injects threats directly into vulnerable points within the agent’s graph, revealing how failures unfold inside the workflow rather than at the surface. By introducing faults along key connections, testing on NVIDIA’s AI-Q uncovered issues such as data leaks through markdown handling and unsafe tool use triggered by manipulated retrieval. This gives teams a precise view of where defenses falter and whether a mitigation holds up in real operational conditions.
- Empirical Testing at Scale: More than 2,300 targeted runs across 22 threat scenarios showed that user inputs were far more likely to slip through workflows than content coming from RAG or APIs. In response, the team introduced compact, low-latency defenses that brought the attack success rate down from 24% to 3.7% while keeping quality and speed steady. The results show that well-placed safeguards can meaningfully reduce risk without hindering usability.
- Contextual and Targeted Defenses: Defenses are applied where agents are most prone to fail, mainly within tool calls and risky decision points. They rely on measures such as schema validation paired with permission controls, and each action is fully logged with metrics that reveal how threats move through the system and where safeguards hold. This gives teams a clear way to audit, adjust, and strengthen agents based on live behavior rather than static policy.
Go Deeper -> A Safety and Security Framework for Real-World Agentic Systems – NVIDIA
Trusted insights for technology leaders
Our readers are CIOs, CTOs, and senior IT executives who rely on The National CIO Review for smart, curated takes on the trends shaping the enterprise, from GenAI to cybersecurity and beyond.
Subscribe to our 4x a week newsletter to keep up with the insights that matter.
