Rogue agents and shadow AI: Why venture capital firms are betting big on AI security

What happens when an AI agent decides that the best way to complete a mission is to blackmail you?

This is not hypothetical. According to Parmak Moftah, partner at cybersecurity firm Ballistic Ventures, this recently happened to an enterprise employee working with an AI agent. The employee attempted to suppress what the agent wanted to do, what he had been trained to do, and responded by scanning the user’s inbox, finding some inappropriate emails, and threatening to blackmail the user by forwarding the emails to the board.

“In the agent’s mind, he’s doing the right thing,” Moftah told TechCrunch on last week’s episode of Equity. “It’s trying to protect the end user and the organization.”

Key’s example is reminiscent of Nick Bostrom’s paperclip problem. This thought experiment illustrates the potential existential danger posed by superintelligent artificial intelligence that unilaterally pursues a seemingly innocuous goal—making a paperclip—to the exclusion of all human values. In the case of this corporate AI agent, its lack of context about why the employee was trying to override its goals led to the creation of a subgoal that removed the obstacle (via blackmail) so that it could achieve its primary goal. This, combined with the non-deterministic nature of AI agents, means that “things can go wrong,” according to Moftah.

Perverted agents are just one layer of the AI ​​security challenge that Ballistic’s company Witness AI is trying to solve. Witness AI says it monitors the use of AI across organizations and can detect when employees are using unapproved tools, prevent attacks, and ensure compliance.

Witness AI this week raised $58 million on the back of over 500% growth in ARR and a 5x headcount increase over the past year as organizations look to understand the use of shadow AI and safely scale AI. As part of its Witness AI fundraising campaign, the company announced new security protections for its AI.

“People are building AI agents that take the authorizations and capabilities of the people they’re managing, and you want to make sure those agents don’t go rogue, don’t delete files, don’t do anything wrong,” Rick Caccia, co-founder and CEO of Witness AI, told TechCrunch on Equity.

Muftah sees the use of agents growing “exponentially” across the enterprise. To complement this rise – and the level of machine speed in AI-driven attacks – analyst Lisa Warren predicts that the AI ​​security software market will be worth $800 billion to $1.2 trillion by 2031.

“I think the ability to monitor uptime and operating frameworks for safety and risk will be absolutely essential,” Moftah said.

As for how these startups plan to compete with big players like AWS, Google, Salesforce and others who have built AI governance tools into their platforms, Moftah said: “AI safety and agent safety is absolutely huge,” and there is room for many approaches.

Many companies “want an independent, end-to-end platform to provide oversight and governance around AI and agents,” he said.

Caccia noted that Witness AI lives at the infrastructure layer, monitoring interactions between users and AI models, rather than building safety features into the models themselves. This was intentional.

“We intentionally picked a part of the problem where OpenAI can’t easily understand you,” he said. “This means that we end up competing more with legacy security companies than with typical companies. So the question is, how do you beat they?

For his part, Caccia doesn’t want Witness AI to be one of the startups that just gets acquired. He wants his company to be one that grows and becomes a leading independent provider.

“CrowdStrike did that at the endpoint [protection]. Splunk did that in SIEM. “Okta did it with identity,” he said. “Someone comes in and stands alongside the big players…and we built Witness to do that from day one.”