The Next AI Governance Challenge: Governing Agent Behavior

As companies begin deploying AI agents at scale, they are creating a new management problem: How do you govern a workforce of digital actors when you may not know exactly which agents are operating, how they are behaving, or when their behavior starts to go off course?

For the past two years, much of the enterprise conversation around AI governance has focused on people: Which employees are using what AI apps and are they approved by IT? Agents change the problem. As they begin to take actions, communicate with other agents, make decisions and operate with increasing degrees of autonomy, a deceptively simple question becomes much harder to answer: Do you actually know what all of your AI agents are doing?

Tony Davis, Chief Innovation Officer at Scout Agentics , believes this is becoming one of the central governance challenges of enterprise AI. His team has been building Cortex, a system designed not simply to monitor whether an agent works, but to examine the behavioral traces underneath what it does.

Most AI monitoring today is still heavily oriented toward performance: Did the agent complete the task and how quickly? Did it hallucinate or did it drift from its assigned task? Davis is interested in a different question: How is this agent behaving?

Performance tells you what happened. Behavior may tell you something else.

During our conversation, Davis drew a distinction between the emerging AI "control tower" model and the behavioral monitoring Scout is exploring. Enterprise platforms are increasingly able to orchestrate large groups of agents and monitor factors such as speed, efficiency, drift and hallucination.

Davis sees another layer becoming necessary: looking beyond whether an agent is functioning to whether its behavior is beginning to shift in concerning ways, such as straying from instructions, failing to explain its reasoning, coordinating inappropriately with other agents, or showing signs of sycophancy, concealment, self-preservation or malicious intent. Cortex analyzes agent traces across a series of factors intended to surface those kinds of patterns. Davis described measures ranging from confusion, trustworthiness and drift to traceability, adherence, transparency and collusion attempts. Some of the governance measures are also designed around elements of ISO 42001. The point isn't that one signal means an agent is dangerous. Agents can get confused just like their human counterparts. They also make mistakes. The more critical point is whether those signals begin to form a pattern.

Who governs the governor?

This led Scout to make an interesting architectural change. Earlier versions of its governance approach relied on a "critic" which essentially was another AI agent evaluating other agents. But that creates a circular problem. If agents themselves can drift, become sycophantic or behave unpredictably, should another probabilistic agent make the final judgment about whether they are behaving properly? Davis eventually concluded that it shouldn't.

Cortex still uses an AI model to analyze an agent's trace and identify things worth examining. But the final scoring is handled by deterministic code rather than another agent. "The real answer was not to let a probabilistic agent do the scoring," Davis told us. This is an important distinction. Davis says the same underlying evidence should therefore receive the same score each time rather than a slightly different judgment depending on the AI evaluator.

This points to a governance issue much larger than Scout. As organizations build oversight systems for AI, they will increasingly have to ask: How much AI should we use to govern AI? At some point, the governance mechanism itself has to become something leaders can trust.

Looking for trajectories, not just failures

Cortex currently produces what Scout calls a rogue index, intended to indicate whether concerning behavioral signals appear to be accumulating. Davis described it less as a binary judgment of safe or unsafe and more as an early-warning approach. A single unusual interaction might mean very little. But suppose subsequent traces begin showing inappropriate coordination with another agent, followed by attempts at concealment or increasingly problematic behavior.

In Davis's view, the goal should be to give humans a chance to intervene before an agent produces an obvious failure, rather than discovering the behavioral pattern afterward. Whether Cortex itself proves to be the answer is less important than the governance principle underneath it. Our current monitoring systems are largely designed to tell us what happened. Agentic systems may require tools that help us understand what appears to be developing.

You can't govern what you can't see

There is an even more basic problem before any of this becomes possible. Companies may not even know where all of their AI is. Scout has been exploring that issue separately through a system called Columbo, which analyzes enterprise logs for evidence of AI usage.

Columbo looks across sources such as DNS records, software inventory and proxy logs to surface AI-related activity across different parts of an organization. If the organization has an approved list of AI tools, that activity can be compared against it. If it doesn't, the exercise can still help create a clearer picture of what is actually being used.

Importantly, unapproved doesn't automatically get flagged as malicious. An employee may simply have discovered a useful AI tool before the company established a process for approving it. But the governance problem remains. One executive in Davis's conversations put it simply: you can't govern what you don't know about.

That may be the first layer of AI governance: What AI is operating inside the organization? Then comes the next question: How is it behaving?

The human may need monitoring too

The conversation with Davis surfaced another dimension that may ultimately be just as important as rogue-agent behavior. Humans can become part of the problem.

We can give agents too little context. Over-trust them. Ask questions that seek confirmation rather than challenge. Accept flattering answers. Or gradually stop scrutinizing recommendations because we have grown accustomed to working with a particular AI.

In other words, the agent may drift toward the human. But the human can also drift toward the agent. Scout has now developed what Davis calls a "human factor" alongside its rogue-agent measure, which is an attempt to evaluate how the person is interacting with the AI, not just how the AI is behaving.

This may be one of the more significant ideas coming out of the work. Imagine an AI governance system that doesn't simply tell IT that an agent behaved badly. It might eventually be able to tell an employee: You routinely give your agent too little context. You appear to be asking it for validation rather than independent analysis. You're accepting recommendations too readily. That begins to look less like surveillance and more like coaching. And it suggests that the future of AI governance isn't simply about governing the machine. It may be about governing, and improving, the human-agent relationship.

A new management layer for AI

It is tempting to think of this simply as another cybersecurity or technology-monitoring problem. I think it is becoming something broader. As agents take on more work, organizations may need something analogous to a management layer for artificial workers.

Not a system that approves every action. That would undermine much of the value of autonomy. Instead, leaders will need mechanisms that can watch across thousands of interactions, identify behavioral shifts and exceptions, and direct human attention to the places where judgment is actually required.

That is what makes Scout's work interesting. Cortex, Columbo and its emerging human-factor measures are tackling different parts of the same problem: knowing what AI is operating inside the enterprise, understanding how it is behaving, and recognizing when the human-agent relationship itself may be creating risk.

It is still early, and Scout will not be the only company trying to solve this. But it is working on a governance problem many organizations have barely begun to define.

We spent the first phase of enterprise AI asking: Are our people using AI appropriately?

The next phase will require companies to ask: Are our AI agents behaving appropriately?

And increasingly: Is the relationship between the human and the agent making both of them better or making them worse?

Scout is betting that answering those questions will require more than traditional monitoring. It will require a new kind of visibility into AI behavior itself. And as companies put more agents to work, that visibility may become less of a nice-to-have and more of a basic requirement for managing them well.

This article was originally published on Forbes.com