Who’s accountable when AI agents get it wrong?

Who’s accountable when AI agents get it wrong?

Why AI agent accountability starts with observability, and how to mitigate risk with AI governance

November 16, 2026
Last modified on

So you’ve started deploying agents in your organization. Maybe they’re automating tedious workflows or even providing external customer support. But are you prepared for what happens when one fails? Say an agent makes a change it shouldn’t have or acts without the right permissions. Who responds? Who’s accountable? And how do you prevent it from happening again?‍

Those are the questions we help answer in episode 10 of our Data & AI Weekly series, “When AI Governance Fails: Who’s Accountable When Agents Get It Wrong.” Daniel Takabayashi, Staff Forward Deployed Engineer at Redpanda, works directly with customers deploying agentic systems. In this episode, he chats about where and why these projects fail, why accountability should be discussed in the planning stage of an AI project, and how to mitigate risk with proper AI governance.

Read on for highlights from the talk, or watch the full episode to catch all of Daniel’s insights.

Why are teams being asked about AI governance now?

By 2028, more than 50% of failed AI projects will occur not from model performance problems but from weak accountability, according to a recent Gartner report titled When AI Governance Fails, Revenue and Executives Take the Hit.

AI governance is quickly shifting from an optional initiative to a must-have. Procurement and security reviews are now starting to test for AI governance the same way they do SOC II compliance. Companies are being pressed to meet the regulatory requirements of the EU AI Act, whether they’re in the EU or not—much like GDPR started impacting organizations globally.

That means an annual review and a spreadsheet inventory of your AI systems won’t cut it. As Daniel points out, the more complex your agentic systems become, the harder they will be to observe and the harder it will be to mitigate damage. 

What does weak AI accountability look like?

Over the years, AI project worries have shifted from “Is it good enough for production?” to “Do we have the skills to train it?” and then to “Is it scalable?” 

Now the open question is observability: how do you monitor nondeterministic agents when you don’t know the end result? According to Daniel, too many companies forget to map risk and accountability at the start. When something goes wrong, suddenly people are pointing fingers and asking questions like:

  • Who’s in charge?
  • Who should respond to the problem?
  • How do we stop this from happening again?

Gartner outlines this scenario in the report: a customer flags unfair pricing from an AI-driven quote engine, legal asks who approved it, sales asks whether quotes can still go out, and the board asks why no one ran a risk assessment.

Daniel shares best practices for implementing AI accountability in the Data & AI Weekly episode, including when and how to map risks. 

Why risk compounds as AI projects grow more complex 

Daniel shares several use cases based on real customer projects to show how quickly complexity (and risk) snowballs when an organization deploys multiple agents and AI tools.

His examples start with a relatively simple use case (an IT agent) and then demonstrate how a company could continue building a complex web of agents based on that original use case. Here’s a quick diagram of how the IT agent could function:

As a company continues developing its network of agents, these straightforward observability requirements quickly shift to a complex web of systems, agents, and permissions. Accountability becomes much harder to apply after the fact, which is why Daniel recommends discussing these topics early in your AI projects.

Below is a summary of the use cases Daniel discusses and their expanding skills, integrations, and observability requirements. (Check out the full Data & AI Weekly episode for a more detailed walkthrough of each use case.) 

Use caseHow it worksSkills & integrationsObservability requirements
Simple IT assist agentInternal agent helps employees open tickets and search catalogsA few skills map to a tool set via the ServiceNow MCP serverUser permissions, skills, tools
Onboarding agentTicketing system for hiring managers and new employees; reuses IT agentA few more skills connect ServiceNow to Workday employee profiles and company policy docsAgent-to-agent calls, user permissions, skills, tools
Agent registryCentralized repository to review all details of active agentsWide set of skills to track all agents, integrated systems, and usersLive roster of all active agents, agent-to-agent calls, tools
Alert triage pipelineHeadless system that captures events, categorizes them, and directs them to agentsComplex set of skills to monitor, tag, and filter events as well as run tool and agent-to-agent callsAgent-to-agent calls, tools, user permissions, skills, inputs and outputs

Where should you start?

Regardless of where you’re at in your agentic journey, you should prioritize assessing observability needs if you haven’t already. Daniel suggested two high-priority action items to focus on:

  1. Centralize observability so you can monitor, observe, and control the whole agentic pipeline in one place. A single AI governance platform makes it much easier for engineers to build, troubleshoot, and deploy agents while overseeing observability and accountability.
  2. Map risks at the start of your project alongside your success metrics. Teams are used to defining what success looks like, but often forget that risk assessment and mitigation are just as impactful. 

Make accountability part of your build with Redpanda Agentic Data Plane

Models are rarely the weak link. The bigger potential problem is whether you know what your agents are doing, what systems they touch, and who owns the outcome when something goes wrong. Get observability in place first and name your owners and risks early, so the controls that come after (alerts, evals, kill switches) have something to stand on.

If you're looking to put that into practice, Redpanda Agentic Data Plane provides a single governed gateway between your apps and agents, with built-in identity, auditing, scope limits, and kill switches. The best part is it works with any model, API, database, and cloud. See the Agentic Data Plane in action.

Watch the full Data & AI Weekly episode for more

If your team is facing post-deployment AI observability challenges, watch the full episode on When AI Governance Fails: Who’s Accountable When Agents Get It Wrong to learn:

  • What weak AI observability looks like day to day, and why it has to come before evals, kill switches, and self-healing
  • How complexity builds as agents get reused, with real-world use cases
  • What to define before you start building: success metrics, risks, and the people accountable for them
  • Why centralized AI governance is crucial as your agentic projects expand

If you’re looking for a unified AI governance platform for enterprise agents, ask our team for a custom walkthrough of Redpanda Agentic Data Plane. 

No items found.

Related articles

View all posts
Tyler Akidau
,
Marc Millstone
,
&
Nov 3, 2026

6 questions to ask before choosing an AI agent governance platform

Make sure you’re buying actual agent governance, not lock-in

Read more
Text Link
Kristin Crosier
,
,
&
Oct 20, 2026

Why agentic AI pilots stall before production

Why the demo worked but the rollout didn’t—and what you need to successfully scale agents

Read more
Text Link
Marc Millstone
,
,
&
Sep 28, 2026

A model-based AI kill switch: an exercise in performance art

Good intentions don’t work. Mechanisms do.

Read more
Text Link
PANDA MAIL

Stay in the loop

Subscribe to our VIP (very important panda) mailing list to pounce on the latest blogs, surprise announcements, and community events!
Opt out anytime.