8 engineering lessons on running AI agents in production

If you're the one being asked to put agents into production, you can attest that the demo is the easy part. The model does something impressive in a notebook, everyone gets excited, and then you have to connect it to real data with real permissions and operators depending on the output. The problems you hit at that stage have less to do with the model you picked than with how your systems and processes were built long before agents showed up.

On a recent Data and AI Weekly episode, Redpanda Chief Marketing Officer Melissa Czapiga spoke with two people who worked on those problems back when agents weren't an option. 

Brian Beyer is co-founder and CEO of Red Canary, now a Zscaler company, and Joe Moles was CTO there before moving to furl, where he's building a platform for autonomous remediation on endpoints. Between them, they built a platform that processed petabytes of security data a day with a relatively lean engineering team, then layered agents on top of it.

This post lays out the lessons from that build, along with the order they did things in, which matters more than any individual decision.

If you’d rather watch the full talk, you can below. If not, keep reading.

1. Start from the outcome, not from the data

Red Canary never set out to build a big data processing system. As Brian puts it, the genesis was a more specific question: how organizations were being harmed by attackers and how you'd go about finding them, which meant asking what evidence an attacker leaves behind.

From there, the team worked backward. Once they understood how adversaries operate, they could ask where they might be caught along the way, and only then what data that would require. The answer was that no narrower dataset would do the job, so they'd have to look at everything, which is how they ended up with a big data problem to solve. The scale followed from the goal rather than the other way around.

Joe approached it from the operations side and describes a similar process: start with the end state and map it backward so you can find the decision points and work out what information you need at each one. Once you have that picture, you can ask a more useful question, which is whether a given step can be done in software and how far you can optimize toward a software-first approach. Where a human decision or validation mattered most, that decision point stayed with a person, and the workflow was built to support it. That collaboration between operations and engineering is why agents were straightforward to introduce later, since by the time AI arrived it was another layer applied to a process the team already understood.

2. Build complex systems out of simple components

If there's one architectural principle Brian credits above the others, it's this:

"We want to build really complex systems made up of very simple components." - An early Netflix architect

To put this into real-world terms, Brian walked through the data flow architecture at Red Canary, where each box has a single job.

The payoff shows up where the work gets routed. Telemetry lands through ingestors, gets shaped and standardized, runs through the detection engines and tuning, and then routes to either an agentic investigation or an automated one, with a detection engineer picking it up when needed. Each of those is its own component with its own job, so the pipeline feeding them doesn't need to know which one handles a given case. That means you can change what happens at that routing point, or add a path to it, without decoupling a tangle of data processing components and breaking things downstream.

Decoupling this way also makes agent experiments cheap enough to run. Brian describes how adding one worked: if someone had an idea for finding threats with some new agentic technology, you'd make them a new box, subscribe it to the standardized data, and put a filter in front so it only saw a sample, without touching the rest of the system. If you trusted the results, it went into the main pipeline, and if you didn't, it ran through a QA process first. In many cases, they managed that without affecting the core production system at all.

Brian walks through the full diagram and the components behind each box in the session, so watch the recording if you want the details.

3. Deterministic first, agentic second

Joe's way of deciding where agents belong starts from the opposite direction: ask where the work is already deterministic. If you can write a signature, or the logic is reliably A plus B equals C, write it in software, because it's cheap and fast and works without the overhead of a deeper stack. That was the first pass at Red Canary, and it mostly meant finding tasks done repeatedly and handing them to software.

Agents became interesting at the next tier up, where the work wasn't purely deterministic but was still well understood. Joe describes that as knowing the questions you want asked and the concepts you need to bring together, without being able to write it out as a script or flowchart. 

For a particular data source and class of behavior, you might have 20 or 30 questions you know you want asked, and at that point you have something you can prompt for: go investigate this endpoint data for a certain behavior, ask these questions, bring back a summary.

Worth noting that using an agent doesn't mean the model does the whole task. You're giving direction in natural language, but you're also giving the agent tools and supporting information. Done well, the agent is a fairly small footprint coordinating a set of tools around it, which is cheaper and easier to reason about than asking a model to do everything.

"Instead of saying AI, think about it as natural language automation." - Joe Moles, CTO, furl

4. Onboard agents the way you onboard employees

The language both guests use for agents is the language you'd use for people. Each agent at Red Canary had what Joe calls a baseball card, laying out its tasks and what it needed to do. New agents got day one treatment with heavy oversight and coaching based on what they produced, and as an agent improved, that relaxed into periodic checks and sampling, mostly to confirm it hadn't gone off the rails and that nothing in the environment had shifted enough to need an update.

The failure mode Joe sees most often is the opposite approach, where someone connects to whatever agent harness is handy, throws in all their data, and tries to solve the whole thing at once. In his words, it leads to sadness, and the alternative is the same incremental process development you'd apply anywhere else: start with something you understand well, where you know what goes in and what good output looks like.

Brian argues this kind of decomposition is easier with agents than with people, partly because agents don't have feelings and don't talk back, and they follow instructions reasonably well. You can tell a QA agent to check its work every time, and if it turns out to be weak at one type of analysis, you can split the job so one agent handles analysis A and another handles analysis B. That lets you break a process apart in ways you can't with a human team, though you give up much of the creativity you'd get from people, and there's a harder dependency underneath.

Read our CTO’s blog, “Agents are in the workforce. Now we need HR.” 

5. Measure them the same way, too

The personnel framing extends to how you evaluate an agent. Joe's starting questions are the ones you'd ask about any specific job: what does performance mean here, what are the KPIs, and what outputs should this produce? From there you decide what the measurable state actually is, whether that's cost, speed, or something else that fits the task.

He describes it as giving the agent a performance review, checking whether it has done this many things in this way, and treating the cost and management overhead of running it as a KPI in its own right. Then you keep building on that until you reach whatever milestone you were aiming at, which is the same loop you'd run with a person in a new role.

6. The dependency: human-trained data and human judgment

Brian says if the team tried to build something like Red Canary today by pointing agents at the problem, it wouldn't work. Agents worked there because they had a deep bench of experienced security operators who'd already spent a decade turning their own judgment into software, so decomposing those brains into agents was a short step from where they already were. Hand the same instruction to a team without that background, and you get something different.

"It would be the equivalent of telling a bunch of kindergarten-style ChatGPTs to go do security operations stuff."- Brian Beyer, co-founder and CEO, Red Canary

As he puts it, about 10% of the time you'd be impressed it found something, and the other 90% you'd be telling it it shouldn't be doing this job.

Joe adds the “data half” of the argument. What made tuning fast was a large body of human-trained data, meaning years of recorded decisions from experts who knew the work, which they could use as exemplars. That let them point an agent at a set they knew was clean and say, “these are good decisions and good examples, and here's the data to use for future decisions,” while keeping it away from data they didn't trust enough to decide from. Hand an agent a pile of data that isn't clean and whose provenance nobody is sure about, and it'll learn from a bad set.

7. Access control: treat agents like new hires

On the subject of what agents should be allowed to touch, Brian offers a test that's easy to run anywhere: ask whether you'd give a newly hired engineer or data analyst access to all of your production data. 

If the answer is yes, you can probably build agents on the same assumption. If it's no, then you shouldn't give an agent that access either, and you need to think about how you'd enforce it.

He notes this is an evolution every team goes through anyway. As teams get larger and more international, it gets harder to vet people, and vetting agents from different places gets harder in much the same way.

8. Where to start if you don't have that foundation

Melissa asked the question most teams are sitting with, what if you don't have a decade of documented decisions and clean exemplar data? 

Both guests gave the same answer: write down what you already do.

Brian calls it the only real place to start, and Joe frames it as the old exercise of explaining how to make a peanut butter and jelly sandwich, where the point is to see whether you can describe your own process step by step. His one caveat is that writing this out across a whole set of business processes is an enormous job, so start with a thin slice. “Don't document the whole sandwich,” he says. “Just document putting the peanut butter on the bread, and you'll turn up steps you hadn't thought of, which you can then document as well.”

Draw the picture, and draw it for humans

Two practical points came out of this. The first is that you shouldn't be solving for the agentic system to read your documentation, because agentic systems aren't going to figure your architecture out by themselves right now. They're guided by a human, which means your own team is the audience that matters.

Second, the artifact that works is a diagram like the one above. Brian's recommendation is to pick Google Slides or PowerPoint, use the real icons for whichever cloud provider you run on, and draw how data moves and where the interface boundaries are. He's quick to acknowledge you can autogenerate a picture of what components are able to touch each other, but that isn't the useful part, because you can't autogenerate why they would do that or what the intended data flows are. 

Joe's version of the habit is a whiteboarding app and a standing request to just draw him the picture, showing where the boxes are and where the decisions get made. Once it exists, the conversation changes, because you can walk through it and ask why a box is there or why the flow loops back. You can also point at one transition and say “this part is well understood and well measured,” so this is where the agent goes, or notice the flow is backtracking in ways nobody intended and flatten it first.

TL;DR: 8 lessons for running agents at scale

If you're planning to run agents in production at scale, the sequence that comes out of this conversation looks like this:

1. Work backward from the outcome. Identify the end state, the processes, and the decision points before you decide what data or technology you need.
2. Decouple everything. Simple, single-job components let you add a new box, filter it to a sample of traffic, and try an idea without touching the production path.
3. Find the deterministic work first. If you can write the rule, write the rule, and save agents for work that's well understood but not fully scriptable.
4. Give each agent one job. Define the task, the inputs, and the expected outputs the way you would for a new hire.
5. Measure them like employees. Set KPIs and expected outputs, decide whether performance is cost-based or speed-based or something else measurable, and review against it.
6. Mind the data you train and tune on. Clean, human-decided exemplars are what made tuning fast, so keep untrusted data out of the decision path.
7. Scope access deliberately. Use the new-hire test, and if you wouldn't give a new person the data, don't give it to the agent.
8. Sketch it out. A diagram drawn for humans, showing data flow and interface boundaries, is the foundational artifact, and you should start with one slice of it.

Very little of that is specific to AI, which is roughly how Joe summarizes the approach.

"It's all system building, and an agentic system is just another system." - Joe Moles, CTO, furl

Watch the full session for a deep dive

The full session covers the architecture walkthrough in detail, along with the discussion of where agents sit relative to people and deterministic software. Watch Data and AI Weekly episode 7 for all of it.

If you want to see what this looks like in practice, book a demo of the Redpanda Agentic Data Plane to chat through how to deploy, govern, and scale agentic systems across your own data.

‍