Over the past year, I have had a lot of success in deploying AI agents into my workplace. In the process, I have significantly improved both the quality and quantity of my work output, and I'd like to share a summary of what that looks like. I'm hoping some of it will be helpful to someone.
1: What is "quality"?
I care very much about quality, and so when I say that the quality of my work output has improved dramatically since adopting AI agents into my workflow, I want to define the bar I would need to clear for that statement to be true.
That bar is not perfect output. Before AI agents, I could not promise perfect output, and AI agents have not changed that. I could do my best, I could write tests, I could double and triple check, but my work routinely had errors. I have learned many procedures to catch my own errors, my team has procedures to catch the errors of its members, and we all had procedures to monitor and respond to unexpected changes in the environment. Nevertheless, sometimes mistakes would get through the process, and when they do, we have a postmortem/retrospective process to learn why and try to prevent it from happening again. This is how the sausage is made.
Humans are unreliable agents. The bar we have to clear is not perfect error-free output. What we want is to invent a process that can produce output that is at least as likely to be correct as an extremely skilled human who cares very much about the work they do and gives the task all of the attention it deserves. It would be extremely useful to us if such a process could scale beyond the limited supply of human attention.
That's not what language models do on their own. Language models produce a kind of vaguely correct-shaped output for any question that you can express with words. It turns out that "vaguely correct-shaped output" is good enough to be useful.
This is an unusual way to think about computing. We're accustomed to the idea that when you enter arithmetic into the arithmetic machine, you get the correct answer according to the rules of arithmetic, and you can do it ten billion times and you will get ten billion correct answers, and your next one will be correct too. We are now dealing with a fundamentally different type of machine.
2: What are "Models" and "Agents"?
There's one more stop I want to make before I get to the point. People tend to use "Language Model" and "AI Agent" somewhat interchangeably. I think it's important to draw the distinction between them.
A Language Model is a very large pile of numbers that, when tickled the right way, transforms coherent inputs into coherent outputs. Note that I said coherent, and not correct. That pile of numbers contains relationships between concepts and some general information that usually has the aura of correctness, but the model does not actually contain any details about the task you're working on, or your problem domain, or the present state of the world. You need to somehow provide those things as part of your input.
When you ask a language model for the weather in San Francisco, the best case is it says "I don't have access to that information" and the worst case is it says "it's 75 degrees and sunny". It says this because it's likely to be true, regardless of whether it actually is.
For any given task, you will need information to complete it. It would be quite a lot of work to collect, in advance, all of the information and documentation that you'd need to do the task and then pass it manually as input into the model. In fact, for most tasks, collecting the information is the hard part, and doing it manually would probably be as much work as just doing the task yourself.
At that point, what do we need language models for? It would be quite pointless to ask a language model for the current weather in San Francisco if you had to look up the current weather yourself just so would have the correct answer to give back to you. That's why you need a harness.
An AI Agent is a language model that's being invoked by an Agent Harness. A harness is a type of software application (the traditional kind) that feeds inputs into the model, but when the model returns its output, the harness looks for a specific sequence of text called a tool call. A tool call looks something like this:
<tool_call>
{
"name": "get_weather",
"arguments": {
"location": "San Francisco, CA"
}
}
</tool_call>When you ask an AI Agent (a language model running in a harness) a question, as part of its response it can output this tool call. This triggers a computer program that has the capability to look up the actual weather via the get_weather function (assuming it has such a function) and the answer is sent to the model as a follow-up message so that it can complete its response. Now your mostly-correct-ish language model has access to real and timely information from a known source.
An AI Agent is a language model running in an agent harness.
3: Let's finally talk about security
Now that we have a foundation in what tools we're talking about, let's look at a real task that a security professional might want to do: break the system.
If we want an AI agent to test and try to break a given system, in theory we really only need to give it one tool: execute arbitrary code. I think a reasonable person could imagine that if you give an AI agent the ability to execute arbitrary code, it might break something. Indeed, it would be very useful for us to know if our system can be broken, so we will put a code-executing agent and our system in a cage and let them fight until only one is left standing.
This is incredibly useful! It's a primary objective of security and quality assurance to ensure that the system is resilient to hostile and chaotic threat actors. Now we have a hostile and chaotic threat actor with coin-operated patience and attention! Furthermore, if an agent wins the cage match, we have an activity log that tells us exactly how it did so. We can fix the discovered problems and then put them back in the cage for a rematch. We can do this even if humans are building the system. It's hard to overstate how valuable this is to a security defender.
But how do you know it's thorough? You don't, but if you think the agent missed something, you can throw ten more agents at the problem and tell each one to focus on a specific aspect of the system, or a specific class of vulnerability, or any aspect of attack you can describe in text. Keep throwing agents at it until they consistently come back empty. If you do this, you have already had more adversarial threat assessment attention on your system than you would if you hired a human pentest team for a week. You can keep these agents going continuously if you wanted to.
This starting place may surprise some people. Maybe they thought my first use for AI agents was going to be building something, rather than destroying. But it turns out that part of building something high-quality is considering the myriad ways that it might be destroyed. If we naively turned a construction task over to the agents, we would have no way to know if they were doing a good job, or if the system they're building is robust. But now that we have our cage match framework, we have the ability to apply some selection pressure.
Let's say now we asked an agent to build a login page. Login pages are mission critical to any webapp. They are the security checkpoint at the front door. There is a right way to do them, and many ways to make a mistake that could lead to a security breach. But just for the fun of it, let's let our agent takes a swing at it, and let's say for the sake of argument that it does a bad job.
Let's also say that we have become very lazy and we haven't looked at the code at all. We don't know if it did a good job or a bad job, but we have our army of attack agents and a cage. The attack agents are never given credentials, and so we know that if they are ever able to send an authenticated request, they must have defeated the login page in some way. If they defeat the login page, we can capture the log of how they did it.
That log can become instructions to feed back into the construction agent. We can say "Here is a log of an agent breaking your login page, please reproduce the attack and then fix the problem." Again, we don't know if it will do that correctly, but if it doesn't, our attack agents will eventually find it.
This back-and-forth is a loop that can be trivially automated.
4: Adversarial review loops
One agent tries to accomplish the goal, and the other agent tries to find problems in the first agent's work. This is the fundamental building block for how you get reliable output out of unreliable agents. I have had good results with this technique across a wide variety of tasks, but even if you believe that this technique is only good for software development, software is good for a lot.
You may notice that this work style is starting to look less like engineering and more like management. After all, managers routinely supervise products for which they do not understand all of the details.
It's useful to consider whether one believes that managers provide a useful contribution to a work process, and the answer probably depends a lot on the individual manager and the individual's general disposition toward management. Similarly, I expect that some people will be more predisposed than others to using agents to produce quality output. I do believe at least that it is a skill and a set of techniques that can be learned.
I still do not claim that adversarial review loops, or the larger systems constructed with them, produce perfect output. A system that can consistently produce perfect output at scale would make one vastly wealthy indeed. We can only hope to improve our work quality incrementally, and over the past year we've learned a lot about how to do that with unreliable AI agents. We have a lot more to learn and many more mistakes to make. But I do hope that this explanation shows that they can produce predictable output at scale, and that alone is useful.