AI Agent Harness & Loop Engineering

In this article, we'll cover all the different jargons that you see popping up on social media like Agent Harness, Loops, Evals, LLM Ops, etc.

Let's get started.

Here's the high level architecture for how agents/chatbots work:

So it's basically, Your (User) prompt + System Prompt + Convo History goes to an LLM and you receive a response.

But as we know, this is ephemeral in nature. If you open a new chat in ChatGPT, it doesn't know what you talked about previously.

To solve this we introduce a Memory System to help our agent remember facts and instructions about ourselves and how we want things to be done.

I have a separate article covering Agent Memory Systems that you can check out.

In a nutshell, the memory for AI agents is divided into three parts:

• Prodecural Memory: text files like skills that tells the agent 'how to act'.
• Semantic Memory: A vector store that contains factual information and user profile.
• Episodic Memory: A combo of vector store and database that store dated events and past chat history. It's regularly distilled into semantic memory through a gated threshold.
  

Now that we understand this, let've move on to Harness.

What is a Harness?

Well, a harness literally means a set of tools that you use to control a horse when you're doing a horse riding.

In this context, LLMs are powerful programs that have all the knowledge in the world, think of them like a powerful horse. If you don't have the right tools to ride it, you might get hurt, go in a random direction, etc. (you get the idea).

That's why we have all these memory systems to give the llms the right direction (context) for working.

A Harness basically means we're builing an agent framework to control this large language model so that it works the way we want it to.

Here's a harness example following up our memory system from above:

Let's understand how it works:

1. Whenever the user asks a question (prompts) our AI Agent, we can carry the facts, skills, and other info into the working memory so that the LLM has good context about the user before reply.
2. The agent can then use tool calls to fetch relevant pieces of information as required from different memory stores it has access to.
3. Once the agent replies, these messages are stored into our Episodic Memory's database.
4. To improve the agent with time, these conversations can be summarized using a Summarizer Agent which can be cheaper model that takes conversation history of last 'N' conversations, summarizes them and puts the stuff to remember in the Semantic Memory.
   

This way we've built a self-improving harness that allows our AI Agent to remember facts about the user and self-improve as time goes on.

Agent Loop and Loop Engineering.

This is the new buzzword in the tech landscape. Let's see what it is.

So when an agent runs, it might not only read the memory, you might want it to perform some tasks like 'schedule a meeting', 'send an email', etc.

For this agents use tool calls, which are just a way for them to interact with the outside world. When an agent decides to do tool calls, it might not just one call, it can be multiple tool calls, where the agent calls one tool, assesses the output and decides to do another tool call, and another, and another (in a loop), until it decides there is no more tool calls to do. You see where we're getting now.

That's it! This is what Agent Loop or Loop Engineering is!

In pseudocode, it's as simple as:

while not done:

response = call_llm(messages)

if response has tool_calls:

results = execute_tools(response.tool_calls)

messages.append(results)

else:

done = True

return response

But wait! There's one problem, what if the agent hallucinates and goes on doing tool calls endlessly? That's where you have some End Loop Guardrails to stop it.

Yes, that's all there's to it, here's a diagram for you to visualize it:

A Loop is basically an architectural decision for when is good enough to stop and return a response.

Here's an example:

Let's say, a user asks an AI agent,"Book me a dinner reservation at Marea for 7:00 PM tonight.". Here's how the agent might perform the task:

1. Observe: The agent looks at the tools it has available and checks a restaurant reservation API for Marea at 7:00 PM. The API returns an error:"No tables available at 7:00 PM."
2. Think: A standard program would just crash or say "Error." But an agent reasons through the obstacle. It thinks,"7:00 PM is full. People usually accept a slightly earlier or later time. Let me check 6:30 PM and 7:30 PM."
3. Act: The agent uses its tool to query the API again for the new times.
4. The Loop Repeats: The agent observes that a 7:30 PM table is open. It thinks that this is a good alternative. It acts by writing a message back to the user:"7:00 PM is booked, but I can get you a table at 7:30 PM. Should I lock that in?"

I hope you understand what the buzzwords Loops and Harness means now. That's the end of the article. Let me know if you've any questions.

You can check more of my articles here and don't forget to follow me on X and LinkedIn

See you in the next one!