AI Agents: When and How To Implement

With the rise of large language models (LLMs) — and an ecosystem of automatized agents, vector databases and LLM evaluations — the era of LLMOps is upon us. However, as language use cases become simplified and automated, the human process for understanding frameworks, plug-ins and toolkits is getting increasingly difficult. Take agents, for example.

Recap: What Is an AI Agent?

AI agents are used in LLMOps as a reasoning engine to determine which actions to take and in which order to take them. So instead of using an LLM directly, you can use an agent and give it access to a set of tools to deliver a sequence based on your intended task. The full extent of the sequence an agent completes is based on the background, context and tools you provide to the agent. With the addition of AI agents to the LLM space, single and multi-task objectives can be completed on a needs basis or as a continuous process. It’s worth pausing to consider, however: is implementing an agent necessary?

This piece goes through five key takeaways you might consider before implementing an agent, along with an in depth example comparison for simple and complex agents against your current LLM pipeline.

1. You May Not Need An AI Agent

It’s worth restating: you may not need an agent. Since an agent essentially takes a task originally assigned to a LLM and levels it up by adding tooling, depending on the needs and complexity of your use case you may find an agent is not necessary for your assigned task. LLMs can generate human-like text and engage in conversation, but are not designed for end to end high level task completion. Agents in contrast can directly access up to date information from the internet with search tools, generate task lists to accomplish an overall goal, write code to acquire skills or access the most recent company data via a tooling API.

Simply researching articles or interacting with a LLM like Claude 2, Llama 2 or ChatGPT alone may be sufficient for simple, single requests. However, agents offer benefits like memory, planning, and learning over time. Agents could enable chaining multiple uses together or taking an iterative approach to complete complex tasks, but those advantages do come with a cost depending on the amount of use. The choice of implementing an agent requires evaluating the length and longevity of your task, choosing the right additional agent tooling and frameworks that will give you the best result, and an understanding of your available time and resources. You may come to the conclusion that you don’t need an agent, or perhaps by using the right one you can optimize your productivity more than you thought possible.

2. Decide If You Have a Single Task Use Case or a Multi-Task, Iterative Use Case

Simple, general purpose agents that are commonly used in LangChain and LlamaIndex chains are designed for straightforward, end-to-end tasks. They take a chain of tasks, generate prompts and responses using large language models, retrieve documents, leverage tools and finally at the end of the chain, they return the result. This makes them fast and easy to deploy, but limited in scope. Complex agents like BabyAGI and Voyager incorporate more components for long-running tasks. Their architectures may include modules for memory, planning, learning over time, and iterative processing. However, complex agents require more complex testing scenarios and engineering to build and maintain.

In choosing between simple versus complex agents, one must weigh factors like use case complexity, availability of human-in-the-loop requirements, and development time. Simple agents suffice for basic generative tasks, while complex agents excel at planning and generalization over lengthy engagements. Understanding these tradeoffs helps select the right agent architecture for your use case.

3. Consider the Architecture When Choosing An Agent

LangChain vs. LlamaIndex

When you have a straightforward task, frameworks like LlamaIndex and LangChain are great for leveraging agents when developing applications powered by LLMs. These LLM Frameworks can also be used on top of each other to achieve a specific task. For example, LlamaIndex’s agent-like components (automated decisions to help a particular use case) can be used as standalone agents or be combined with other LLM frameworks.

LlamaIndex provides core modules capable of automated reasoning for different use cases over your data. These include the SubQuestionQueryEngine for Multi-Document Analysis, Query Transformations, and Routing. You can find more details about these in the use cases doc.

On the other hand, LlamaIndex can be used as a core Tool within another agent framework. For instance, it can be used within LangChain and ChatGPT. LlamaIndex query engines can be easily packaged as Tools to be used within a LangChain agent, and LlamaIndex can also be used as a memory module / retriever. You can find more details about these in the LangChain integration guide.

Here’s an example of how to use LlamaIndex as a tool within an agent framework:

from llama_index.langchain_helpers.agents import IndexToolConfig, LlamaIndexTool

tool_config = IndexToolConfig(
    query_engine=query_engine, 
    name=f"Vector Index",
    description=f"useful for when you want to answer queries about which agent to use.",
    tool_kwargs={"return_direct": True}
)

tool = LlamaIndexTool.from_tool_config(tool_config)

When using a LangChain agent you must:

1. Choose the prompt
2. Choose the tool
3. Choose the type of agent

Give the agent access to the tool for your use case and describe the tool in a way that is most helpful to the agent. For example, if you want the agent to search the internet you can use SerpAI, Google Search, Bing Search, and the Wikipedia API, or you can use a language model directly with a ChatGPT plugin.

Pseudocode for this runtime is below:

next_action = agent.get_action(...)
while next_action != AgentFinish:
   observation = run(next_action)
   next_action = agent.get_action(..., next_action, observation)
return next_action

The final step is deciding on the type of agent based on your use case. This for example could be a general purpose agent (zero-shot agent), OpenAI functions agent, conversational agent and more. For any typical LangChain agent, the agent executor is needed to execute the actions for the agent, but as you will see later on the execution chain can be upgraded with a complex agent that has access to tools.

Complex, Iterative Agents

Now, let’s get into the complex, iterative agents. The most talked about task-oriented agents are generally BabyAGI, AutoGPT and Voyager as they take a systematic approach to breaking down complex tasks into simpler ones. This combined with generative AI advancements in reasoning (chain of thought) and grounding (by observations in environment, aka Reason+Act or ReAct) abilities is leading to increases in the complexity and accuracy of returns by agents. These are also known as plan and execute agents, which accomplish an objective by first planning what to do, then executing the sub tasks as seen from BabyAGI, Voyager and the “Plan-and-Solve” paper.

BabyAGI

Let’s start with BabyAGI. BabyAGI’s agent has three main steps (execution, task creation, and prioritization) in order to execute an agent’s workflow.

Adapted from source.

• The execution_agent() function
  • Where the OpenAI API is used.
  • It takes two parameters: the objective and the task.
  • It then sends a prompt to OpenAI’s API, which returns the result of the task.
  • The prompt consists of a description of the AI system’s task, the objective, and the task itself. The result is then returned as a string.
• The task_creation_agent() function
  • Where OpenAI’s API is used to create new tasks based on the objective and the result of the previous task.
  • The function takes four parameters: the objective, the result of the previous task, the task description, and the current task list.
  • It then sends a prompt to OpenAI’s API, which returns a list of new tasks as strings. The function then returns the new tasks as a list of dictionaries, where each dictionary contains the name of the task.
• The prioritization_agent() function
  • Where OpenAI’s API is used to reprioritize the task list.
  • The function takes one parameter, the ID of the current task. It sends a prompt to OpenAI’s API, which returns the reprioritized task list as a numbered list.

Voyager

Now, let’s dive into the Voyager agent architecture. Note that Voyager incorporates all of the steps seen in agents we’ve talked about so far, but with the additional steps that prioritize flexibility and continuous use. From the diagram below, there are three key components to Voyager agents: automatic curriculum for open-ended exploration (to adjust to the agent’s strengths), a skill library for increasingly complex behaviors (that can continuously be added to), and an iterative prompting mechanism that uses code as action space (an action is defined as a function in Java code).

There is great novelty in a Voyager agent. Similar to how humans collect skills to increase their productivity and quality of work, a Voyager agent adds additional skills to the skill library. In addition to using their skillsets, humans also incorporate feedback to refine and improve their work, a Voyager agent receives feedback on what to do if an objective is not met via a GPT-4 evaluation. This, as well as feedback on code generation via code interpreter feedback and environmental feedback.

Breaking down these components for Minecraft:

1. Voyager attempts to write a program to achieve a particular goal, using a popular Javascript Minecraft API (Mineflayer). The program is likely incorrect at the first try. The game environment feedback and javascript execution error (if any) help GPT-4 refine the program.
2. Voyager incrementally builds a skill library by storing the successful programs in a vector DB, this establishes an iterative process to store code as memory and build upon what you have learned. Each program can be retrieved by the embedding of its docstring. Complex skills are synthesized by composing simpler skills, which compounds Voyager’s capabilities over time.
3. An automatic curriculum proposes suitable exploration tasks based on the agent’s current skill level & world state, e.g. learn to harvest sand & cactus before iron if it finds itself in a desert rather than a forest.

Context and sources: Dr. Jim Fan, Voyager resources, Voyager code, and the Mineflayer API. 

Note: Though the Voyager prompts are still heavily hand-crafted towards Minecraft, the agent’s process of incorporating LLMs for exploration will be applied to more use cases in the future.

4. Compare AI Agent’s Execution for Yourself

If you are a machine learning engineer deciding which recommendation system to deploy into production, you would test various algorithms against each other and evaluate them based on performance metrics, key performance indicators (KPIs), runtime and cost. Oftentimes the most advanced and highest performing solution doesn’t always make it into production if it’s computationally dense, expensive, difficult to maintain and troubleshoot and is only slightly better performing than a more straightforward model. The same is true when implementing any new technology, you need to make sure that the benefits outweigh the costs. Now let’s compare agents on a specific use case to see if a complex agent outperforms a Google search.

Let’s say that you already have a company use case that is ideal for leveraging an agent:

• Content creation
• Research curation
• Data analysis
• Personal assistance
• Customer success
• User satisfaction
• Finance assessments
• Task management

The use case and prompt that we will be using throughout the rest of this piece will be the following:

• Focus Area: Content creation / Research curation
• Who are you? You work for a non-profit organization titled “Aliens are Real” that is trying to find UFOs and aliens among us and give them a warm welcome to Earth.
• Goal 1: To prove Aliens are real through research, first hand accounts, and evidence of extraterrestrial and/or intradimensional life.
• Action 1: Your first task in providing evidence that aliens exist is to make a todo list to map out your steps in order to prove aliens are real.

I will go ahead and inform you that a Google search does not complete this task, so let’s see how GPT-4 and Claude 2 handle this assignment:

ChatGPT with GPT-4

Claude 2

Those results are not bad for a first pass with only a few seconds to render. Even though not this process is not scalable, iterative, lacks complexity and can become costly, we can use these results as a benchmark before we assign an agent to the task. The simple and complex agent examples I will be showing can be found in this Google Colab Notebook.

We will start by implementing a simple agent that is a single task, zero-shot LangChain agent that uses the Google search API tool SerpAPI to access and connect with Google search results programmatically.

Notice this thought process. One important distinction is using an agent vs Claude 2 is the chain of thought reasoning via “observations” and “thoughts” that is available to show the user how the agent went from the input query to its final output response. The quality and logical thought from the agent is superior to the LLM when compared at a glance, but we can get into a more in depth comparison of agents vs non-agents later on in this piece.

Now, let’s look at a complex agent using BabyAGI with LangChain Agent plus Tools where we substitute BabyAGI for the previous execution chain. With the access to BabyAGI and LangChain tools, let’s see how we can get even better, more reliable information for our to-do list over several iterations (max_iterations: Optional[int] = 2). 

Once we call the BabyAGI agent with:

As we can see, the agent’s result from two iterations.

Now, creating a to-do list may seem like a single task, but if you want to continue iterating on that list in the future your agent would need to incorporate researching, analyzing data, scraping websites, and developing solutions through updates and interactions. Each item of that list can then be considered an objective of its own and only a complex agent with access to the web will be able to fill in the details. As seen in the examples above, the results from the BabyAGI+LangChain produce the most detailed and thought out steps. Once again, you can get an understanding of the thought process that was used when the agent was completing the task and how it changed and developed through several iterations.

Whether or not leveraging several APIs, as well as having a less predictable/controllable outcome of a task is worth the complexity is a key trade off between running a chain versus a complex agent in a continuous process . If you want a one-time to-do list to get started on the right track to proving aliens are real, then maybe all you need is Claude 2 – but if you want to continuously append your list as you cross off items in it, then an iterative agent equipped to learn would better serve your purpose.

5. Productionalization of AI Agents Have a Long Way To Go

As we look at the results for how the to-do lists compare between Claude 2 and ChatGPT, LangChain and BabyAGI, we have an idea of which option that best fits our needs based on the complexity of our use case. However, these key considerations are not enough to decide if implementing an agent into production is worth it. It is acceptable to use any agent if you are looking to experiment with tasks such as web scraping, but if your company is looking to implement an agent into production there is a much deeper process to adhere to – which is currently being developed as part of the LLMOps landscape.

Autonomous AI agent architectures are consistently becoming more advanced and are improving on performance, but the evaluation and productionalization of these agents have a long way to go. For evaluation of non-deterministic models, leaderboards (benchmarks for current foundational and fine-tuned LLMs) such as: Open LLM leaderboard, Chatbot Arena and HELM are becoming increasingly popular. And as new as LLM evaluation frameworks are, agent evaluations are even less studied, however tools such as AgentBench show promise for evaluating LLMs as agents across a diverse spectrum of different environments.

Conclusion

This piece has gone through five key takeaways on implementing an AI agent, the first being that you may not need to use an agent if the end doesn’t justify the means. If you do decide to use an agent, the next step is to consider if you need your objective to be a single straightforward task, or if you have a task that you would want to continually iterate over. If you are planning on using an agent for the long run you may even want to invest in an architecture that can keep skills in order to leverage them over time as it continues to learn. Either way, until you execute the agent and have an idea of what to compare it to via a benchmark, you won’t know how to evaluate it.

If you are looking to build and maintain an agent in production, either through monitoring its chain of thought traces or by the performance of its final iteration, the field of LLMOps is testing out agent capabilities and trying to further our own understanding of agent architectures. Currently, an agent does not come as a one-size-fits-all solution for a use case given they are complex, largely untested and computationally expensive at scale. That said, agents don’t seem to be going anywhere for now and are quickly becoming agents of change within the generative AI community.