---
title: "Building Block: Augmented Large Language Models in Agentic Systems"
source: Panaversity Learn Agentic AI
source_url: https://github.com/panaversity/learn-agentic-ai
licence: Apache-2.0
domain: agentic-ai
subdomain: panaversity-agentic
date_added: 2026-04-25
---

# Building Block: Augmented Large Language Models in Agentic Systems

## Introduction

Agentic systems are AI systems in which an LLM (Large Language Model) can autonomously decide how to achieve a goal, including what tools to use and what information to gather, rather than following a rigid, pre-defined script. In complex tasks, an LLM alone (even a very large one) is often not enough – real-world queries may require fresh knowledge, extended reasoning, or specialized computations beyond the model’s internal capacities. For example, answering a detailed analytical question might require **planning**, **memory**, using **different tools**, and breaking the problem into sub-parts. To address these needs, modern LLM-based agents augment the core LLM with external modules for retrieval, tools, and memory. In short, an *LLM agent* is essentially an LLM with added capabilities for complex reasoning, an integrated memory, and the means to execute tasks via tools. These augmentations overcome key limitations of standalone LLMs (such as limited context window, fixed knowledge cutoff, or lack of interaction with external systems) and empower the LLM to handle more complex, dynamic tasks.

**Retrieval**, **tools**, and **memory** are three fundamental augmentations that enhance an LLM’s capabilities in an agentic architecture. Retrieval augmentation connects the LLM to external knowledge sources (documents, databases, or the web), so it can fetch up-to-date or domain-specific information on demand instead of relying solely on its training data. Tool augmentation allows the LLM to invoke external programs or APIs – for example, running calculations, executing code, querying databases, or calling web services – effectively letting the model act on the world and obtain new results. Memory augmentation provides the system with the ability to retain and recall information over longer timescales than the LLM’s built-in context window, addressing the problem of “digital amnesia” (the inability of AI to remember past interactions). In the following sections, we describe the architecture of an augmented LLM agent and explain how **retrieval**, **tool use**, and **memory** integrate with the LLM during inference to greatly expand its problem-solving abilities.

## Architecture Overview of the Augmented LLM

![](block.webp)

Figure 1: Conceptual architecture of an **augmented LLM** in an agentic system. The LLM (center) serves as the agent’s core, taking user input and producing output. It interfaces with three key modules: a **retrieval** system, a set of external **tools**, and a **memory** store. During inference, the LLM can send queries to the retrieval module and receive back relevant information (dashed arrow labeled “Query/Results”), call external tools and obtain their return values (“Call/Response”), and read from or write to the long-term memory store (“Read/Write”). This design allows the LLM to dynamically pull in knowledge, perform actions, and store or recall information beyond its native capacities.* ****

At the heart of this architecture is the **LLM core**, which generates plans and decisions as it processes a task. Rather than working in isolation, the LLM is augmented with well-defined interfaces to external modules (as shown in Figure 1). Whenever the LLM determines that additional information or an action is needed, it can leverage these interfaces *in the middle of its reasoning process*. The model’s prompts are structured to inform it about available tools and retrieval APIs, and to incorporate memory context. Our current advanced models are capable of actively using such augmentations – for instance, the LLM can generate its own search queries when it needs more data, select an appropriate tool from its toolbox, or decide what information from the current interaction should be written to long-term memory for future use. By tailoring these capabilities to the use case and giving the LLM a clear interface to each module, we create a *composable, modular agent* that flexibly invokes different functionalities during a single inference session.

The augmented LLM architecture can be seen as the basic building block for agentic AI systems. In this design, the LLM remains the decision-making brain, but retrieval, tools, and memory modules extend the boundaries of what the LLM can do. In the next sections, we delve deeper into each of these components, explaining their roles and how they interact with the LLM to enhance its performance.

## Retrieval Augmentation

**Retrieval augmentation** integrates an external information retrieval system (such as a search engine or a vector database of documents) into the LLM’s inference loop. This gives the LLM access to **grounding data** beyond what it learned during training. In a retrieval-augmented architecture, the model can issue a query to a knowledge base and get back relevant documents or facts, which it then uses to formulate a more informed response. Microsoft’s RAG (Retrieval Augmented Generation) describes this concept well: it is an architecture that *“augments the capabilities of a Large Language Model by adding an information retrieval system that provides grounding data”*. In other words, by fetching up-to-date or context-specific information on demand, retrieval augmentation ensures the LLM’s answers are based on a reliable, current knowledge source rather than just the static training corpus.

There are different strategies to implement retrieval. One common approach is to use a **vector store**: documents are indexed as embedding vectors, and at query time the agent generates an embedding for the user’s query to find semantically relevant documents. The top relevant snippets are then injected into the LLM’s context (prompt) as reference material. This process gives the model controlled access to information that grounds its response in facts. For example, if a user asks a question about the latest financial report of a company, the agent can embed the question, retrieve the most relevant passages from an internal database of reports, and supply those to the LLM. The LLM then has the necessary facts at its fingertips to answer accurately. Importantly, the retrieval step can be repeated or refined iteratively: the LLM might first search broadly, read the results, and then formulate a more specific follow-up query if needed. The orchestrating system (sometimes called the *agent core*) coordinates these hand-offs between the LLM and the retrieval subsystem, but from the LLM’s perspective it is as if it had an extended memory of relevant facts. Retrieval augmentation greatly expands an agent’s knowledge scope, enabling **up-to-date information access, domain-specific expertise (by indexing proprietary data), and supporting evidence** for the LLM’s reasoning process.

## Tool Augmentation

**Tool augmentation** allows an LLM to perform actions and computations by invoking external tools or APIs. Tools are essentially defined operations outside the LLM’s own text prediction capabilities – they could be functions like a calculator, a code interpreter, a database query interface, a web search API, a translation service, or even calls to other AI models. By equipping the LLM with a suite of tools (and instructions on how to use them), we enable it to **extend its functional capabilities** beyond language modeling. In an agentic system, tools are treated as discrete skills the LLM can use when appropriate. For instance, if the task is a complex analysis that involves math, the LLM can call a calculation tool; if it requires live information (weather, stock prices, etc.), the LLM can invoke a web API; if it needs to analyze an image or run code, it uses a specialized tool for that. Each tool usually has a defined interface (e.g. a function signature or an API endpoint) and the agent is given documentation (in its prompt or configuration) about what the tool does and how to call it.

When the LLM “decides” to use a tool, it will output a special **action** (often in a structured format) indicating which tool to use and with what inputs. The agent framework or orchestrator will detect this and execute the tool, then feed the tool’s output (the **observation** or result) back into the LLM’s context. The LLM can then incorporate that result into its reasoning and continue the inference. This loop can repeat multiple times with different tools until the task is solved. In essence, the LLM+tools setup lets the model *interact with its environment*: it can query the world and get results, not unlike a human consulting a calculator or searching a database during problem solving. Tools are often implemented as third-party APIs or code that the agent is allowed to run, so an agent’s “toolbox” might include, for example, a search engine API, a Python interpreter, a weather service, and a custom database query function.

Using tools dramatically enhances what the LLM can do. It provides reliability and precision for subtasks that LLMs alone might struggle with (like arithmetic or executing exact logical operations), and it gives access to **real-time or specific data** that isn’t in the model’s training data. Crucially, tool use is tightly integrated into the LLM’s inference process: the LLM determines *when* a tool is needed and *which* tool to use based on the task at hand. By augmenting an LLM with tool use, we get an agent that can **act** (not just speak) – it can carry out multi-step procedures, fetch external evidence, calculate results, and then incorporate all those outcomes into a coherent answer for the user.

## Memory Augmentation

**Memory augmentation** gives the LLM agent a way to retain information over long conversations or across multiple sessions, addressing the limitation that an LLM’s built-in context window is finite and ephemeral. In an augmented architecture, the agent is equipped with a **memory module** that stores important pieces of information (conversation history, previously acquired knowledge, intermediate results, etc.) and retrieves them when needed. This can be thought of as the agent’s extended memory or knowledge base about its interactions and experiences. There are generally two types of memory in such systems:

* **Short-Term Memory:** This is the temporary context the agent maintains during a single conversation or query. It might include the most recent dialogue turns, the chain-of-thought the LLM has produced so far (i.e. the model’s recent “scratchpad” of thoughts and actions), and any intermediate results. Short-term memory is typically implemented by simply keeping a window of the latest messages or steps in the prompt (since the LLM will remember whatever is in its current context up to the token limit). It serves as an immediate working memory for the ongoing task – the *“agent’s train of thought”* for answering the current query.

* **Long-Term Memory:** This is a persistent store of knowledge and past interactions that the agent can fall back on even after the context window has been exceeded or between sessions. Long-term memory could be a vector database of conversational embeddings, a log of past dialogues, or any structured database that records what the agent has deemed important. It retains the agent’s past behaviors, facts learned, or results from prior tasks over extended time frames. For example, after solving a problem or answering a question, the agent might store a summary of the solution or key facts into a vector store. Later, if a related question comes up, the agent can retrieve those stored facts by semantic similarity search and remind itself of past relevant knowledge.

A robust memory module often combines both short-term and long-term memory (sometimes called **hybrid memory**). In practice, when a new user query arrives, the agent will query its long-term memory store (using the query content or its embedding) to fetch any relevant records from previous conversations or data. Those results are then included in the prompt as context (this is sometimes referred to as context augmentation with pertinent memory). Throughout the inference, the agent’s short-term memory (its ongoing thought process and recent dialogue) is maintained in the prompt. At the end of the interaction, the agent may distill and save new information into long-term memory for future use. By **reading** from long-term memory at the start of inference and **writing** to it at the end, the LLM agent can effectively **remember important information indefinitely**. This enables continuity across sessions and cumulative learning of a sort – the agent doesn’t repeatedly forget earlier discussions or facts it looked up. Memory augmentation is thus critical for applications like personal assistants that need to recall user preferences from past interactions, or research agents that gather knowledge incrementally. It prevents the AI from having to start from scratch on every query and allows it to build up an internal knowledge base over time, mitigating the “amnesia” problem inherent to stateless language models.

## Inference Process and Component Interaction

With the core components of retrieval, tools, and memory in place, the augmented LLM operates as an **interactive reasoning loop** that can plan, act, observe, and remember. The inference process in an agentic system is a cyclical workflow in which the LLM and its augmentations continuously feed into each other’s outputs. A typical inference (handling one complex user query or task) might proceed in the following generalized steps:

1. **Initial Context Assembly:** The user’s request is received by the system. The agent then **retrieves relevant context** from its memory before the LLM starts generating an answer. For instance, the agent will pull up any pertinent facts or past dialogues from the long-term memory store by semantic search (using the user’s query as the key). The most relevant memory entries are added to the prompt, along with a short window of recent conversation (short-term memory) if applicable, and a list or description of available tools (so the LLM knows what tools exist and how to invoke them). This assembled prompt – containing the user query, retrieved background information, tool documentation, and any system instructions – is then given to the LLM to begin processing.

2. **LLM Reasoning and Tool Invocation:** The LLM processes the prompt and starts generating a response. Guided by its prompt and internal logic, the LLM may determine that it needs more information or needs to perform a specific operation to effectively answer the query. At this point, the model can formulate a **plan** (often implicitly via its chain-of-thought) and then output an **action**. The action could be a command to use a tool or to perform a retrieval query. For example, the LLM might produce something like: `Thought: I should look up the latest market data. Action: Search["Q2 2025 earnings results for Company X"]`. This kind of reasoning-before-acting approach is exemplified by the ReAct framework, where the model interleaves *Thought → Action → Observation* in its output. In our scenario, the LLM’s output explicitly calls the **Search** tool with a query string.

3. **Tool Execution (Act) and Observation:** The agent’s controller intercepts the LLM’s action and executes the requested tool. In the retrieval case, this might mean sending the query to a search engine or database and getting back documents. In the case of a different tool (say a calculator or code runner), it means running that operation and capturing the result. The result of the tool’s execution is then returned to the LLM as an **observation**. Typically, the observation is appended to the LLM’s context in a standardized format, for example: `Observation: [Result of the search or tool output]`. This effectively updates the LLM’s short-term memory with new information obtained from the environment. The LLM is not restarted from scratch; instead, it continues the same generation session with the new observation in context.

4. **Iterative Reasoning Loop:** With the new information from the tool, the LLM resumes reasoning. It may now have the data it needed to answer the question, or it might decide further steps are required. The LLM can again produce a thought and possibly another action. For instance, it might say: `Thought: The search results suggest I need to calculate growth rate. Action: Calculator["(2025_revenue - 2024_revenue)/2024_revenue"]`. The cycle of tool execution and observation can repeat multiple times, allowing the agent to break down complex tasks into a sequence of smaller operations. Throughout this loop, the **short-term memory** (the accumulated thoughts, actions, and observations so far) grows in the context, enabling the LLM to “remember” what it has already done and learned in this session. This iterative approach will continue until the LLM determines that it has gathered enough information and completed all necessary computations to produce an answer.

5. **Final Answer Generation:** Once the LLM has all the needed information and has finished any tool-based sub-tasks, it produces the final output for the user. This is typically a natural language answer or the completed result of the task. The agent stops when the LLM’s output indicates a final answer (as opposed to an intermediate action). For example, the LLM might now output the answer to the original question, incorporating the factual data it retrieved and any calculations it performed along the way.

6. **Memory Update:** After responding, the agent can store new useful information into its long-term memory. This may include the entire conversation that just took place (so the agent remembers having answered this question) or any especially important piece of data that was uncovered. Storing a summary or vectorized representation of the interaction allows the agent to recall this exchange in the future if needed. The memory module might tag this entry with keywords or metadata (e.g. “earnings report analysis”) for efficient retrieval later. By writing the highlights of the interaction to memory, the agent *learns* from the task, gradually accumulating knowledge. This ensures that if a related query comes up, the agent won’t have to repeat all the same steps – it can leverage past work.

Throughout this inference cycle, the LLM is effectively orchestrating the entire process: it generates the next steps (questions, tool calls) based on the state of the conversation and available info, and the external modules execute those steps and feed results back. The tight integration of **planning, tool use, and memory** means the agent can handle tasks far beyond the scope of a single LLM prompt-response. It can approach problems in a **decide-act-observe-refine** loop similar to how a human might work through a complex task, consulting references and tools along the way. This design leads to more **accurate, context-aware, and multi-step capable** AI behavior. By maintaining a memory of what it has done and learned, the agent also exhibits continuity and avoids redundant actions. In summary, the retrieval, tool, and memory components collectively function as extensions of the LLM’s reasoning process: *retrieval* gives it knowledge, *tools* give it power to act, and *memory* gives it persistence.

## Conclusion

Augmenting LLMs with retrieval, tools, and memory has become a powerful paradigm for building advanced AI agents. This architecture preserves the generative and reasoning strengths of large language models while mitigating their limitations through integration with external systems. **Retrieval augmentation** ensures that the LLM always has access to relevant, up-to-date information grounded in real data, rather than relying only on what it was trained on. **Tool augmentation** grants the LLM the ability to perform actions in the world – whether calculating an equation, running a database query, or interacting with a web service – thus expanding the scope of tasks it can handle and improving the accuracy and reliability of results. **Memory augmentation** enables long-term contextual awareness, allowing the agent to carry knowledge and learning forward over time and complex dialogues, instead of forgetting everything after each response.

In an agentic system, these components work in unison during inference: the LLM dynamically decides when to tap into the retrieval system, which tool to call for a given subtask, and what information to store or recall from memory. The conceptual design outlined in this whitepaper shows an LLM-centric architecture where modular augmentations plug in to enhance the model’s capabilities at runtime. By using these augmentations, an LLM agent can **plan** and decompose complex problems, **consult external knowledge bases**, **execute specialized operations**, and **remember important information** – all within a single integrated workflow. This results in a more powerful and flexible AI system that can tackle real-world tasks that would be impossible for an unaugmented LLM. As LLM-based agents continue to evolve, the principles of retrieval, tool use, and memory integration will remain central to building systems that are not only intelligent in understanding language, but also *actively competent* in achieving goals in complex environments. The augmented LLM architecture thus represents a pivotal step toward AI agents that can reason, act, and continually learn in pursuit of their objectives, leveraging the best of both world: the generative prowess of large models and the precise capabilities of external systems.

**Sources:** The concepts and architecture described are informed by recent works and frameworks on LLM agents and augmentation techniques, which demonstrate how retrieval, tools, and memory can be combined to create robust AI agent systems.
