Managing Agentic Meomery with LangMem [3/5] — Assistant Agent with Semantic Memory

In the first two parts of this series, we built a simple email assistant and connected it with LangMem’s agentic memory framework. In this third part, we’ll extend the assistant by giving it semantic memory — the ability to store facts about users and recall them later when relevant.

Semantic memory is what allows an assistant to remember things like “John is my brother” or “Jack Oliver asked about model endpoints”. Without it, each conversation would start from scratch. With it, the assistant can maintain continuity across multiple email interactions.

Let’s walk through how to build this step by step.

Table of Contents:

1. Setting the Stage: The Existing Email Agent

2. Introducing Tools for Semantic Memory

• Setting Up the Memory Store

• Creating Memory Management Tools with langmem

3. Integrating Memory into the Agent

4. Testing the Memory: A Simple Conversation

5. Building the Final Graph

6. Putting It All Together: A Real-World Scenario

---

Building agents with memory capabilities requires the right tools — and that’s where LangChain’s ecosystem comes in. LangChain provides the open-source foundation for LLM-powered applications, while LangGraph gives developers the framework to design stateful, multi-actor systems.

Now, with the addition of the Langmem SDK, there’s a dedicated library to handle the logic of creating, updating, and retrieving different types of long-term memory inside LangGraph.

Langmem is both flexible and modular: you can start with simple in-memory storage or connect to a production-ready database, depending on your needs.

In this series, we’ll explore how to put these ideas into practice by building an email assistant that demonstrates how semantic, episodic, and procedural memory can be effectively managed with LangGraph and Langmem.

• Introduction to Agentic Meomery [Published!]

• Baseline Writing Assistant Agent [Published!]

• Writing Assistant Agent with Semantic Meomery [You are here!]

• Writing Assistant with Semantic & Episodic Memory [Coming Soon!]

• Writing Assistant with Semantic, Episodic & Procedural Memory [Coming Soon!]

---

---

1. Setting the Stage: The Existing Email Agent

First, let’s quickly re-establish the foundation of our email assistant. This setup will look familiar if you followed along in Part 2. We’ll define a user profile, set triage rules, and create our initial set of tools for writing emails and managing a calendar.

import os
from dotenv import load_dotenv
_ = load_dotenv()

# Define our user profile and prompt instructions
profile = {
    "name": "Youssef",
    "full_name": "Youssef Hosni",
    "user_profile_background": "Senior Data Scientist leading a team of 5 developers",
}

prompt_instructions = {
    "triage_rules": {
        "ignore": "Marketing newsletters, spam emails, mass company announcements",
        "notify": "Team member out sick, build system notifications, project status updates",
        "respond": "Direct questions from team members, meeting requests, critical bug reports",
    },
    "agent_instructions": "Use these tools when appropriate to help manage John's tasks efficiently."
}

email = {
    "from": "Oliver Jack <Oliver.Jack@ToDataBeyond.com>",
    "to": "Youssef Hosni <Youssef.Hosni@ToDataBeyond.com>",
    "subject": "Quick question about API documentation",
    "body": """
Hi Youssef,

I was reviewing the API documentation for the new authentication service and noticed a few endpoints seem to be missing from the specs. Could you help clarify if this was intentional or if we should update the docs?

Specifically, I'm looking at:
- /auth/refresh
- /auth/validate

Thanks!
Oliver""",
}

# Import necessary libraries and initialize the LLM
from pydantic import BaseModel, Field
from typing_extensions import TypedDict, Literal, Annotated
from langchain.chat_models import init_chat_model
from langchain_core.tools import tool

llm = init_chat_model("openai:gpt-4o-mini")

# Define the initial tools for our agent
@tool
def write_email(to: str, subject: str, content: str) -> str:
    """Write and send an email."""
    # Placeholder response - in real app would send email
    return f"Email sent to {to} with subject '{subject}'"

@tool
def schedule_meeting(
    attendees: list[str], 
    subject: str, 
    duration_minutes: int, 
    preferred_day: str
) -> str:
    """Schedule a calendar meeting."""
    # Placeholder response - in real app would check calendar and schedule
    return f"Meeting '{subject}' scheduled for {preferred_day} with {len(attendees)} attendees"

@tool
def check_calendar_availability(day: str) -> str:
    """Check calendar availability for a given day."""
    # Placeholder response - in real app would check actual calendar
    return f"Available times on {day}: 9:00 AM, 2:00 PM, 4:00 PM"

---

2. Introducing Tools for Semantic Memory

Now for the exciting part: adding memory. The core idea of semantic memory is to store facts and retrieve them based on meaning, not just keywords. To do this, we need two main components: a memory store and tools for the agent to interact with that store.

1. Setting Up the Memory Store

We’ll start with an InMemoryStore from LangGraph for simplicity. The crucial step here is to configure an index. This index will use an embedding model to convert our memories into numerical vectors, which is what enables the “semantic” search.

from langgraph.store.memory import InMemoryStore

store = InMemoryStore(
    index={"embed": "openai:text-embedding-3-small"}
)

By specifying “openai:text-embedding-3-small”, we’re telling our store to use this model to understand the meaning of the text it stores.

2. Creating Memory Management Tools with langmem

LangMem is a high-level library built on LangGraph that simplifies memory operations. It provides convenient functions to create pre-configured tools for managing and searching memory.

from langmem import create_manage_memory_tool, create_search_memory_tool

manage_memory_tool = create_manage_memory_tool(
    namespace=(
        "email_assistant", 
        "{langgraph_user_id}",
        "collection"
    )
)
search_memory_tool = create_search_memory_tool(
    namespace=(
        "email_assistant",
        "{langgraph_user_id}",
        "collection"
    )
)

The namespace parameter is critical here. It organizes memories, preventing them from getting mixed up.

• “email_assistant”: A general bucket for this application.

• “{langgraph_user_id}”: This is a template. At runtime, we’ll provide a unique user ID, ensuring that memories for different users are kept completely separate. This is essential for building multi-tenant applications.

• “collection”: A final sub-folder for this user’s memories.

Because these are standard LangChain tools, we can inspect them to see how they work.

print(f"Manage Memory Tool Name: {manage_memory_tool.name}")

Manage Memory Tool Name: manage_memory

print(manage_memory_tool.description)

Create, update, or delete persistent MEMORIES to persist across conversations.
Include the MEMORY ID when updating or deleting a MEMORY. Omit when creating a new MEMORY — it will be created for you.
Proactively call this tool when you:

1. Identify a new USER preference.
2. Receive an explicit USER request to remember something or otherwise alter your behavior.
3. Are working and want to record important context.
4. Identify that an existing MEMORY is incorrect or outdated.

print("\nSearch Memory Tool Name: {search_memory_tool.name}")

Search Memory Tool Name: search_memory

print("\nSearch Memory Tool Description:")
print(search_memory_tool.description)

‘Search your long-term memories for information relevant to your current context.’

The manage_memory tool takes an action (create, update, or delete), content, and an optional ID. The search_memory tool simply takes a text query.

---

3. Integrating Memory into the Agent

Now we need to make our agent aware of these new tools. First, we’ll update the system prompt to include manage_memory and search_memory in the list of available tools.

agent_system_prompt_memory = """
< Role >
You are {full_name}'s executive assistant. You are a top-notch executive assistant who cares about {name} performing as well as possible.
</ Role >

< Tools >
You have access to the following tools to help manage {name}'s communications and schedule:

1. write_email(to, subject, content) - Send emails to specified recipients
2. schedule_meeting(attendees, subject, duration_minutes, preferred_day) - Schedule calendar meetings
3. check_calendar_availability(day) - Check available time slots for a given day
4. manage_memory - Store any relevant information about contacts, actions, discussion, etc. in memory for future reference
5. search_memory - Search for any relevant information that may have been stored in memory
</ Tools >

< Instructions >
{instructions}
</ Instructions >
"""

Next, we’ll create our response agent, adding the new tools to its toolkit and — most importantly — passing in the store we initialized earlier.

from langgraph.prebuilt import create_react_agent

# Helper function to format the system prompt
def create_prompt(state):
    return [
        {
            "role": "system", 
            "content": agent_system_prompt_memory.format(
                instructions=prompt_instructions["agent_instructions"], 
                **profile
            )
        }
    ] + state['messages']

# Our full list of tools
tools= [
    write_email, 
    schedule_meeting,
    check_calendar_availability,
    manage_memory_tool,
    search_memory_tool
]

# Create the agent, passing the tools and the store
response_agent = create_react_agent(
    "anthropic:claude-3-5-sonnet-latest",
    tools=tools,
    prompt=create_prompt,
    # This ensures the store is passed to the agent's tools
    store=store
)

---

4. Testing the Memory: A Simple Conversation

Before integrating this into our full email workflow, let’s test the agent’s memory in isolation. To do this, we need to define a configuration that specifies the langgraph_user_id.