How to Monitor AI Agents in Production

How to Monitor AI Agents in Production with LangSmith

As a builder or solopreneur deploying AI agents, understanding their real-world performance is critical. This guide walks you through setting up production monitoring for your AI agents using LangSmith, a platform designed to help you observe, debug, and improve your LLM applications.

LangSmith offers a comprehensive solution for tracking agent behavior in production, allowing you to identify issues, measure latency, and evaluate outputs. Here’s how to get started:

Step 1: Set Up Your LangSmith Account

First, you need a LangSmith account. Navigate to the LangSmith platform and sign up or log in. Once inside, you’ll be presented with your dashboard.

Your dashboard provides an overview of your projects. If you’re new, you’ll likely start with an empty or default project.

Step 2: Instrument Your Agent with LangSmith

To monitor your agent, you need to integrate LangSmith into your code. This typically involves setting environment variables and wrapping your agent calls. The LangChain framework, for example, makes this straightforward.

For Python applications, you’ll set the following environment variables:

• LANGCHAIN_TRACING_V2=true: Enables LangSmith tracing.
• LANGCHAIN_API_KEY=YOUR_LANGSMITH_API_KEY: Your unique API key for authentication.
• LANGCHAIN_PROJECT=your-project-name: Assigns traces to a specific project within LangSmith.

Here’s a basic Python example of how you might instrument an agent:

import os
from langchain import LLMChain, OpenAI
from langchain.agents import initialize_agent, AgentType

# Set environment variables (replace with your actual key and project name)
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "sk-..." # Your LangSmith API Key
os.environ["LANGCHAIN_PROJECT"] = "my_production_agent"

# Initialize your LLM
llm = OpenAI(temperature=0)

# Define your tools (example: a simple calculator)
def calculate(expression: str) -> str:
    try:
        return str(eval(expression))
    except Exception as e:
        return f"Error: {e}"

tools = [
    {
        "name": "Calculator",
        "func": calculate,
        "description": "A tool to perform mathematical calculations."
    }
]

# Initialize your agent
agent = initialize_agent(
    tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True
)

# Run the agent
response = agent.run("What is 10 + 5 * 2?")
print(response)

When this agent runs, its interactions (prompts, LLM calls, tool usage, final output) will be automatically logged to LangSmith under the specified project.

Step 3: Access Traces in LangSmith

Once your instrumented agent is running, return to your LangSmith dashboard. Navigate to the project you defined (e.g., “my_production_agent”). Here, you’ll see a list of “Traces,” each representing a full execution of your agent.

Each trace provides a detailed breakdown of the agent’s actions:

• Input and Output: The initial prompt and the final response.
• Steps: A chronological view of each step the agent took, including LLM calls, tool invocations, and any intermediate thoughts.
• Latency: How long each step and the overall run took.
• Tokens Used: The number of input and output tokens consumed by LLM calls.

Clicking on individual steps within a trace allows you to drill down further, examining the exact prompts sent to the LLM and the responses received. This level of detail is invaluable for debugging unexpected agent behavior.

Step 4: Monitor and Analyze Key Metrics

Beyond individual traces, LangSmith provides aggregate statistics and dashboards for your projects. You can monitor trends over time for metrics like:

• Latency: Average and P90/P99 latency for your agent runs.
• Token Usage: Total input/output tokens, which can help with cost management.
• Error Rates: Identify how often your agent encounters errors or fails to produce a valid response.

These insights help you understand the overall health and performance of your AI agent in production. By regularly reviewing these metrics, you can proactively identify regressions, optimize performance, and ensure your agent is meeting user expectations.

LangSmith’s monitoring capabilities empower you to move beyond “fire and forget” with your AI agents, giving you the visibility needed to confidently deploy and iterate on your LLM-powered applications.

Disclosure: This article may contain affiliate links… produced with AI assistance and human review — see How We Work.

Sources

• How to Monitor AI Agents in Production
• LangSmith Official Page