Bring your AI app
to production.

Observability, analytics and tests for AI agents and chatbots.

Integrate now — it's free

free up to 1k events / day

Log LLM calls

Log all your prompts and results, see how agents are performing in production.

Edit calls in the Playground
Search & filter instantly
Live tail
Label data for fine-tuning

Stay on top of costs

Monitor requests and costs segmented by user and model.

Auto cost & token calculations
Optimize costs
Costs by user, prompts & agents

Debug complex agents

Replay agent executions with traces and find out what went wrong and where.

Trace agent executions
Error stack traces
Filter & instant search

Be alerted when things go wrong

Write evaluations in English or code and run them against your queries.

AI-powered assertions & tests
Automated Alerts
CI/CD pipeline integration
Sentiment Analysis

Replay user chats

Record user conversations and identify gaps in your chatbot's knowledge.

Capture user feedback
React hooks
Dev-friendly frontend integration

Create training datasets

Label your outputs based on tags and feedback then easily export datasets to fine-tune your models, improving your app's quality and reducing costs.

Trusted by 550+ AI developers at leading companies

Model-agnostic tracking

Monitor all your LLMs whether they're the latest OpenAI models or an open-source model rising in the benchmarks.

Integrate in seconds.

Our libraries provide automatic instrumention to get setup in seconds.

import { monitorOpenAI } from "llmonitor/openai"

const openai = monitorOpenAI(new OpenAI())

const result = await openai.chat.completions.create({
  model: "gpt-3.5-turbo",
  messages: [{ role: "user", content: "Hello friend" }],
})

Explore the docs →

We recorded a 1 minute demo video.

(expect better as we ship a lot)

What our users say

Join our Open Source community.

Everything we build is 100% open-source. Check out the source-code or our product roadmap.

llmonitor: 491

Developers also love these

Live TailNew logs and runs are shown in realtime.

Instant SearchSearch across your whole data in milliseconds.

Label dataAdd tags to your data to easily filter and fine-tune.

Self HostSetup on your own servers for maximum data security.

PlaygroundCollaboratively iterate on prompts.

Data APIFetch your data to integrate it in other services.

AlertsSetup alerts to be notified for outlier results and errors.

React HooksTrack chats and feedback direcly in your frontend.

AI EvaluationsUse GPT-4 to auto-evaluate and judge responses.

Get started in minutes.

Run it on your own or get started in minutes with our hosted version.

Get Started (it's free)

Browse the docs

Prompt Playground

Evaluate & Test

Alerts

Public API

Exports

Open Source

Self Hostable

Agent Tracing

Analytics

Chat Replays

User Tracking

Feedback Tracking

LangChain Support