Open Source: 491

Bring your AI app
to production.

Observability, analytics and tests for AI agents and chatbots.
Integrate now — it's free
free up to 1k events / day

Log LLM calls

Log all your prompts and results, see how agents are performing in production.
  • Edit calls in the Playground
  • Search & filter instantly
  • Live tail
  • Label data for fine-tuning
Log LLM calls

Stay on top of costs

Monitor requests and costs segmented by user and model.
  • Auto cost & token calculations
  • Optimize costs
  • Costs by user, prompts & agents
Stay on top of costs

Debug complex agents

Replay agent executions with traces and find out what went wrong and where.
  • Trace agent executions
  • Error stack traces
  • Filter & instant search
Debug complex agents

Be alerted when things go wrong

Write evaluations in English or code and run them against your queries.
  • AI-powered assertions & tests
  • Automated Alerts
  • CI/CD pipeline integration
  • Sentiment Analysis
Be alerted when things go wrong

Replay user chats

Record user conversations and identify gaps in your chatbot's knowledge.
  • Capture user feedback
  • React hooks
  • Dev-friendly frontend integration
Replay user chats

Create training datasets

Label your outputs based on tags and feedback then easily export datasets to fine-tune your models, improving your app's quality and reducing costs.
    Create training datasets
    Trusted by 550+ AI developers at leading companies
    AlaynaVoiceflowWelltoryParagon OneSUPASUPAVponTextyessNational University of SingaporeAlaynaVoiceflowWelltoryParagon OneSUPASUPAVponTextyessNational University of Singapore

    Model-agnostic tracking

    Monitor all your LLMs whether they're the latest OpenAI models or an open-source model rising in the benchmarks.

    Integrate in seconds.

    Our libraries provide automatic instrumention to get setup in seconds.
    import { monitorOpenAI } from "llmonitor/openai"
    const openai = monitorOpenAI(new OpenAI())
    const result = await openai.chat.completions.create({
    model: "gpt-3.5-turbo",
    messages: [{ role: "user", content: "Hello friend" }],
    })
    Explore the docs →

    We recorded a 1 minute demo video.

    (expect better as we ship a lot)

    What our users say

    Join our Open Source community.

    Everything we build is 100% open-source. Check out the source-code or our product roadmap.
    llmonitor: 491

    Developers also love these

    Live TailNew logs and runs are shown in realtime.
    Instant SearchSearch across your whole data in milliseconds.
    Label dataAdd tags to your data to easily filter and fine-tune.
    Self HostSetup on your own servers for maximum data security.
    PlaygroundCollaboratively iterate on prompts.
    Data APIFetch your data to integrate it in other services.
    AlertsSetup alerts to be notified for outlier results and errors.
    React HooksTrack chats and feedback direcly in your frontend.
    AI EvaluationsUse GPT-4 to auto-evaluate and judge responses.

    Get started in minutes.

    Run it on your own or get started in minutes with our hosted version.
    Prompt Playground
    Evaluate & Test
    Alerts
    Public API
    Exports
    Open Source
    Self Hostable
    Agent Tracing
    Analytics
    Chat Replays
    User Tracking
    Feedback Tracking
    LangChain Support
    Prompt Playground
    Evaluate & Test
    Alerts
    Public API
    Exports
    Open Source
    Self Hostable
    Agent Tracing
    Analytics
    Chat Replays
    User Tracking
    Feedback Tracking
    LangChain Support