LLM Observability & Evals
LLM observability tools capture traces of prompts, tool calls, and responses, then layer on evaluation, cost/latency monitoring, and datasets so you can debug and improve quality over time. They are the equivalent of APM for LLM applications, and increasingly overlap with prompt management and evaluation.
| Arize Phoenix Open-source LLM tracing and evaluation, OpenTelemetry-native. | Open source (Elastic License 2.0) | Self-hosted, Managed | Open source (free) | Yes | Yes | No | Yes |
| Braintrust Eval-first platform for iterating on LLM quality. | Proprietary | Managed, Hybrid | Free tier + paid | Yes | Yes | Yes | Yes |
| Datadog LLM Observability LLM tracing and monitoring inside the Datadog platform. | Proprietary | Managed | Usage-based | Yes | Yes | No | Yes |
| Helicone Open-source LLM logging and monitoring via a proxy. | Open source (Apache-2.0) | Managed, Self-hosted | Free tier + paid | Yes | Yes | Yes | No |
| Langfuse Open-source LLM observability, evals, and prompt management. | Open source (MIT core (some enterprise features separate)) | Managed, Self-hosted | Open source (free) | Yes | Yes | Yes | Yes |
| LangSmith Tracing, evals, and prompt management from the LangChain team. | Proprietary | Managed, Hybrid | Free tier + paid | Yes | Yes | Yes | Yes |
| Opik Open-source LLM evaluation and tracing from Comet. | Open source (Apache-2.0) | Self-hosted, Managed | Open source (free) | Yes | Yes | Yes | Yes |
| Portkey AI gateway with observability, guardrails, and prompt management. | Open source (Apache-2.0 (gateway core); platform is managed) | Managed, Self-hosted | Free tier + paid | — | — | — | — |
| Promptfoo Open-source, CLI-first prompt testing and LLM evals. | Open source (MIT) | Self-hosted | Open source (free) | — | Yes | — | — |
| Traceloop (OpenLLMetry) OpenTelemetry-based LLM instrumentation and monitoring. | Open source (Apache-2.0 (OpenLLMetry SDK)) | Self-hosted, Managed | Free tier + paid | Yes | Yes | No | Yes |
| W&B Weave LLM tracing and evaluation from Weights & Biases. | Proprietary | Managed | Free tier + paid | Yes | Yes | No | No |
Compiled from public docs and vendor sites; verify current pricing with the vendor. No vendor pays for placement. Each tool links to a fuller profile.
How to choose
If you want open-source and self-hosting, look at Langfuse, Arize Phoenix, Helicone, and Opik. If you want a managed platform with strong evals, consider LangSmith, Braintrust, or W&B Weave. If you already run Datadog, its LLM Observability keeps everything in one pane. OpenTelemetry support (Traceloop/OpenLLMetry) matters if you want vendor-neutral instrumentation.
FAQ
What does LLM observability actually capture?
Full traces of a request: the prompts, retrieved context, tool/function calls, model responses, token usage, latency, and cost — usually with a way to attach evaluation scores and user feedback.
Do I need evals or just tracing?
Tracing tells you what happened; evals tell you whether it was good. Most teams start with tracing and add automated or human evals once they need to catch regressions before shipping.
Can I self-host these tools?
Several are open-source and self-hostable (Langfuse, Phoenix, Helicone, Opik). Managed-only options trade that for less operational work and built-in scaling.