Opik

Opik is an open-source LLM observability and evaluation platform built by Comet ML, used to debug, test, and monitor LLM applications, RAG systems, and AI agents from development through production. It traces every LLM call an application makes, capturing inputs, outputs, and intermediate steps so developers can see exactly what an agent did and why, rather than treating the model as a black box. For evaluation, Opik provides 30+ built-in metrics along with LLM-as-a-judge scoring for things like hallucination detection, relevance, moderation, and RAG quality, plus support for datasets, experiments, and human annotation queues. It integrates with popular frameworks and can be wired into CI/CD pipelines (via a PyTest integration) so LLM behavior is checked automatically on every commit, and it includes guardrails and PII-handling features along with prompt management and a prompt playground for iterating on prompts directly. Opik is aimed at ML and software engineers building production LLM applications who need visibility into failures that are hard to catch with traditional testing — hallucinations, drifting prompt behavior, or agent tool-calling errors. The core feature set is open source and can be self-hosted or run locally, with a hosted free tier and a paid enterprise version for teams that need scalable, compliance-ready deployments. As with any LLM-as-a-judge approach, its automated evaluation metrics are themselves generated by an AI model, so scores still benefit from spot-checking against human judgment rather than being treated as ground truth.