confident-ai/deepeval: Evaluation and Tracing Skills for LLM Apps

What is confident-ai/deepeval

confident-ai/deepeval (17,857 GitHub stars) is an open source framework for testing and evaluating LLM applications: agents, RAG pipelines, and multi-turn chatbots. Its own project description centers on ready-to-use evaluation metrics, including a research-backed one called G-Eval, that run against any LLM you choose, statistical methods, or NLP models running locally, so a test suite is not tied to one specific model provider.

Flowy indexes 3 components from the repository, and all 3 are skills: deepeval, deepeval-otel, and deepeval-tracing. This guide covers what each one does on its own, and how the three fit together.

What does the deepeval skill actually do

deepeval is the core workflow skill, and it is the one to reach for when the question is whether an LLM app gives correct answers, not what it is doing internally while it runs. Its description points it at the whole evaluation loop: helping to evaluate or improve an AI agent, a tool-using workflow, a multi-turn chatbot, or a RAG pipeline, adding evals to a project, and generating datasets or goldens to test against. It also triggers directly on requests to run deepeval generate or deepeval test from the command line, so a plain instruction to run the DeepEval tests is enough to invoke it.

What do the two tracing skills add on top

The other two skills are about observability rather than evaluation, and both send data to the same destination: Confident AI, the platform behind DeepEval.

deepeval-tracing instruments an application with DeepEval's own native tracing, its @observe based approach, wiring in a framework, model provider, or vector database so an app's behavior becomes visible inside Confident AI while it runs. deepeval-otel covers a narrower case: exporting raw OpenTelemetry traces and spans, in OTLP format, from an application to Confident AI's Observatory, and configuring the Confident AI OTLP endpoint to receive them.

The practical difference is which instrumentation already exists in a codebase. One with no tracing at all reaches for deepeval-tracing first. One that already emits OpenTelemetry data can send it to the same place with deepeval-otel, instead of adding a second, competing form of instrumentation.

How the three skills usually get used together

None of the three depend on each other, but they answer different halves of the same problem. deepeval checks whether an LLM app's output is correct, using metrics and generated test data, typically during development. deepeval-tracing and deepeval-otel show what actually happened inside a run, step by step, once that same app is live, which tends to be the more useful view once a metric drops and the question turns into why.

Who is this plugin for

Anyone building or maintaining an agent, a RAG pipeline, or an LLM-backed chatbot who wants that testing to run through an established open source library instead of one-off scripts. Because two of the three skills point specifically at Confident AI, the tracing half of this plugin fits best for a team already using, or willing to use, that platform. The deepeval evaluation skill itself carries no such dependency.

How do I install it

Flowy indexes confident-ai/deepeval from its public GitHub repository. Install it from the listing page linked at the top of this guide to make all 3 skills available in a session.