AI Agents Should Not Be Accepted on Vibes Alone: How DeepEval Brings LLM Evaluation into the pytest Workflow

The hardest part of maintaining an AI application is often not making the model call work. Quality can quietly drift after a prompt, model, tool, or retrieval change. Traditional unit tests can check a return value, but they struggle to answer whether an answer is faithful, whether an agent completed its task, or whether it used the right tool.

DeepEval is an open-source LLM evaluation framework. Its official README describes it as “pytest for LLM apps”: developers describe inputs, actual outputs, expected outputs, and retrieval context in test cases, then use configurable metrics and thresholds to decide whether a change should pass. That design turns evaluation from a one-off manual review into a repeatable quality gate.

This article treats DeepEval as more than a list of metrics. Based on its official README and source project materials, it explains a practical path: start with black-box output tests, trace agent trajectories, and then connect datasets, metrics, and CI into regression testing.

Why ordinary tests are not enough for AI applications

LLM output is not a stable fixed string. Even for the same input, wording may change with model versions, sampling settings, tool responses, or context. Instead of comparing strings for equality, evaluation should express interpretable criteria:

Correctness: whether the answer matches the expected answer.
Relevancy: whether the response addresses the user’s question.
Faithfulness: whether a RAG answer is supported by retrieved context.
Task completion: whether an agent actually achieved its goal.
Tool correctness: whether the right tool and arguments were used.
Step efficiency: whether the agent took unnecessary steps.

These criteria may use an evaluator model or other NLP and statistical methods, so they are not absolute truth. The engineering goal is to fix the dataset, preserve results, and use a threshold to encode the team’s definition of acceptable quality.

DeepEval’s core model: a Test Case plus a Metric

DeepEval’s basic data structure is LLMTestCase. A minimal case can contain input and actual_output. A more meaningful comparison can add expected_output and retrieval_context. A metric defines how to score the case, while threshold defines the pass line.

The following is a reduced version of the official README quickstart, using GEval to judge whether an actual answer matches an expected answer:

from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams
def test_customer_support_answer():
    correctness = GEval(
        name="Correctness",
        criteria="Determine if the actual output is correct based on the expected output.",
        evaluation_params=[
            SingleTurnParams.ACTUAL_OUTPUT,
            SingleTurnParams.EXPECTED_OUTPUT,
        ],
        threshold=0.5,
    )
    case = LLMTestCase(
        input="What if these shoes do not fit?",
        actual_output="You have 30 days to get a full refund at no extra cost.",
        expected_output="We offer a 30-day full refund at no extra costs.",
        retrieval_context=[
            "All customers are eligible for a 30 day full refund at no extra costs."
        ],
    )
    assert_test(case, [correctness])

There are three important boundaries here. First, actual_output should come from the real application, not from an ideal answer rewritten inside the test. Second, criteria is part of the product’s quality definition and should be specific enough for the team to understand failures. Third, threshold=0.5 is not a universal truth; it is a product decision that should be calibrated on representative data.

Start with black-box tests instead of rewriting the application

DeepEval’s black-box approach fits an existing chatbot or RAG pipeline. You do not need to rewrite the entire application around a particular framework. Feed real inputs into the application, then pass the outputs to metrics.

The installation and execution flow is:

python -m pip install -U deepeval
export OPENAI_API_KEY="set this in your local environment"
deepeval test run test_chatbot.py

OPENAI_API_KEY is only the provider example used by the official quickstart. The README also documents custom LLMs, local NLP models, and other evaluation methods. Never put a key in a test file or CI log; use environment variables or a secret manager.

A practical test suite can have three layers:

1. Smoke eval: a small fixed set on every pull request to catch prompt or tool-interface breakage.

2. Regression eval: a larger golden dataset before merge or release to detect metric degradation.

3. Offline and production review: anonymized real failures added to the dataset to track drift and new error classes.

This separation controls cost and keeps blocking checks distinct from slower deep evaluations.

For RAG, evaluate the evidence as well as the answer

A common RAG failure is a fluent answer that is not supported by retrieved material. Testing only actual_output misses it. Preserve retrieval_context and pair it with metrics for different layers:

Answer Relevancy checks whether the response addresses the question.
Faithfulness checks whether the response aligns with the retrieval context.
Contextual Recall checks whether necessary information was retrieved.
Contextual Precision checks whether relevant nodes are ranked higher.
Contextual Relevancy checks the overall relevance of the retrieved context.

This split separates generation failures from retrieval failures. If faithfulness falls, inspect chunking, metadata filters, and retrieval. If context precision falls, inspect ranking and recall strategy before changing the prompt.

Agent evaluation is about the whole trajectory

For an agent, the final sentence is not the whole result. It may retrieve data, call tools, retry, hand off to a sub-agent, and only then produce a response. The DeepEval README documents tracing interfaces such as observe, update_current_span, and trace, allowing intermediate steps to be collected and evaluated with trajectory-based metrics.

Conceptually, the instrumentation can look like this:

from deepeval.metrics import TaskCompletionMetric
from deepeval.test_case import LLMTestCase
from deepeval.tracing import observe, update_current_span
@observe()
def retrieve_customer_policy(question: str) -> str:
    result = "policy result"
    update_current_span(
        test_case=LLMTestCase(input=question, actual_output=result)
    )
    return result
@observe()
def support_agent(question: str) -> str:
    policy = retrieve_customer_policy(question)
    return f"Answer based on: {policy}"
# Run support_agent inside dataset.evals_iterator(
#     metrics=[TaskCompletionMetric()]
# ).

The evaluation questions become more useful:

Did the agent complete the user’s goal?
Did it call the correct tool?
Were the tool arguments valid?
Did it take unnecessary steps?
Did it follow the expected plan?

The official README lists agentic metrics including Task Completion, Tool Correctness, Goal Accuracy, Step Efficiency, Plan Adherence, Plan Quality, Tool Use, and Argument Correctness. Start with one or two metrics for the most expensive or dangerous failure modes instead of adopting everything at once.

The evaluator also needs governance

LLM-as-a-judge can turn subjective quality into a score quickly, but it introduces bias. An evaluator may prefer longer answers, favor a particular tone, or lack domain knowledge. DeepEval should therefore be treated as an evaluation runner and framework, not an automatic source of truth.

A responsible rollout should:

calibrate thresholds against a small human-labelled dataset;
retain human review for high-risk cases;
fix the evaluator model, prompt, and data version for comparisons;
preserve failure cases and reasons instead of watching only an average score.

DeepEval supports custom LLMs and documents local models and statistical or NLP methods. That can keep sensitive data local, but the team must still validate evaluator accuracy and cost.

Putting it into CI

A basic CI gate might look like this:

name: llm-eval
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: python -m pip install -U deepeval
      - run: deepeval test run tests/evals/test_chatbot.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

This is a workflow example. A production setup still needs dataset versioning, retries, cost controls, and masking of sensitive inputs. A stronger design runs a small, low-cost set on pull requests and a complete trajectory evaluation overnight or for release candidates.

CI should preserve more than pass or fail: each metric score, threshold, evaluator model, and dataset commit should be recorded. That makes it possible to distinguish a quality regression from an evaluator change.

When DeepEval is a good fit

DeepEval is a strong fit when:

an existing RAG, chatbot, or agent needs regression tests;
model, prompt, and retriever changes should share one quality workflow;
the team needs checks for tool calls, agent trajectories, or MCP usage;
Python test files and a CLI fit the current pytest and CI habits.

It is not a replacement for data labelling, and it is not a complete observability system. If the team has not defined what a good answer means, organizing failure cases and human labels comes before adding more metrics.

Conclusion: make AI quality a regression asset

DeepEval’s most useful contribution is not merely its catalogue of metrics. It connects LLM application quality to a familiar testing model: describe cases with LLMTestCase, express quality with metrics, create gates with thresholds, and use tracing to see an agent’s behavior.

Start with a correctness case, add RAG faithfulness and context metrics, and then introduce task completion and tool correctness. This gradual path is easier to maintain than building a giant evaluation platform on day one. When every prompt, model, or architecture change produces comparable evidence, an AI application gains a real regression capability.