AI-Chain
RAG LLM Developer Tools Open Source AI

From RAG to Agents: How Haystack 3.0 Turns LLM Apps into Observable Pipelines

Share:
From RAG to Agents: How Haystack 3.0 Turns LLM Apps into Observable Pipelines

From RAG to Agents: How Haystack 3.0 Turns LLM Apps into Observable Pipelines

If you have integrated RAG, tool calls, conversational memory, and model providers in one program, you quickly encounter the same problem: the demo works, but the system is hard to change. Replacing a retriever can require changing the prompt, switching a model can require rewiring the flow, and adding an Agent tool can make it difficult to tell where data was changed.

Haystack takes a different approach: split an LLM application into explicit components and connect them with a Pipeline. It is not a black-box Agent framework that hides every detail. Retrieval, routing, memory, generation, and tool calls remain composable, replaceable, and observable. That makes it worth a closer look for AI engineering teams.

This article uses the official repository, current source code, and official documentation as its primary sources. It explains the core design of Haystack 3.0, demonstrates a minimal RAG Pipeline, and discusses the situations where it is useful and the decisions engineers still need to make.

The project: why look at it now?

As checked on August 21, 2026, deepset-ai/haystack had more than 26,000 GitHub stars and its latest push was on August 20, 2026. It uses Apache License 2.0 and is primarily written in Python. This is not a collection of articles or prompts: it is an implementation framework containing a pipeline runtime, components, dataclasses, serialization, and integrations with generators.

The official README describes Haystack as an open-source AI orchestration framework for building production-ready LLM applications in Python. Its current front page highlights Haystack 3.0 and places RAG, multimodal applications, semantic search, question answering, and autonomous agents within one composable architecture. That makes it a strong engineering topic: the interesting question is not only whether it can call a model, but how a changing AI system can be decomposed into testable data flows.

The core idea: components are data-flow nodes

The basic unit in Haystack is a component. Prompt builders, retrievers, rankers, generators, tools, and custom logic can all become nodes in a Pipeline. Each node has explicit inputs and outputs; the Pipeline validates connections, passes data, and can execute branches, loops, and asynchronous flows when needed.

This design provides three engineering benefits:

  • Low replacement cost: change a model, retriever, or prompt builder without rewriting the whole application around a new abstraction.
  • Readable data flow: the Pipeline declaration shows where documents are retrieved, ranked, inserted into a prompt, and sent to a model.
  • Clear testing boundaries: test each component first, then test connections, instead of relying only on black-box tests of the final answer.

The Haystack README also lists native async support, streaming, branches, loops, and conditional logic. A Pipeline is therefore not limited to a straight question-to-model flow; guardrails, tool routing, and multi-stage processing can use the same composition model.

A minimal RAG Pipeline in five minutes

Install the package first:

pip install haystack-ai

The following example is a reduced version of the official PromptBuilder usage in the Haystack repository. Two Document objects stand in for retrieval results. PromptBuilder renders the documents and question into a prompt, and OpenAIChatGenerator sends it to a chat model. In a real application, replace the fixed documents with a retriever output.

from haystack import Document, Pipeline
from haystack.components.builders import PromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.utils import Secret
documents = [
    Document(content="Joe lives in Berlin"),
    Document(content="Joe is a software engineer"),
]
prompt_template = """
Given these documents, answer the question.
Documents:
{% for doc in documents %}
    {{ doc.content }}
{% endfor %}
Question: {{ query }}
Answer:
"""
pipeline = Pipeline()
pipeline.add_component(
    instance=PromptBuilder(template=prompt_template),
    name="prompt_builder",
)
pipeline.add_component(
    instance=OpenAIChatGenerator(
        api_key=Secret.from_env_var("OPENAI_API_KEY")
    ),
    name="llm",
)
pipeline.connect("prompt_builder", "llm")
result = pipeline.run({
    "prompt_builder": {
        "documents": documents,
        "query": "Where does Joe live?",
    }
})
print(result)

The important part is not the answer itself, but the two connection boundaries. PromptBuilder outputs a prompt, while OpenAIChatGenerator consumes it and returns replies. To use another generator, preserve a compatible input and output contract. To use Elasticsearch, OpenSearch, Qdrant, or a custom database, replace the fixed documents with the relevant retriever output.

Another useful detail is that PromptBuilder uses Jinja2 templates. The official source supports overriding the template or template_variables on each pipeline.run(), so prompt engineering can be treated as an input variant for evaluation instead of scattered string concatenation. The same data flow can use different prompts and model parameters for comparison.

From RAG to Agents: control remains in the Pipeline

A typical RAG flow looks like this:

Question → Retriever → Ranker/Filter → PromptBuilder → Chat Generator

An Agent adds a loop that decides what to do next:

Question → Agent
             ├─ call search tool → receive result → reason again
             ├─ call internal API → receive data → reason again
             └─ finish the answer

The Haystack README describes an Agent with lifecycle hooks and mentions hooks such as before_llm, before_tool, and on_exit, together with information such as step_count, token_usage, and tool calls. The value is that an Agent is not merely a model with a list of tools: a team can insert validation, logging, cost controls, and security rules around the model and tool steps.

The README also mentions SkillToolset, which loads skill descriptions into context only when needed. This is useful when the number of tools keeps growing: instead of sending every full tool description to the model every time, first discover a smaller index and load only the capabilities needed for the current task.

That does not mean every flow should become an Agent. If the steps are fixed, an explicit Pipeline is usually easier to test, debug, and estimate. Introduce an Agent loop only when the next step truly depends on intermediate results. Haystack’s advantage is that both modes can coexist within the same component and Pipeline mindset.

Provider abstraction does not erase model differences

The Haystack README lists integrations for OpenAI, Mistral, Anthropic, Cohere, Hugging Face, Google, Azure OpenAI, AWS Bedrock, and local models. This keeps provider SDK formats from spreading through business code.

However, an abstraction layer cannot remove behavioral differences between models. Validate at least these points:

1. Whether chat messages and tool-calling formats match.

2. Whether streaming, token usage, and error details reach observability correctly.

3. Whether context windows, structured output, and retry behavior meet product requirements.

4. The quality and cost of different models with the same retriever and prompt.

Haystack can reduce integration coupling, but it does not guarantee that an evaluation is unnecessary after switching models. The Pipeline makes comparison easier; the team still owns evaluation.

A practical development order for teams

1. Define the data contract first

Define the shapes of documents, queries, generated results, and errors before choosing a retriever. This prevents the application from being coupled to one vector database or model SDK too early.

2. Start with a linear Pipeline

Build the smallest loop of retrieve, build a prompt, and generate. Record each node’s inputs and outputs. Confirm that the right data reaches the prompt before discussing Agents.

3. Treat prompts as replaceable inputs

Use PromptBuilder templates and template_variables to create prompt versions and evaluation cases. Do not make prompt changes untraceable manual edits.

4. Add dynamic tool routing last

Add Agents, tools, hooks, and loops only when the fixed Pipeline can no longer express the requirement. Record every tool name, argument, duration, result size, and failure reason; otherwise debugging becomes guesswork.

Where Haystack fits—and where it does not

Haystack is a good fit when:

  • A Python service needs to combine RAG, Agents, semantic search, and generation.
  • The team wants explicit control over retrieval, routing, memory, and generation rather than a fully hidden workflow.
  • Models, data sources, or custom components need to be replaced while preserving testable boundaries.
  • A product will evolve from simple question answering into multi-step tool-using workflows.

It is not the best fit for every project. A simple SDK may be faster for a one-off prompt demo. If a team does not plan to maintain pipelines, evaluations, and observability, a framework adds conceptual overhead. Composability also means that engineers still need to make the architecture choices: document splitting, embeddings, retrieval, reranking, prompts, models, and security policies do not become correct automatically after installation.

Conclusion: from “the model answers” to “the system evolves”

Haystack is interesting not merely because it supports many models, but because it brings LLM applications back to familiar software engineering concerns: component boundaries, data contracts, replaceability, testing, and observability.

Any framework can make a notebook RAG demo run. When that demo becomes a product with permissions, cost limits, tool calls, and continuous evaluation, a transparent Pipeline and explicit component interfaces become much more valuable. Haystack 3.0 is an option for handling those requirements within one architecture.

If your team is moving from “make a demo that answers” to “maintain an AI system that keeps changing,” try Haystack with a real RAG use case rather than judging it only by its star count.

Sources

GitHub stars, recent activity, and version-related information in this article were checked against the GitHub API and repository README on August 21, 2026. The project changes over time; use the official documentation for the current API.