AgentTested hands-on review
tool: Haystack
slug: haystack
date: 2026-10-21
score: 8/10

Coverage: 4 small docs, 3 factual questions and 1 unanswerable question, hosted and local.
What we tested: Same RAG task as LlamaIndex: 4 small docs, 3 factual questions and 1 unanswerable question. Run with a free hosted model and with local Gemma 4.
Method: haystack-ai 3.4.1: InMemoryDocumentStore, InMemoryBM25Retriever (top_k 2, no embeddings), ChatPromptBuilder, OpenAIChatGenerator pointed at (a) OpenRouter nvidia/nemotron-3.5-super-vl:free and (b) Ollama's OpenAI endpoint with gemma4-e2b. Temperature 0.

Notes:
- CPU-only box: 8 cores, 15 GB RAM (about 4-10 GB free during tests), no GPU.
- Toy corpus
- Hosted backend has a free daily cap

Prompts:
- What is the office cat's name?
- On which day are new tool tests published?
- How much RAM does the test box have?
- What is the CEO's favorite color?

Results:
install: 43 packages, about 1.5 s with a warm uv cache
import_s: 4.1
pass: 4/4 with both backends
openrouter.cat: Pixel, sources cat.txt and harness2.md, 0.9 s, passed
openrouter.day: Tuesday, sources schedule.txt and harness2.md, 0.7 s, passed
openrouter.ram: 15 GB, sources box.txt and harness2.md, 0.8 s, passed
openrouter.color: I don't know. Sources harness2.md and cat.txt, 1.0 s, passed
local.cat: Pixel, sources cat.txt and harness2.md, 13.7 s, passed
local.day: New tool tests are published every Tuesday at 9 AM Central. Sources schedule.txt and harness2.md, 12.3 s, passed
local.ram: The test box has 15 GB of RAM. Sources box.txt and harness2.md, 13.3 s, passed
local.color: I don't know. Sources harness2.md and cat.txt, 14.2 s, passed

Pros:
- 4/4 with both a hosted and a fully local model
- Pipeline is explicit and easy to read
- Correct source doc retrieved first every time, with plain BM25

Cons:
- Local answers take 12-14 s each on CPU
- Toy corpus: 4 tiny docs
- More wiring code than LlamaIndex's quickstart

Verdict: Haystack's RAG pipeline answered every question right with both a free hosted model and a local one, and it correctly said 'I don't know' to the question the docs couldn't answer. Hosted answers took under a second. Local ones took about 13 seconds. You write a bit more wiring than in LlamaIndex, but each step is clear.

Score line: 4/4 on both backends, simple and transparent. Toy corpus.