Haystack

• CPU-only box: 8 cores, 15 GB RAM (about 4-10 GB free during tests), no GPU.
• Toy corpus
• Hosted backend has a free daily cap

Haystack's RAG pipeline answered every question right with both a free hosted model and a local one, and it correctly said 'I don't know' to the question the docs couldn't answer. Hosted answers took under a second. Local ones took about 13 seconds. You write a bit more wiring than in LlamaIndex, but each step is clear.

4/4 on both backends, simple and transparent. Toy corpus.

Coverage

4 small docs, 3 factual questions and 1 unanswerable question, hosted and local.

What we tested

Same RAG task as LlamaIndex: 4 small docs, 3 factual questions and 1 unanswerable question. Run with a free hosted model and with local Gemma 4.

Method

haystack-ai 3.4.1: InMemoryDocumentStore, InMemoryBM25Retriever (top_k 2, no embeddings), ChatPromptBuilder, OpenAIChatGenerator pointed at (a) OpenRouter nvidia/nemotron-3.5-super-vl:free and (b) Ollama's OpenAI endpoint with gemma4-e2b. Temperature 0.

Prompts

1. What is the office cat's name?
2. On which day are new tool tests published?
3. How much RAM does the test box have?
4. What is the CEO's favorite color?

Measured

• install43 packages, about 1.5 s with a warm uv cache
• import_s4.1
• pass4/4 with both backends
• openrouter.catPixel, sources cat.txt and harness2.md, 0.9 s, passed
• openrouter.dayTuesday, sources schedule.txt and harness2.md, 0.7 s, passed
• openrouter.ram15 GB, sources box.txt and harness2.md, 0.8 s, passed
• openrouter.colorI don't know. Sources harness2.md and cat.txt, 1.0 s, passed
• local.catPixel, sources cat.txt and harness2.md, 13.7 s, passed
• local.dayNew tool tests are published every Tuesday at 9 AM Central. Sources schedule.txt and harness2.md, 12.3 s, passed
• local.ramThe test box has 15 GB of RAM. Sources box.txt and harness2.md, 13.3 s, passed
• local.colorI don't know. Sources harness2.md and cat.txt, 14.2 s, passed

Raw test log