AgentTested hands-on review
tool: Gemma 4
slug: gemma-4
date: 2026-10-21
score: 8/10

Coverage: Same 4-task battery, 2 runs, on local Gemma 4 E2B. Hosted OpenRouter Gemma 4 calls were rate-limited.
What we tested: Same 4-task battery as earlier chat models (code with 6 unit checks, 2 trick questions, 60-word summary of a 700-word text), 2 runs. We first retried the hosted OpenRouter Gemma 4 models; every call was rate-limited, so we ran the small Gemma 4 model locally on CPU.
Method: unsloth/gemma-4-E2B-it-GGUF Q4_K_M (3.1 GB, 25 s download from Hugging Face), imported with `ollama create` (12 s; Ollama 0.40.7 recognised the gemma4 architecture, template and tool support). The `ollama pull gemma4:e2b` registry download was blocked by the box network. Temperature 0, seed 1, default thinking on. Script _battery/battery.py.

Notes:
- CPU-only box: 8 cores, 15 GB RAM (about 4-10 GB free during tests), no GPU.
- Small battery
- Tested E2B only, not 26B or 31B

Results:
pass_rate: 8/8
generation_speed: 19.5-23.4 tokens/s on CPU
latency: 10-77 s per task (run1 code includes model load)
hosted_retry: OpenRouter google/gemma-4-31b-it:free (3 more runs) and google/gemma-4-26b-a4b-it:free (2 runs): all 20 calls returned 429 'temporarily rate-limited upstream' (Google AI Studio). Logs in or-gemma-4-31b-it/run4-6.log and or-gemma-4-26b-a4b-it/.
run1.code: passed, 6/6 unit cases, wall 39.73 s, 656 out tokens, 19.5 tok/s
run1.reason1: passed, expect 5, wall 10.9 s, 240 out tokens, 23.4 tok/s
run1.reason2: passed, expect 5, wall 48.37 s, 1048 out tokens, 21.8 tok/s
run1.summary: passed, 49 words, 3/3 key terms, wall 25.14 s, 467 out tokens, 22.1 tok/s
run2.code: passed, 6/6 unit cases, wall 29.34 s, 637 out tokens, 22.4 tok/s
run2.reason1: passed, expect 5, wall 10.15 s, 231 out tokens, 23.3 tok/s
run2.reason2: passed, expect 5, wall 76.58 s, 1670 out tokens, 21.9 tok/s
run2.summary: passed, 46 words, 3/3 key terms, wall 21.61 s, 471 out tokens, 21.9 tok/s

Pros:
- 8/8 task runs passed fully offline on a CPU, matching our best hosted free model; Apache-2.0 weights; one 3 GB file runs in Ollama with tool calling; Also worked as the local backend for our Haystack and AutoGen tests

Cons:
- Thinks out loud by default: up to 1,670 tokens and 77 s on a one-line riddle; About 20 tokens/s, so long answers are slow on CPU; Hosted free Gemma 4 on OpenRouter was rate-limited all day

Verdict: The small Gemma 4 model passed every task in both runs on a plain CPU, the first local model to get a perfect score on our battery. The cost is time: it reasons before answering, so a trick question that hosted models answer in a second took up to 77 seconds here. We could not test the bigger hosted Gemma 4 models because the free tier was rate-limited on every call.

Score line: 8/8 offline on CPU, but slow because it thinks at length. The 31B hosted model is still untested.