GPT4All

• CPU-only box: 8 cores, 15 GB RAM (about 4-10 GB free during tests), no GPU.
• One model only
• Desktop app not tested (no GUI test)

GPT4All loaded and ran a 3B model on our CPU with a few lines of Python and answered quickly once the box was idle. But the catalog's Llama 3.2 3B got both trick questions wrong in both runs and went one word over the summary limit. The SDK and catalog look stale, and the latest models aren't listed. Ollama or llama.cpp with a newer model gave far better results on the same machine.

Easy and fast, but weak, dated catalog models: 2/8.

Coverage

Same 4-task battery, 2 runs, on the catalog's Llama 3.2 3B Instruct. Desktop app not tested.

What we tested

Same 4-task chat battery, 2 runs, with a model from GPT4All's own catalog (Llama 3.2 3B Instruct Q4_0).

Method

gpt4all 2.8.2 Python package, device cpu, 8 threads, temp 0, max 512 tokens. Model file (1.9 GB) fetched from the catalog's Hugging Face URL. Run 1 shared the CPU with other tests; run 2 was on an idle box.

Measured

• pass_rate2/8 (code passed twice; both riddles wrong twice: 85 and 25; summaries 61 words, one over)
• startup_noisePrints CUDA library load errors on a CPU-only box, then runs fine
• run1.codepassed, 6/6 unit cases, wall 83.35 s, output starts: ```python\ndef is_palindrome(s):\n """\n Returns True if
• run1.reason1failed, expect 5, wall 12.78 s, output starts: 85
• run1.reason2failed, expect 5, wall 13.66 s, output starts: 25
• run1.summaryfailed, 61 words, 3/3 key terms, wall 153.68 s, output starts: Here's a summary of the text in 60 words or less:\n\nHarness²
• run2.codepassed, 6/6 unit cases, wall 8.81 s, output starts: ```python\ndef is_palindrome(s):\n """\n Returns True if
• run2.reason1failed, expect 5, wall 1.42 s, output starts: 85
• run2.reason2failed, expect 5, wall 1.35 s, output starts: 25
• run2.summaryfailed, 61 words, 3/3 key terms, wall 34.14 s, output starts: Here's a summary of the text in 60 words or less:\n\nHarness²

Raw test log