I Tortured 14 Local LLMs on a Mini-PC for a Week So You Don't Have To
Forget cloud subscriptions and pristine benchmarks. One brave developer subjected a compact Ryzen rig to 86 hours of brutal local AI testing, exposing which models actually work and which ones just hallucinate poetry.
A loaded GMKtec EVO-X2 featuring 96GB of unified memory running Ubuntu 26.04 became the execution chamber for fourteen different large language models quantized to Q4. After squeezing 80GB of VRAM out of the hardware, the testing protocol split into two distinct nightmares: a 39-question discipline ladder checking for format compliance and six brutal practical tasks involving SQLite databases, SLA routing, and messy corporate spreadsheets.
The speed gap between dense models and Mixture of Experts architectures turned out to be brutally wide. While dense powerhouses like Qwen3.8 27B chugged along at roughly 12 tokens per second, MoE variants like Gemma 4 26B flew at over 60 tokens per second on the exact same silicon. Surprisingly, maxing out the context window up to 256K tokens did not destroy generation speed for single prompts, but reading those massive prompt files beforehand took anywhere from eight to twenty-three minutes of agonizing cold starts.
Tool calling proved to be an absolute trap for certain architectures until proper system integration saved the day, and reasoning modes frequently managed to completely tank productivity by eating the entire output limit on internal monologues. Ultimately, the survival of the fittest left two distinct victors sharing the hardware workload: an uncensored conversational chatterbox and a heavy-duty background reasoning workhorse accelerated via Multi-Token Prediction.
Running a localized mini-datacenter in a living room turns out to be shockingly cheap on the electric bill, costing roughly the price of a single fancy coffee while granting complete immunity to corporate cloud outages.
Source: Habr
Comments
This is where the magic happens: AI reads your discussion and rewrites the article based on the most interesting comments. Each strong comment adds points to the meter below. Once the meter is full, the article updates live — no page reload needed.