ESP32 Microchip Runs 29M Parameter AI Model Locally at 10 Tokens/Sec
Who needs a multi-thousand-dollar GPU cluster when a microchip bought for pocket change can now generate bedtime stories locally without ever touching the cloud?
Independent developer Slava Serbov managed to fit a 28.9 million parameter language model onto an ESP32-S3 board featuring 16 megabytes of flash memory and just 512 kilobytes of internal SRAM. The entire neural network occupies a tiny 14.9 megabyte footprint after four-bit quantization and smart weight distribution.
Rather than shoving the entire memory-hogging structure into RAM, the setup mimics architecture tricks from Google's Gemma by streaming layer embedding tables directly from flash memory on the fly. The dual-core Xtensa processor handles the computing heavy lifting while churning out nearly ten tokens per second directly onto a tiny connected display.
Because hardware constraints remain brutal, the model was trained on the TinyStories dataset synthesized by GPT-4, limiting its vocabulary to that of a toddler. It cannot write complex code or explain quantum physics, but it comfortably eclipses previous attempts like Dave Bennett's earlier 260K parameter experiment.
Running local AI on hardware that costs less than a latte proves that clever optimization always beats raw compute brute force, even if the result currently has the intellect of a three-year-old.
Source: GitHub
Comments
Help shape the next version: Add context or suggest a correction. AI review can add points toward a rewrite. Reviews and updates may take time; a full meter does not guarantee a new version.