28.9M-Parameter LLM Runs on an $8 ESP32 with Per-Layer Embeddings
A 28.9M-parameter LLM runs entirely on an ESP32-S3 (about $8) by using Google’s Gemma-inspired Per-Layer Embeddings: ~25M parameters are stored in flash and only ~450 bytes are loaded per token, with a 4-bit quantized 14.9MB model that generates short stories at ~9.5 tokens/sec and is not intended for questions, coding, or complex tasks.