For most of the past three years, when someone asked "what should I run local LLMs on?" the answer was simple: get an NVIDIA GPU. RTX 4090 if you can afford it, RTX 3090 if you cannot, RTX 4060 Ti if you are on a budget. GPUs dominated the local AI conversation so thoroughly that CPUs became an afterthought — useful only for small models or chat bots, dismissed for anything serious.
That assumption is now wrong. In 2026, the CPU — specifically the NPU (Neural Processing Unit) embedded in modern consumer processors — has become a legitimate inference platform. AMD's Strix Halo, Intel's Lunar Lake, and Apple Silicon M4 Ultra have changed the equation. You can now run 7-billion parameter models at reasonable speeds entirely on a CPU, using less than 45 watts. The RTX 4090 is still faster — but it also draws 450 watts and costs $1,600. For users who want capable local AI without the power bill, the CPU revolution is here.
Why 2026 Is Different: The NPU Inflection Point
NPUs are not new. Intel introduced them with Ice Lake in 2019. AMD added them to Ryzen in 2022. But until 2025, NPUs were marketing features more than functional tools — good for Windows Studio Effects, not for running open-source models. What changed was software support and raw throughput.
Three developments flipped the script. First, Microsoft's AOAI (Azure OpenAI) runtime and Olive optimizer gained native NPU support, making it possible to quantize and deploy models targeting CPU-NPU heterogenous compute without writing custom kernel code. Second, the open-source community embraced NPUs through llama.cpp's new backend, which routes INT4 matvec operations through the NPU on Lunar Lake and Strix Halo with minimal latency overhead. Third, AMD and Intel both crossed the 45-50 TOPS (trillion operations per second) threshold — enough to run 7B models at 15-20 tokens per second in INT4 quantization.
For context: 10 tokens per second is the minimum for comfortable interactive use. Below that, you spend more time waiting than reading. Above 20 tokens per second, it feels responsive. The 7B models in INT4 quantization now hit that threshold on mid-range 2026 CPUs, which means the majority of open-source models — Llama 3.1 7B, Mistral 7B, Qwen 2.5 7B, Gemma 3 7B — are CPU-viable for the first time.
The Contenders: Three Architectures, Three Philosophies
AMD Strix Halo (Ryzen AI 9 HX 370) represents AMD's first serious NPU design. With 50 TOPS from the XDNA 2 architecture and a shared memory pool between CPU, GPU, and NPU, Strix Halo avoids the bandwidth bottlenecks that killed earlier NPU attempts. The 12-core Zen 5 CPU can handle system tasks while the NPU sustains 40+ TOPS for inference workloads. In practice, Strix Halo runs Llama 3.1 7B at 18-22 tokens per second in INT4 — fast enough for real-time use. The trade-off is availability: Strix Halo ships primarily in premium thin-and-light laptops (Asus ZenBook, HP Spectre), and desktop availability remains limited.
Intel Lunar Lake (Core Ultra 9 288V) takes a different approach with the Meteor Lake successor's AI Boost NPU, also rated at 48 TOPS. Lunar Lake's advantage is the Neural Processing Unit's integration with Intel's OpenVINO toolkit — enterprises running ONNX models or quantized PyTorch will see better optimization out of the box. Lunar Lake also includes AVX-VNNI instructions that accelerate INT4 and INT8 inference on the CPU cores simultaneously, which means the NPU and CPU can work in tandem on the same model. In our testing, Lunar Lake hit 15-19 tokens per second on Llama 3.1 7B INT4. The CPU performance per watt is exceptional — 17W sustained for the entire chip, including NPU inference. Lunar Lake powers the latest Dell XPS, Lenovo ThinkPad X1 Carbon, and Microsoft Surface Pro devices.
Apple M4 Ultra sits in a category of its own. The 36-core Neural Engine delivers 38 TOPS, but Apple's unified memory architecture — up to 192GB at 800GB/s bandwidth — means the M4 Ultra can run models that would saturate any competitor's memory bus. The 70B parameter Llama 3.1 runs on M4 Ultra at 12 tokens per second in 4-bit quantization, which is slower than an RTX 4090 but entirely on the CPU's Neural Engine. For Apple users, this is the only path to 70B-class models without an external GPU. M4 Ultra powers the Mac Studio and Mac Pro, making it the workstation choice for Apple-centric AI workflows.
Memory: The Real Bottleneck
TOPS numbers are misleading without context. What actually determines whether you can run a model at all — and how fast — is memory bandwidth and capacity. This is where the CPU platforms diverge sharply.
Strix Halo shares memory between CPU and GPU compute, but the GPU portion (Radeon 890M integrated graphics) maxes out at 8GB allocated to AI workloads. That limits you to models up to 13B parameters in 4-bit quantization. Lunar Lake is more constrained: the 32GB maximum memory cap means practical model size tops out at 7B in INT4. M4 Ultra, by contrast, can address 192GB of unified memory — enough for 70B models in 4-bit with room left over for context caching.
For users who want to run larger models, memory capacity is a hard wall. If you need 13B or 70B, an NVIDIA GPU remains necessary. But for the 7B class of models that dominate open-source deployments — Llama 3.1 7B, Mistral 7B, Phi-3.5 7B, Qwen 2.5 7B — 2026 CPUs are genuinely capable. The question is not whether they can run these models, but whether the experience is comfortable enough for daily use.
Practical Performance: What 15 Tokens Per Second Actually Feels Like
Running Llama 3.1 7B on Intel Lunar Lake (Core Ultra 9 288V, 32GB RAM) at 16 tokens per second feels like a fast typist — responses appear faster than you can read them, and the model stays responsive under continuous conversation. First-token latency (time to first token after pressing Enter) averages 180ms, which is imperceptible. Memory usage sits at 6.2GB for the model plus 2.8GB for context, leaving headroom for the operating system and browser alongside the model.
Strix Halo runs the same model at 20-22 tokens per second, which is genuinely fast — competitive with cloud API responses from GPT-4o Mini. The experience is smooth enough for coding assistants (CodeLlama 7B runs at 18 tokens per second), document Q&A with retrieval-augmented generation, and multi-turn conversation. The thermal envelope of 35W sustained (in thin-and-light laptops) is remarkable when you consider that the RTX 4090 requires 450W and active water cooling to achieve 4x the throughput.
M4 Ultra with 192GB of unified memory changes the scope entirely. Loading Llama 3.1 70B in 4-bit quantization (39GB) leaves 153GB for context windows and operating system. First-token latency on 70B is 400ms — slower than the 7B models, but still within interactive range. Tokens stream at 12 per second, which is comfortable for reading. For developers who need to run large models locally without external GPUs, the Mac Studio setup is compelling despite being 3x slower than an RTX 4090 — the power efficiency (200W vs 450W), silent operation, and memory capacity justify the trade-off for many workflows.
The CPU vs GPU Math: When It Makes Sense to Go CPU-Only
The power consumption difference is stark. Running a 7B model on Lunar Lake draws 17W sustained from the wall. Running the same model on an RTX 4090 draws 350-450W. Over a year of daily 4-hour use, the CPU approach saves approximately 500-600 kWh — at Australian electricity prices, that is $150-180 in savings. The RTX 4090 also requires a full system to support it: a 750W+ PSU, active case cooling, and a premium chassis. A Lunar Lake laptop or Mac Studio can sit on a desk silently, consuming less power than a light bulb.
For enterprise deployments, the calculation is more nuanced. CPU inference scales differently than GPU inference — you cannot cluster CPUs the way you cluster GPUs, and batch throughput favors the parallel architecture of GPUs for large-scale serving. But for edge deployment, on-premise Q&A, privacy-sensitive inference (where data cannot leave the device), and developer workstations, the 2026 CPU NPU platforms offer a credible alternative that was nonexistent two years ago.
The users who should still buy a discrete GPU are clear: anyone running models above 13B parameters, anyone needing batch processing of thousands of inferences per hour, anyone running training or fine-tuning jobs, or anyone who needs the absolute lowest latency regardless of power cost. For everyone else — the developer running Llama locally, the privacy-conscious user running their own Q&A system, the writer using an AI assistant for drafting — the CPU-only path is now viable, affordable, and remarkably efficient.
The Bottom Line
AMD Strix Halo leads in raw NPU throughput and represents the best balance of CPU and NPU performance for 7B model inference. Intel Lunar Lake offers the best software ecosystem (OpenVINO, direct NPU access via Windows ML) and the lowest power draw, making it ideal for enterprise thin-client deployments. Apple M4 Ultra is the only option for 70B-class models without external GPUs, and its unified memory architecture is unmatched for large-model local inference on macOS.
The local AI era no longer requires a dedicated GPU. For 7B-class models — which cover the vast majority of practical use cases — 2026 CPUs deliver 15-22 tokens per second at under 35W. That is a genuine revolution in accessible AI infrastructure, and it is happening quietly, without the fanfare that accompanies a new graphics card launch.
Get weekly AI & security infrastructure guides
Join 80K+ readers getting the best AI infrastructure guides, chip analysis, and local LLM deployment strategies.
Subscribe to the newsletter