What We Measure

We run identical workloads on different hardware setups to produce comparable, reproducible benchmarks. Here's what we track.

⚡ Raw Token Throughput

Tokens generated per second for both prompt processing (prefill) and generation (decode). The fundamental performance metric.

tokens/sec = total_tokens / wall_time

💰 Cost-Per-Token

Hardware cost divided by lifetime token throughput. Includes power consumption. Compares against cloud API pricing.

$ / 1M tokens = (hw_cost / lifespan_tokens) + (power_cost × hours)

🌡️ Thermal Behaviour

Sustained vs. burst performance. Does the hardware throttle after 10 minutes? We log temperature and clock speed over 30-minute sustained runs.

📏 Model Fit Score

Which models can this hardware actually run? We test 7B, 14B, 32B, 70B parameter models and rate usability (smooth, usable, marginal, too slow).

🔋 Power Efficiency

Tokens per watt. Critical for 24/7 deployment scenarios and for comparing Apple Silicon vs. GPU vs. CPU-only setups.

tokens/watt = tokens/sec / watts_drawn

📊 Memory Utilisation

RAM/VRAM usage, swap behaviour, and quantisation impact (Q4_K, Q5_K, Q8, FP16). We show what fits and what doesn't.

Benchmark Methodology

Our methodology is fully reproducible. We publish all test scripts, configurations, and raw data so anyone can verify our results.

🔬 Testing Protocol

1. Environment Setup

OS: macOS (latest stable) for Apple Silicon; Ubuntu 22.04 LTS for x86/GPU rigs
Inference engine: Ollama (latest stable) with identical model files (same quantisation, same GGUF source)
Models tested: Llama 3.1 8B (Q4_K_M), Mistral 7B (Q4_K_M), Qwen2.5 14B (Q4_K_M), Llama 3.1 70B (Q2_K when possible)
No background processes: Fresh boot, minimal services, no browser, no Docker
Power profile: Performance mode, no throttling, plugged in (laptops)

2. Test Workloads

Short prompt (100 tokens in, 200 out): Simulates chat interactions — measures burst performance
Medium prompt (2000 tokens in, 500 out): Simulates document Q&A — measures context processing
Long prompt (8000 tokens in, 1000 out): Simulates code analysis / long-context tasks — measures context window performance
Sustained generation (30 min continuous): Repeated generation to test thermal throttling and sustained throughput
Concurrent requests (1, 2, 4, 8): Tests how throughput scales with parallelism

3. Measurement

Prefill speed: Time to process input prompt (tokens/sec)
Decode speed: Time to generate output tokens (tokens/sec) — this is what users "feel"
Time to first token (TTFT): Latency before first output token appears — critical for chat UX
Power draw: Measured via `powermetrics` (macOS) or `nvidia-smi` / `turbostat` (Linux), logged every 5 seconds
Temperature: CPU/GPU die temperature, logged every 5 seconds during sustained runs
Memory usage: RAM/VRAM consumed, swap activity, and any OOM events

4. Hardware Configurations Tested

MacBook Pro 16" M1 Max (32GB, 64GB) — headless and laptop mode
MacBook Pro 14" M4 Pro (24GB) — current gen comparison
Mac mini M2 Pro (16GB) — budget entry point
Custom GPU rig: RTX 4090 (24GB VRAM), RTX 3090 (24GB), RTX 4070 Ti (12GB)
CPU-only: Ryzen 9 7950X (64GB DDR5) — baseline for "no GPU" scenarios
Salvaged M1 Max (broken screen, headless via SSH) — cost-effectiveness test

5. Reproducibility

All test scripts published on GitHub (MIT licensed)
Raw benchmark data published as CSV with each review
Exact model files and quantisation specified (SHA256 hashes included)
Ollama version and configuration files published
Reviews dated and version-stamped — we re-benchmark when major updates land

Hardware Review Template

Here's the template we use for every hardware review — consistent, comparable, and reproducible.

🖥️ [Hardware Name] — Local LLM Benchmark Review

📅 Review Date: [Date] · 📦 Ollama v[version] · 🏷️ Price at review: $[X] · 📝 Reviewer: [Name]

Hardware Specs

Spec	Value
Chip / CPU	[e.g., Apple M1 Max, 10-core CPU]
GPU / NPU	[e.g., 32-core GPU, 16-core Neural Engine]
Memory	[e.g., 32GB unified memory, 400GB/s bandwidth]
Storage	[e.g., 1TB SSD, read speed X MB/s]
Power Draw (idle / load)	[X watts / Y watts]
Source / Cost	[e.g., Salvaged broken-screen unit — $900 AUD]

Benchmark Results — Token Throughput (tokens/sec)

Model	Quant	Prefill (100 tok)	Decode (200 tok)	TTFT	Sustained (30 min)
Llama 3.1 8B	Q4_K_M	[X] tok/s	[X] tok/s	[X]ms	[X] tok/s
Mistral 7B	Q4_K_M	[X] tok/s	[X] tok/s	[X]ms	[X] tok/s
Qwen2.5 14B	Q4_K_M	[X] tok/s	[X] tok/s	[X]ms	[X] tok/s
Llama 3.1 70B	Q2_K	[X or N/A]	[X or N/A]	[X]ms	[X] tok/s

Thermal & Power Behaviour

Idle temperature: [X]°C · Idle power: [X]W
Peak temperature (30 min sustained): [X]°C
Sustained temperature (steady state): [X]°C
Thermal throttling? [Yes/No] — [X]% performance drop after [Y] minutes
Power draw (sustained load): [X]W average
Tokens per watt: [X] tok/W (Llama 3.1 8B, Q4_K_M)
Noise level: [Silent / Audible / Loud] — [dB reading if available]

Model Fit Assessment

Model Size	Quant	Memory Used	Usability	Verdict
7B	Q4_K_M	[X] GB	🟢 Smooth	[Chat-ready, great experience]
14B	Q4_K_M	[X] GB	🟢 Smooth	[Excellent for most tasks]
32B	Q4_K_M	[X] GB	🟡 Usable	[Slower but functional]
70B	Q2_K	[X or OOM] GB	🔴 Marginal/No	[Too slow / doesn't fit]

Cost Analysis

Hardware cost: $[X] AUD
Estimated lifespan: [X] years
Cost per 1M tokens (at sustained throughput): $[X.XX]
Cloud API equivalent (GPT-4o-mini): ~$0.15 per 1M tokens — break-even at [X]M tokens/month
Cloud API equivalent (Claude 3.5 Sonnet): ~$3.00 per 1M tokens — break-even at [X]M tokens/month
Annual power cost (24/7 operation): $[X] AUD

Pros & Cons

✅ Pros: [3–5 bullet points on what this hardware excels at]
❌ Cons: [3–5 bullet points on limitations, issues, or trade-offs]

Best Use Cases

[e.g., "Best budget option for running 7B–14B models for personal AI assistants"]
[e.g., "Great headless server for Local-LLM-as-a-Service deployments"]
[e.g., "Not recommended for 70B+ models — insufficient memory"]

Verdict

Overall score: [X]/10 · Value score: [X]/10 · Performance score: [X]/10
[2–3 sentence summary: who should buy this, what it's good for, and whether it represents good value in the current market.]

Local LLM Hardware Benchmark Channel