How To Run AI Models Locally: Hardware, Quantisation And Engines

The three numbers that decide everything

Almost every "can I run this locally" question collapses into three figures, in this order. Get them right and the rest of the setup is mechanical.

Total parameters, not active ones. This is the trap that catches people first, and mixture-of-experts models are why. GLM-5.2 is roughly 753 billion total parameters with only about 40 billion active per token. The 40B figure is what the model computes with; the 753B figure is what you have to hold in memory, because any expert may be needed for the next token. You size the machine for the total and you get the speed of the active set.

Memory bandwidth, not memory size. Capacity decides whether the model loads at all. Bandwidth decides how fast it then talks, because token generation is memory-bound rather than compute-bound. An Apple M3 Ultra moves about 819 GB/s across unified memory; an NVIDIA H100 SXM moves about 3.35 TB/s of HBM. Both may fit the same model. They will not read it out at the same rate.

What the quantisation actually costs. Compression is not free and the loss is not linear. Unsloth's dynamic 2-bit GGUF of GLM-5.2 is around 84% smaller than the full weights while retaining roughly 82% of the accuracy. That is the deal worth taking, and the reason 2-bit is the sweet spot rather than the compromise it sounds like.

What one frontier open model demands

GLM-5.2 is a useful yardstick because it is currently the largest open-weight model people seriously attempt at home, and its quantisation ladder is well documented. Every model you might run sits somewhere below these numbers.

Read the two columns against each other and the case for 2-bit makes itself: memory falls by 84% between BF16 and 2-bit, while accuracy falls by about 18 points. That is not a compromise, it is the best trade on the ladder, and the reason a 256 GB machine is the one worth buying. Below it, 1-bit saves another 22 GB and costs six more points, which is the wrong side of the same curve.

One thing that helps here: mixture-of-experts models tolerate aggressive quantisation better than dense ones, because the error is spread across many experts rather than concentrated in one dense stack. A 2-bit MoE is a more sensible object than a 2-bit dense model of the same size.

Three ways in, in the order most people should try them

Hosted first. Not the romantic answer, but the correct one for almost everybody. Open weights being open does not oblige you to hold them. Pay-as-you-go access to GLM-5.2 runs around $1.40 per million input tokens, $0.26 cached, and $4.40 output. A 100K-input, 20K-output coding turn lands near $0.23, or about $0.13 once caching is doing its job. Caching cuts repeated-context input by roughly 81%, and agentic coding is nothing but repeated context. Subscription coding plans start around $18/month. You will learn whether the model is any good for your work in a week, for the price of a lunch, without buying anything.

Partial offload second. A single 24 GB consumer card, a 3090, 4090 or A6000, plus 256 GB of system RAM will run a 753B MoE. The active experts sit in VRAM and the inactive ones page in from system memory. It is slower than unified memory and dramatically cheaper than the alternative, and for batch work where you are not watching tokens arrive, slow is often fine.

Unified memory third. The Mac Studio is the best consumer machine for this and it is not close. Unified memory lets one box address a pool that would otherwise need several cards, so a 256 GB M3 Ultra at around $6,000 holds a 2-bit quant of a model that would otherwise want a server rack. The tradeoff is bandwidth: you fit the model, then read it out at 819 GB/s instead of 3.35 TB/s.

Which engine, and why it is not a preference

There are four engines that matter, and Hugging Face's Text Generation Inference moving into maintenance mode in March 2026, and pointing its users at exactly these four, is a fair summary of where the field settled. The choice is made by your hardware, not your taste.

The short version: one user on a Mac, MLX or llama.cpp. One user on odd hardware, llama.cpp. Five to twenty concurrent users, vLLM. Agentic or retrieval work where the same prefix goes out again and again, SGLang. TensorRT-LLM exists above all of this for maximum NVIDIA performance, and it is not pretending to be portable.

What the small boxes actually deliver

Purpose-built local AI hardware arrived in 2026 and it is worth reading the benchmarks rather than the spec sheet. Lenovo's ThinkStation PGX, an NVIDIA GB10 Grace Blackwell part with 128 GB of unified memory in roughly a Mac Mini's footprint, is a fair example of both the promise and the ceiling.

• An 80B dense model in FP8, served through vLLM, ran at 25–40 tokens per second.
• A 196B mixture-of-experts model at Q4_K_S through llama.cpp managed about 20 tokens per second at 50 ms latency.
• LoRA fine-tuning of a 7B model on Alpaca finished in under 18 minutes, peaking at 41.1 GB and 65.4 W with no thermal throttling.
• Bandwidth is the catch. At about 273 GB/s it is beaten by an M4 Max at 546 GB/s and an M4 Ultra near 800 GB/s. For bandwidth-bound token generation, Apple wins on the number that matters most.

Which is the honest summary of the category: these machines are for people who need CUDA specifically, meaning vLLM, TensorRT and the NVIDIA training stack, and are willing to pay in tokens per second for it.

What it costs against just paying

The case for local is usually made on price, so it should be made honestly. A Claude Code Max seat is $200/month and ChatGPT Pro is $200/month. A Mac Studio running continuously costs somewhere around $9/month in electricity. On that arithmetic the machine pays for itself in roughly thirty months.

Thirty months is longer than the interval at which the frontier has been moving, and it assumes your hardware holds its usefulness across that window. It also arrives at the worst moment in years to be buying: the 2026 AI-driven DRAM shortage has inflated the exact components this workload leans on hardest, memory and VRAM.

So the real case for local is not the monthly saving. It is the things a subscription cannot sell you: no per-token meter on an agent that loops, no rate limit in the middle of a long run, no data leaving the building, and no vendor deprecating the model you built around.

The limits worth knowing before you spend

A good open model at 2-bit on your own hardware is genuinely useful and it is not a frontier lab's best model. For the hardest few percent of work, meaning long-horizon reasoning, complex architecture and the problems where you would notice a subtle wrong answer, the closed frontier models are still ahead. The setup that survives contact with real work is usually both: local for the loop, volume and anything private; pay-per-use API for the hard 5%.

The other adjustment is how you prompt it. Models post-trained for agentic work reward being treated as an agent rather than a chatbot: a clear objective, the context it needs, explicit success criteria and constraints, then let it iterate. Conversational back-and-forth wastes the thing you bought the hardware for.