Hardware For Local AI Models: What The Memory Shortage Changed

Capacity gets you in the door. Bandwidth decides the rest

Buying for local inference goes wrong in a predictable way: people shop for memory size, get the model loaded, and then discover it answers at reading pace. Size and speed are two different purchases, and on this hardware they pull in opposite directions.

The reason is that generating a token is a memory-bound operation, not a compute-bound one. For every single token, the machine reads the weights it needs out of memory. Do that arithmetic and the ceiling falls out of it directly: a dense 70B model at 8-bit reads roughly 40 GB per token, so a bus that moves about 215 GB/s cannot exceed about five tokens per second no matter what the chip on the other end is capable of. No amount of compute fixes a bus.

The five machines, measured against each other

Bar is memory bandwidth, scaled against the fastest consumer card here. The meter beside it is usable memory. Read them together, because the whole difficulty of this category is that they run in opposite directions: the machines that hold the big models read them slowly, and the one that reads fastest cannot hold them.

Two of those figures deserve a note. The Strix Halo number is measured rather than quoted: AMD's Ryzen AI Max+ 395 specifies a 256 GB/s LPDDR5X bus and delivers around 215 GB/s under mixed CPU and GPU load, which is a normal gap for this memory type and the number you should plan against. Its capacity is also operating-system dependent, at roughly 96 GB addressable under Windows against about 110 GB under Linux, which is a real reason to wipe the machine on arrival.

For scale above all of this, an H100 SXM moves about 3.35 TB/s. Nothing on a desk is close, and that gap is what you are accepting when you buy one of these.

Dense or mixture-of-experts changes the answer by ten times

This is the part that makes the spec sheets misleading, and it is the single most useful thing to understand before spending anything.

A dense model activates every parameter for every token. A mixture-of-experts model activates a small fraction of its total. Both have to be held in memory, because any expert may be needed next, but only the active set gets read per token. Since reading is the bottleneck, the same machine behaves like two different machines depending on which kind of model you hand it.

On the same Ryzen AI Max+ 395 hardware:

• A dense 70B caps near 5 tokens per second at 8-bit, reaching roughly 15 with aggressive 4-bit quantisation.
• A 120B mixture-of-experts model runs at 31 to 55 tokens per second, despite being the larger object on disk.
• A 30B model runs around 100 tokens per second.

A 120B model outrunning a 70B one by ten times on identical silicon is not a benchmarking error. It is the architecture. The practical consequence: if you are buying a high-capacity, modest-bandwidth box, which is what every 128 GB unified-memory machine is, you are buying it to run mixture-of-experts models. Fill it with dense 70B weights and you have bought the wrong machine.

NVIDIA's GB10 boxes show the same shape from the other side. Through vLLM, an 80B dense model at FP8 served at 25 to 40 tokens per second, while a 196B mixture-of-experts model at Q4_K_S through llama.cpp managed about 20 tokens per second at 50 ms latency. LoRA fine-tuning of a 7B model on Alpaca finished in under 18 minutes, peaking at 41.1 GB and 65.4 W without thermal throttling, which is the workload these boxes are genuinely good at.

What the shortage did, and what it withdrew

Every price in this category moved the wrong way in 2026, and the cause is the same one running through 28 separate stories in this site's own briefing archive: AI demand has taken the memory market apart. SK Hynix approved about $38 billion for two new fabrication plants in August, and NVIDIA locked down SK Hynix supply as part of a $500 billion arrangement in July. Consumer hardware is competing with that.

• NVIDIA DGX Spark: launched at $3,999 in October 2025, raised to $4,699 in February 2026, explicitly on memory supply.
• Lenovo ThinkStation PGX, the same GB10 part in a workstation shell: roughly $3,900 to $4,100.
• GMKtec EVO-X2, the Strix Halo box: around $2,000, against the $1,800 it was widely written up at.
• Mac Studio: Apple raised prices across the line in June 2026, the M4 Max configuration by $500 and the M3 Ultra by $1,300.
• RTX 5090: $1,999 at launch, with street pricing well above that.

The more important change is not price, it is that capacity was taken off the market. Apple's Mac Studio specification page now lists the M3 Ultra at 96 GB of unified memory, not configurable, and the M4 Max from 36 GB to 64 GB. The 512 GB and 256 GB options are gone.

That directly retires a recommendation this site made in the guide before this one. It said a 256 GB M3 Ultra at around $6,000 was the best consumer machine for holding a 2-bit quantisation of a 753B mixture-of-experts model, which needs roughly 245 GB. That was true when it was written. You can no longer buy that machine new. The largest Apple configuration available today holds 96 GB, which does not fit that model at any quantisation on the ladder. If you read that advice and were waiting, the window closed.

So which one, honestly

What none of them do

They serve one person. These machines are sized for a single user, and two at a push. Concurrency needs batching, which needs bandwidth and VRAM headroom that none of them have. A team needs cloud GPUs or one box each.

They do not train anything serious. Parameter-efficient fine-tuning of a small model works well, as the 18-minute LoRA run shows. Full fine-tuning of anything large does not. A single H100 is not in the same conversation, and renting one by the hour is cheap next to owning any of this.

They are not the frontier. A good open model on your own hardware is genuinely useful and it is still behind the best closed models on the hardest few percent of work. The setup that survives contact with real work is both: local for volume, loops and anything private, and pay-per-use API for the problems where a subtle wrong answer would cost you.

The case for buying was never really the monthly saving, and at 2026 prices that case is weaker than it was a year ago. What you actually buy is the absence of a meter: an agent that can loop overnight without you doing arithmetic about it, no rate limit mid-run, no data leaving the building, and no vendor deprecating the model you built around. If those are worth four figures to you, buy. If you are doing it to save money on a subscription, run the numbers again at today's prices.