Inference Engines Compared: llama.cpp, MLX, ExLlama, vLLM, SGLang, TensorRT-LLM

An inference engine is a scheduler, not just a model loader

An inference engine is the traffic cop, memory manager, kernel dispatcher, scheduler, cache accountant, and parallelism planner. It loads weights, tokenizes input, runs the forward pass, maintains the KV cache, samples tokens, and streams results. Serious engines also handle batching, scheduling, prefix caching, quantization, and API serving.

There are two distinct phases. Prefill is compute-intensive: it reads the prompt and builds the initial KV cache. Decode is memory-bandwidth-bound: it generates one token at a time, repeatedly reading weights and KV cache. Decode speed tracks memory bandwidth more than it tracks peak compute.

The four real bottlenecks

1. Memory bandwidth, not just VRAM size. VRAM determines whether the model fits. Bandwidth determines decode speed. Apple M3 Ultra unified memory moves about 819 GB/s. NVIDIA H100 SXM HBM moves about 3.35 TB/s. Fit is not speed.

2. KV cache growth with batch size and context length. Long-context workloads run out of memory even when weights fit. PagedAttention partitions the KV cache into blocks, which increases utilization.

3. Interconnect cost at GPU boundaries. Tensor parallelism needs frequent all-reduce. Pipeline parallelism communicates at stage boundaries. Expert parallelism needs all-to-all for MoE.

4. Scheduler quality. The scheduler decides which requests enter a batch, how prefill and decode share the accelerator, and whether long prompts block short decodes. Supporting batching is not the same as having a production scheduler.

llama.cpp: portability king

llama.cpp is the answer for weird, constrained, offline, CPU-heavy, or non-NVIDIA datacenter hardware. It supports ARM NEON, Accelerate, Metal on Apple Silicon, x86 AVX/AVX2/AVX512/AMX, RISC-V, low-bit quantization, CUDA, AMD HIP, MUSA, Vulkan, SYCL, and CPU+GPU hybrid offload.

Its HTTP server provides OpenAI-compatible routes, an Anthropic Messages API, reranking, continuous batching, multimodal support, JSON schema constraints, function calling, speculative decoding, and a web UI. It is not for serious multi-node production: the RPC backend is fragile and remains proof-of-concept.

MLX and MLX-LM: the Apple Silicon weapon

MLX is Apple's array framework; MLX-LM is the LLM package built on it. The key hardware fact is unified memory: CPU and GPU have direct access to the same memory pool. The local inference tradeoff becomes whether the model fits and whether the memory system can feed the GPU fast enough.

MLX-LM adds Hugging Face Hub integration, quantization, LoRA and fine-tuning, distributed inference, and a large MLX Community ecosystem. Distributed communication supports MPI, Ring over TCP, JACCL for RDMA over Thunderbolt, and NCCL for CUDA. The project warns that its server is not recommended for production, with only basic security checks.

ExLlamaV2 and V3: consumer CUDA tuned

ExLlamaV2 makes quantized models fast on modern CUDA GPUs, especially consumer cards. It supports paged attention, dynamic batching, prompt caching, KV cache deduplication, batched generation, streaming, and speculative decoding. It is best for a single RTX 3090, 4090, or 5090 box, a local coding assistant, chat, and EXL2 quantized models.

ExLlamaV3 extends toward multi-GPU and local MoE inference. It adds EXL3 quantization based on QTIP, flexible tensor and expert parallelism for consumer hardware, an OpenAI-compatible server via TabbyAPI, continuous dynamic batching, and multimodal support. It is the better fit for two to four or more consumer NVIDIA GPUs, or local MoE.

vLLM: the default open production server

vLLM is the first engine most teams should evaluate for serious open-source LLM serving. It offers PagedAttention KV management, continuous batching, chunked prefill, prefix caching, CUDA and HIP graphs, extensive quantization including FP8, MXFP8/MXFP4, NVFP4, INT8/INT4, GPTQ, AWQ, and GGUF, optimized attention/GEMM/MoE kernels, speculative decoding, and torch.compile.

It is flexible: tensor/pipeline/data/expert/context parallelism, streaming, structured outputs, tool calling, OpenAI and Anthropic Messages APIs, gRPC, and multi-LoRA. It supports NVIDIA, AMD, x86/ARM/PowerPC CPUs, plus TPUs, Gaudi, Ascend, and Apple Silicon plugins. The docs note that multi-node typically uses Ray, and that without NVLink, pipeline parallelism may beat tensor parallelism.

SGLang: vLLM's systems-brained cousin

SGLang is for serving workloads that are ugly: structured outputs, long context, MoE, disaggregation, and routing. It offers RadixAttention prefix caching, prefill-decode disaggregation, speculative decoding, continuous batching, paged attention, tensor/pipeline/expert/data parallelism, structured outputs, chunked prefill, and multi-LoRA batching.

It supports NVIDIA, AMD, Intel Xeon, Google TPUs, and Ascend NPUs. The differentiator is serving architecture. Prefill-decode disaggregation separates compute-intensive prefill from memory-intensive decode into specialized instances, transferring KV cache between them. That prevents long prefill batches from spiking decode latency.

TensorRT-LLM: maximum NVIDIA performance

TensorRT-LLM is the NVIDIA-max-performance stack: optimized, specialized, powerful, and not pretending to be portable. It provides Python APIs to build TensorRT engines with state-of-the-art optimizations, plus Python and C++ runtimes. It includes custom kernels for attention, GEMMs, and MoE; prefill-decode disaggregation, Wide Expert Parallelism, speculative decoding, and a high-level Python API integrated with NVIDIA Dynamo and Triton Inference Server.

B200 GPUs load FP4 weights with optimized kernels. H100 and later support FP8 quantization that doubles performance and halves memory consumption versus 16-bit with minimal accuracy loss. It shines in H100, H200, B200, GB200, and GB300-class fleets, NVIDIA-only datacenters, FP8/FP4 deployment, multi-node serving, and MoE at scale.

Which engine should you use

Remember that the engine is only half the story. Bandwidth sets the ceiling. An Apple M3 Ultra moves about 819 GB/s; an NVIDIA H100 SXM moves about 3.35 TB/s. The same model fits on both; they will not decode at the same speed.

Browse AI infrastructure tools and LLM tools, or read the companion guides on hardware for local AI models and how to run AI models locally.