EXL2 \/ EXL3<\/strong><\/td>ExLlama’s flexible-bitrate formats<\/td> Model fits entirely in NVIDIA VRAM and you want maximum speed<\/td> Racing stripes<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nOne person, one machine? GGUF is the right default. Serving a whole team from one GPU box? Look at vLLM with AWQ, GPTQ or FP8 instead. GGUF was built for your desk, not for a hundred users at once.<\/p>\n\n\n\n
\ud83d\udd2c Q4_K_M: A Letter-by-Letter Autopsy<\/h2>\n\n\n\n Back to that wall of files. Every suffix is a recipe. Let’s dissect one.<\/p>\n\n\n\n
The number: bits per weight<\/h3>\n\n\n\n Q4<\/code> means each weight is stored in roughly<\/em> 4 bits. Q8<\/code> means roughly 8. Q2<\/code> means roughly 2 and a lot of regret. Smaller number, smaller file, more rounding error. Hold on to the word “roughly”. It comes back to bite later.<\/p>\n\n\n\n_0<\/code> and _1<\/code>: the boomer formats<\/h3>\n\n\n\nQ4_0<\/code>, Q4_1<\/code>, Q5_0<\/code> and Q8_0<\/code> are the original llama.cpp recipes. They chop the weights into blocks of 32 and give each block one shared scale (_0<\/code>), or a scale plus an offset (_1<\/code>).<\/p>\n\n\n\nPicture a group photo where everyone has to share one exposure setting. Simple, fast, fine on a good day. But if one person in the block is an outlier (wearing a high-vis vest, say), everyone else gets rounded badly to make room for them. Just like a shared office thermostat.<\/p>\n\n\n\n
Q8_0<\/code> is still excellent, because at 8 bits there’s precision to spare. But Q4_0<\/code> and Q4_1<\/code> are retirement material. The next family beats them at the same size.<\/p>\n\n\n\n_K<\/code>: the K-quants<\/h3>\n\n\n\nThe K-quants (Q2_K<\/code> to Q6_K<\/code>), added to llama.cpp in mid-2023, are smarter. They group weights into super-blocks of 256<\/strong>, split those into smaller sub-blocks, give each sub-block its own scale, and then compress the scales themselves<\/em> to claw back space. Compression inside compression. Very Inception.<\/p>\n\n\n\nBack to the group photo: now every small cluster of people gets its own exposure. One person in high-vis no longer ruins it for everyone else. Same file size, measurably better accuracy.<\/p>\n\n\n\n
_S<\/code>, _M<\/code>, _L<\/code>: coffee sizes, except medium is actually the right answer<\/h3>\n\n\n\nThis is the bit almost nobody explains. A model isn’t one uniform blob. It’s hundreds of separate weight matrices (tensors), and some of them matter far more than others<\/strong>. Certain attention and feed-forward tensors are drama queens: round them a little too hard and the whole model suffers.<\/p>\n\n\n\nSo the last letter tells you how generous the recipe is with the drama queens:<\/p>\n\n\n\n
\n_S<\/code> (small):<\/strong> nearly everything at the base bit-width. Smallest file, most damage.<\/li>\n\n\n_M<\/code> (medium):<\/strong> the sensitive tensors get bumped up a tier. In Q4_K_M<\/code>, some of them get 6 bits. This is the sweet spot.<\/li>\n\n\n_L<\/code> (large):<\/strong> even more tensors upgraded. Only offered on low-bit quants like Q3_K_L<\/code>, where every extra bit is worth fighting for.<\/li>\n<\/ul>\n\n\n\nThat’s why Q4_K_M<\/code> became the community default, and the one Ollama usually hands you: it spends the extra bits exactly where they buy the most brain.<\/p>\n\n\n\nIQ<\/code>: the i-quants (for when you’re desperate)<\/h3>\n\n\n\nThen there’s the newer family with names that sound like rejected Star Wars droids: IQ1_S<\/code>, IQ2_XXS<\/code>, IQ3_XXS<\/code>, IQ3_M<\/code>, IQ4_XS<\/code>, IQ4_NL<\/code>.<\/p>\n\n\n\nInstead of rounding each weight to the nearest mark on a ruler, i-quants snap groups<\/em> of weights to the nearest entry in a carefully designed codebook, a lattice of patterns chosen to match what real weights look like. It’s cleverer packing, and it shines exactly where K-quants start falling apart: below 4 bits<\/strong>.<\/p>\n\n\n\nTwo catches, because there are always catches:<\/p>\n\n\n\n
\nSpeed depends on your hardware.<\/strong> Unpacking a codebook is more work per weight. On a GPU you’ll barely notice. On an older CPU, an i-quant can be noticeably slower than a K-quant of the same size.<\/li>\n\n\nThey really want an imatrix<\/strong> (next section). An i-quant made without one is a wasted opportunity.<\/li>\n<\/ul>\n\n\n\nimatrix: free quality you didn’t know you had<\/h3>\n\n\n\n You’ll see repos labelled “imatrix quants”, or files with i1<\/code> in the name. An importance matrix<\/strong> is made by running sample text through the full-precision model and recording which weights actually do the heavy lifting. The quantizer then protects those and squashes the freeloaders.<\/p>\n\n\n\nSame file size, less damage, biggest payoff at low bit-widths. It’s like discovering your credit card came with airport lounge access all along. When you have the choice, take the imatrix version.<\/p>\n\n\n\n
\ud83d\udcc9 So How Much Dumber Does It Actually Get?<\/h2>\n\n\n\n Enough metaphors. Receipts. Here’s llama.cpp’s own reference table for Llama 3 8B<\/strong>, straight from the source code of its quantization tool. “Perplexity increase” measures how much more surprised<\/em> the compressed model is by real text than the original. Lower is better. Zero means no measurable damage.<\/p>\n\n\n\nQuant<\/th> File size<\/th> Real bits\/weight<\/th> Perplexity increase<\/th> Verdict<\/th><\/tr><\/thead> Q8_0<\/strong><\/td>8.5 GB<\/td> 8.5<\/td> +0.003<\/td> Identical twin<\/td><\/tr> Q6_K<\/strong><\/td>6.6 GB<\/td> 6.6<\/td> +0.022<\/td> Effectively lossless<\/td><\/tr> Q5_K_M<\/strong><\/td>5.7 GB<\/td> 5.7<\/td> +0.057<\/td> You’d need a lab to tell<\/td><\/tr> Q4_K_M<\/strong><\/td>4.9 GB<\/td> 4.9<\/td> +0.175<\/td> The sweet spot<\/td><\/tr> Q4_K_S<\/td> 4.7 GB<\/td> 4.7<\/td> +0.269<\/td> Fine if every MB counts<\/td><\/tr> Q4_0<\/td> 4.7 GB<\/td> 4.6<\/td> +0.469<\/td> Same size as Q4_K_S, twice the damage. Why?<\/td><\/tr> Q3_K_M<\/strong><\/td>4.0 GB<\/td> 4.0<\/td> +0.657<\/td> Starting to forget things<\/td><\/tr> Q2_K<\/td> 3.2 GB<\/td> 3.2<\/td> +3.520<\/td> Lights on, nobody home<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nSource: tools\/quantize\/quantize.cpp<\/code> in llama.cpp. Sizes converted from the GiB figures listed there.<\/em><\/p>\n\n\n\n\n\nQuality holds, holds, holds\u2026 then falls off a cliff<\/text>\nPerplexity increase vs the original model, Llama 3 8B (lower is better)<\/text>\n\nQ8_0<\/text>\nQ6_K<\/text>\nQ5_K_M<\/text>\nQ4_K_M<\/text>\nQ4_0<\/text>\nQ3_K_M<\/text>\nQ2_K<\/text>\n<\/g>\n\n\n\n\n\n\n\n\n+0.003 (8.5 GB)<\/text>\n+0.022 (6.6 GB)<\/text>\n+0.057 (5.7 GB)<\/text>\n+0.175 (4.9 GB) \u2190 the sweet spot<\/text>\n+0.469 (4.7 GB) legacy format, same size as Q4_K_S<\/text>\n+0.657 (4.0 GB)<\/text>\n+3.520 (3.2 GB)<\/text>\n<\/g>\nBars drawn to scale. Data: llama.cpp, tools\/quantize\/quantize.cpp.<\/text>\n<\/svg><\/figure>\n\n\n\nThree things jump out, and they’re the whole point of this post:<\/p>\n\n\n\n
One: the damage is not a straight line.<\/strong> From 8 bits down to 5 costs almost nothing. From 4 to 3 costs more than everything above it put together. From 3 to 2, the damage goes up five-fold. Quality doesn’t leak out slowly. It holds, holds, holds, then falls off a cliff. Much like a project timeline.<\/p>\n\n\n\nTwo: look at Q4_0<\/code> next to Q4_K_S<\/code>.<\/strong> Practically the same size. The old format does almost twice the damage. That single row is the entire case for K-quants.<\/p>\n\n\n\nThree: remember “roughly”?<\/strong> Here it is. Q4_K_M<\/code> actually lands at 4.9 bits per weight, not 4. The upgraded tensors, the stored scales and the higher-precision embedding and output layers all add up. When you’re working out whether something fits, use the real number, not the label. The label is marketing.<\/p>\n\n\n\nOne honest caveat before anyone tattoos that table on their arm. Perplexity is a blunt instrument. It averages over a lot of text, and it can hide damage that’s concentrated in one skill: maths, code, long documents, non-English languages. Serious quantizers increasingly also report KL divergence<\/strong>, which measures how far the compressed model’s predictions drift from the original’s, token by token. If one task matters to you, test that task. Don’t trust any single number. Including mine.<\/p>\n\n\n\n\ud83d\udc18 Big and Squashed Beats Small and Pristine<\/h2>\n\n\n\n Here’s the decision people actually face: I’ve got 24 GB. Do I run a 14B model at Q8, or a 32B model at Q4?<\/em><\/p>\n\n\n\nInstinct says “the higher-quality file wins”. Instinct is usually wrong. A bigger model at a sensible quant generally beats a smaller model at high precision, because the bigger model simply knows more and reasons better<\/em>, and Q4_K_M<\/code> only shaves a sliver off that. A slightly blurry photo of the Mona Lisa still beats a crystal-clear photo of a stick figure.<\/p>\n\n\n\nThe rule of thumb:<\/p>\n\n\n\n
\nPick the biggest model that fits at Q4_K_M or better. Only go below Q4 when the jump in model size is large.<\/strong><\/p>\n<\/blockquote>\n\n\n\nWhere the rule breaks: below about 3 bits the cliff is so steep that a crushed big model can lose to a clean small one. And a model from a newer generation often beats an older one twice its size outright. “Bigger wins” means bigger within the same family and era<\/em>, not “grab the 2023 70B because the number is larger”. That’s how you end up with a very large, very confident dinosaur.<\/p>\n\n\n\n\ud83e\udde9 Mixture-of-Experts: the plot twist<\/h3>\n\n\n\n More and more open models are Mixture-of-Experts (MoE)<\/strong>: Mixtral, DeepSeek, the Qwen3 MoE models, OpenAI’s gpt-oss. They come with two numbers, like 30B-A3B<\/em>: 30 billion parameters in total, but only about 3 billion active<\/strong> for any single token. It’s a company of 30 specialists where only three turn up to each meeting. Which, to be fair, is also how most companies work.<\/p>\n\n\n\n\nMemory<\/strong> scales with the total<\/em>. You still have to store all 30B somewhere.<\/li>\n\n\nSpeed<\/strong> scales with the active<\/em> count. Each token only touches about 3B of them.<\/li>\n<\/ul>\n\n\n\nSo an MoE model needs the memory of a 30B model but writes at roughly the speed of a 3B one. That makes them brilliant for machines with lots of ordinary RAM and a modest GPU. llama.cpp can even keep the bulky expert weights in system RAM while the always-used layers sit on the GPU.<\/p>\n\n\n\n
\ud83e\uddee Will It Fit? Maths You Can Do on a Napkin<\/h2>\n\n\n\nStep 1: the weights<\/h3>\n\n\n\n\nSize in GB \u2248 billions of parameters \u00d7 real bits per weight \u00f7 8<\/strong><\/p>\n<\/blockquote>\n\n\n\nModel size<\/th> Q4_K_M (~4.9 bits)<\/th> Q8_0 (~8.5 bits)<\/th><\/tr><\/thead> 8B<\/td> ~4.9 GB<\/td> ~8.5 GB<\/td><\/tr> 14B<\/td> ~8.6 GB<\/td> ~15 GB<\/td><\/tr> 27B<\/td> ~16.5 GB<\/td> ~29 GB<\/td><\/tr> 32B<\/td> ~19.6 GB<\/td> ~34 GB<\/td><\/tr> 70B<\/td> ~43 GB<\/td> ~74 GB<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nSpot the 27B row? That’s where last post’s “about 17 GB” came from: 27 billion \u00d7 4.9 bits \u00f7 8 \u2248 16.5 GB. And that’s why one 15 GB Kaggle T4 couldn’t hold it and two could. See? Napkin maths. Not witchcraft.<\/p>\n\n\n\n
Step 2: the KV cache (the bit that ambushes everyone)<\/h3>\n\n\n\n While the model reads and writes, it keeps a running memory of every token in the conversation. That’s the KV cache<\/strong>, and it grows with every token. On long conversations it can rival the model itself.<\/p>\n\n\n\nFor Llama 3 8B at the default 16-bit precision, the cache costs 128 KB per token<\/strong>. Sounds tiny. Now multiply:<\/p>\n\n\n\n\n8,000 tokens \u2192 about 1 GB<\/strong><\/li>\n\n\n32,000 tokens \u2192 about 4 GB<\/strong><\/li>\n\n\n128,000 tokens \u2192 about 16 GB<\/strong>, more than three times the size of the Q4_K_M model itself<\/li>\n<\/ul>\n\n\n\nThis is the number one reason a model that “definitely fits” crashes or crawls the moment you paste in a long document. It’s also why Ollama ships with a small default context: a bigger window costs real memory. Last time we bumped it to 32K with OLLAMA_CONTEXT_LENGTH<\/code>. Now you know what that bump costs.<\/p>\n\n\n\nPro tip: llama.cpp can compress the cache too. --cache-type-k q8_0 --cache-type-v q8_0<\/code> roughly halves it for very little quality loss. Ollama exposes the same thing through OLLAMA_KV_CACHE_TYPE=q8_0<\/code>, with flash attention switched on.<\/p>\n\n\n\nStep 3: leave headroom<\/h3>\n\n\n\n Add 1 to 2 GB for the runtime, its scratch buffers and, if it’s also your display card, your desktop. A model that fits with 100 MB to spare will find a way not to fit. It’s like packing a suitcase to exactly 23.0 kg: the airport scale will disagree.<\/p>\n\n\n\n
\ud83d\udda5\ufe0f Where to start, by hardware<\/h3>\n\n\n\nWhat you’ve got<\/th> Comfortable choice<\/th><\/tr><\/thead> 8 GB VRAM \/ 16 GB laptop<\/td> 7\u20139B at Q4_K_M to Q5_K_M<\/td><\/tr> 12\u201316 GB VRAM<\/td> 12\u201314B at Q4_K_M to Q6_K<\/td><\/tr> 24 GB VRAM<\/td> ~30B-class at Q4_K_M, or 14B at Q8_0 with a long context<\/td><\/tr> 32\u201364 GB Mac (unified memory)<\/td> 30B-class at Q6_K to Q8_0; 70B-class at Q3\u2013Q4 on the bigger configs<\/td><\/tr> 64 GB+ system RAM, modest GPU<\/td> Large MoE models with the experts parked in RAM<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n\ud83d\udc0c Why Is It So Slow? (It’s the Memory, Not the Maths)<\/h2>\n\n\n\n Local AI has two speeds, and they’re limited by completely different things.<\/p>\n\n\n\n
Prompt processing<\/strong> (reading your input) is limited by raw compute. GPUs flatten it; CPUs suffer. That’s the awkward silence after you paste a 20-page PDF into a CPU-only setup.<\/p>\n\n\n\nToken generation<\/strong> (writing the answer) is limited by memory bandwidth<\/strong>. To produce each token, the machine has to read essentially every active weight from memory, once. Every. Single. Token. So the speed limit is roughly:<\/p>\n\n\n\n\nMax tokens per second \u2248 memory bandwidth \u00f7 size of the model in memory<\/strong><\/p>\n<\/blockquote>\n\n\n\nPlug in some real hardware for a 4.9 GB Q4_K_M 8B model:<\/p>\n\n\n\n
\nDual-channel DDR5 desktop (~80 GB\/s): ceiling \u2248 16 tokens\/s<\/strong><\/li>\n\n\n