What is quantisation in LLMs? #llm #ai #ollama #claude #codex

You download an open model, 8 billion parameters, you hit run, and your laptop chokes. It's out of memory. Why? A model is just numbers, billions of them, and all of them have to sit in your RAM. Each number takes 16 bits, times 8 billion, that's 16 gigs. Quantization rounds every number down to 4 bits. Every model, same number of weights, but now it's under 5 gigs and it fits. It runs faster too, there's less memory to read for every word. The catch, rounding loses a little precision. At 4 bits you barely notice, go lower, and the model gets dumb. So when you download a model, pick the one with Q4 in the name, that's the sweet spot. for more.