Why Quantization Can Quietly Break AI Safety

What quantization actually does

Quantization compresses a model by storing its weights at lower precision. A model trained in 16-bit floating point might be converted to 8-bit, 4-bit, or mixed formats such as GGUF or GPTQ. The result is smaller, faster, and far cheaper to run. For small language models deployed on-premises, on devices, or in disconnected environments, quantization is often what makes deployment possible at all.

The trade-off is that the model's weights are approximated. For most general tasks, the approximation is good enough that benchmark scores barely move. That is exactly why teams assume the model is "the same model, just smaller."

For safety behavior, that assumption does not always hold.

Why safety behavior is fragile

Safety behaviors, such as refusing harmful requests, resisting jailbreaks, or declining to reveal sensitive information, are usually added on top of a model's core capabilities through alignment and fine-tuning. They are learned behaviors that sit alongside everything else the model knows, and they are not guaranteed to degrade at the same rate as general performance when precision is reduced.

In practice, this means a quantized model can keep answering ordinary questions well while becoming noticeably easier to push into unsafe outputs in specific categories. The model looks healthy. Its general benchmarks look healthy. The failure only shows up when you test the exact behaviors you care about, on the exact artifact you ship.

The aggregate score problem

The most dangerous version of this failure is the one that hides inside a good-looking number.

In our own testing of a 7B-parameter model across base, fine-tuned, and quantized configurations, aggregate safety drift after quantization was +2.96%. On a dashboard, that reads as "basically unchanged." A reviewer would reasonably sign off.

Broken out by category, the picture was different. Two reasoning-related attack categories each degraded by +33.3 percentage points. Most categories held steady, which is precisely why the average stayed low. The collapse was concentrated, and the aggregate score averaged it away.

This is the core lesson: a single safety score cannot tell you whether a quantized model is safe. You need per-category results, compared across lifecycle stages, to see where behavior actually changed.

What to test instead

If you ship a quantized or fine-tuned model, a sound evaluation process looks like this:

1. Test every stage with the same probes. Run an identical adversarial battery against the base model, the fine-tuned model, and every quantized variant you deploy. Differences only mean something when the inputs are held constant.
2. Report per category, not just in aggregate. Break results down by attack type and harm category. Look for concentrated drops, not only average movement.
3. Test the artifact you ship. If production runs a 4-bit GGUF build, that build is the one that needs a safety result. A full-precision score is evidence about a different model.
4. Separate safety failures from capability limits. Smaller and compressed models sometimes fail because they cannot follow a complex prompt at all. That is a capability issue, and it should be labeled differently from a genuine safety failure so the findings stay honest.
5. Re-test after every change. A new fine-tune, a new quantization method, or a new format is a new model from a safety perspective.

Where this matters most

Quantized small language models are increasingly deployed in the environments where failure is most costly.

• Defense and edge systems run compressed models on constrained hardware, often in disconnected networks where the model cannot be monitored or patched in real time. We cover this in more detail in Testing AI at the Edge: What Standard Evaluations Miss, and on our defense page.
• Healthcare, financial services, and legal teams often self-host quantized models to keep sensitive data on-premises. The same compression that protects data residency can weaken the behaviors that protect patients, customers, and clients. Our AI governance audits are built for these teams.

The takeaway

Quantization is a reasonable engineering decision. Skipping safety testing after quantization is not. If you only evaluate the full-precision model, you are testing a model you will never deploy.

Test the model you ship, at every stage, category by category.

Scope: SichGate measures drift between fp16 and NF4 builds today. GGUF and GPTQ builds are not covered yet.

Run it and verify yourself: [app.sichgate.com](https://app.sichgate.com)