Red Hat's asago and the Open Question of Lifecycle Testing

What asago does

Governance policy lives in documents. Controls live in infrastructure. The translation between them is manual, bespoke, and rebuilt at every company that attempts it.

asago automates that translation across four stages:

• Risk mapping. The framework parses an organization's uploaded governance policies and maps requirements onto established risk frameworks — the NIST AI RMF, the OWASP LLM Top 10, and the EU AI Act, routed through the IBM AI Risk Atlas. Policy language becomes a structured risk profile.
• Risk assessment. It generates and runs use-case-specific test scenarios against the risks identified, rather than relying solely on generic benchmarks.
• Risk mitigation. It recommends controls matched to what the assessment surfaced.
• Deployment. It emits declarative configurations for Kubernetes, Terraform and Ansible, so posture holds across clouds and on-prem.

Underneath all four runs an audit trail, designed so an auditor can trace a live production control back to the policy clause, the test, and the evidence behind it.

Two of those design decisions deserve more attention than they've gotten. The first is that traceability is treated as a first-class output rather than a reporting afterthought. The second is the explicit position that generic benchmarks aren't sufficient on their own — scenarios are generated against your identified risks. Both are the right calls, and having them encoded in open infrastructure by organizations with this reach will move defaults across the industry.

The dimension the framework doesn't model

Based on the public announcement materials, asago's unit of analysis is a system: policy in, risk profile, tests, guardrails, config out.

That's a snapshot of a thing that is actually a sequence. A model that reaches production has usually passed through at least three states — base weights, fine-tuned, then quantized for the device or the budget. Each transition modifies the artifact. Whether it modifies safety behavior is the interesting empirical question, and the answer is more complicated than either optimists or pessimists tend to say.

What the literature actually shows

Fine-tuning is the settled half. Qi et al. (2023) demonstrated that safety alignment can be compromised by fine-tuning on a small number of adversarially designed examples — they jailbroke GPT-3.5 Turbo's guardrails with ten examples for under twenty cents. Their broader conclusion was that fine-tuning introduces safety risks that existing safety infrastructure doesn't cover, and that initial alignment quality doesn't guarantee alignment survives customization. That result has been widely replicated and is no longer controversial.

Quantization is the contested half, and the disagreement is the most useful thing in this literature.

On one side, several studies find meaningful degradation. Kharinaev et al. (2025) report that both post-training quantization and quantization-aware training can degrade safety alignment. Wee et al. (2025) find that naive PTQ methods like RTN and GPTQ at 4-bit weights and activations produce large SafetyBench drops — often more than twenty points — concentrated precisely in the categories alignment tuning targets. Their framing is worth sitting with: quantization becomes a safety vulnerability when the optimization target is low perplexity, because low perplexity is a poor and sometimes misleading proxy for retained alignment. Al Hakim et al. (2026) find consistent degradation across static and dynamic methods, notably worse in non-English settings. Rath et al. (2026) evaluate five precision levels and find a dose-response pattern, with 3-bit quantization producing new stereotypical behavior in 6-21% of previously unbiased items.

On the other side, Prasad et al. (2026) ran a large factorial study — 8 instruction-tuned models, 3 precisions, 6 temperatures, 144 configurations, roughly 2 million responses — and found standard quantization approximately safety-neutral. For 7 of 8 models, AWQ INT4 kept attack success within about 1.6 percentage points of FP16 or improved it. The one clear exception was SmolLM3-3B, which moved from 34.5% to 44.1%.

These results are less contradictory than they look. The degradation studies concentrate on aggressive schemes — W4A4, 3-bit, naive PTQ — while the neutral result uses a well-tuned INT4 method. The reasonable synthesis is that the safety impact of quantization is method-dependent and bit-width-dependent, which means "does quantization hurt safety" isn't really answerable in the abstract. It's a property of a specific model compressed a specific way.

Two findings from the Prasad paper are worth flagging separately, because they generalize beyond the quantization debate:

• Single-benchmark evaluation understates risk badly. Several models scored 0% on AdvBench while exceeding 80% on ManyHarm. Any conclusion drawn from one suite is fragile.
• Tooling fails quietly. Their default AWQ export silently produced degenerate outputs for two of eight models, recoverable only with a dedicated re-quantization run. A pipeline that doesn't check its own artifacts can ship something nobody tested.

Sampling temperature, in their results, mattered more than precision — decision instability rose sharply at higher temperatures. That's a deployment parameter most eval harnesses hold fixed and most production systems don't.

Why this matters for the governance stack

If the safety properties of an artifact are contingent on how it was compressed, what temperature it runs at, and which benchmark you happened to pick, then the interesting design question for any governance framework is what the evidence is attached to.

A control traced back to a policy clause and a test result is a real improvement over a spreadsheet. But the value of that trace depends on whether the tested artifact is the deployed artifact — same weights, same precision, same decoding parameters. Where that binding isn't explicit, the audit trail can be perfectly intact and still describe something other than what's running.

That's not a criticism of asago specifically. It's an open problem across the current generation of AI governance tooling, and it's the natural next thing for these frameworks to model. asago is in formation phase with open governance and a public repository, which makes it exactly the right time for people who care about this to say so in the issue tracker rather than in blog posts.