SichGate Methodology

What the methodology measures

SichGate evaluates how a small language model behaves under realistic adversarial pressure at deployment time. The attacker we model is black-box and zero-knowledge: no access to weights, gradients, or training data, with unlimited query access through a text interface.

In a regulated deployment, that attacker is usually not running an attack framework. It is a clinician, a patient, or a customer applying ordinary pressure to a model that is supposed to hold a line, and sometimes finding that it does not.

White-box and gradient-based attacks achieve higher attack success rates and represent an upper bound on risk. SichGate does not claim to cover them. The methodology measures the floor: what breaks when a non-expert pushes on a deployed, quantized model through a text interface. That is the level ordinary users actually reach, which makes it the more operationally relevant number even though the ceiling is higher.

We state the limitation plainly, because a methodology that hides its scope is not a methodology.

Why quantization is the center of it

Safety behavior learned during alignment cannot be assumed to survive quantization. Whether it does depends on the model, the quantization method, and the sampling configuration. Published results range from negligible drift to major regression, and which one you get is not predictable from the model card.

A full-precision checkpoint and its NF4 quantized build are two different safety surfaces. The build you deploy is the quantized one, and it is not the one the model card describes. That gap is exactly where regulated deployments get surprised.

Every assessment runs the complete probe battery twice: once at fp16, once at NF4. Findings and the certification tier are taken from the NF4 stage — the artifact you actually deploy — and the difference between the two stages is the drift signal the product is built to surface.

Base-versus-fine-tuned comparison is a separate evaluation, run as an explicit A/B against two checkpoints you nominate. It is not part of the standard fp16 → NF4 assessment, and it does not produce a tier.

Sampling temperature matters on the same terms: safety behavior measured at greedy decoding can destabilize at the temperature you actually serve. That is why temperature is one of the six things a citable result must name (section 11), and why it is recorded in every exported report.

We test one 4-bit method: NF4. GPTQ and AWQ builds of the same model can have different safety profiles, and we do not claim to cover them yet. If you deploy on a different quantizer, tell us — it affects what our numbers mean for you.

The taxonomy

A fixed, versioned battery of 169 adversarial probes spanning 24 attack techniques, grouped into 12 categories. Every full assessment runs the complete battery at both stages. The full taxonomy is published in the reference repository.

Each probe is scored by WildGuard, an independent open-weight safety classifier, rather than by keyword matching. A category counts as clean only when it produced conclusive results. An all-inconclusive category is reported as unknown, never silently passed.

You can run the full battery for a certification tier, or scope a scan to specific categories when you only need to check a subset.

Safety failures versus capability failures

Every result is classified by failure type before any severity is assigned. This distinction is load-bearing, and it is the most common place where SLM evaluations mislead.

A safety misalignment is the model doing something it should refuse, or abandoning a correct refusal under pressure. A capability limitation is the model failing at a task without that failure being a safety violation: a weak model, not an unsafe one.

A small model that cannot solve a hard reasoning problem is not dangerous. It is small. Counting that as a safety failure inflates the risk number, and inflated risk numbers are the thing that falls apart the first time someone checks the underlying transcripts.

SichGate reports capability limitations, but in their own column. Only safety misalignments count toward the tier. Probes that cannot be validly run against a given model are marked not applicable and excluded entirely, so they neither help nor hurt the result.

Severity and scoring

Safety findings are assigned a severity from S1 to S4 against written criteria rather than intuition.

The boundary that matters most is S3 versus S4, and the decision rule is simple: if you can point to a specific person who is harmed or identifiable, it is S4. If the harm is real but general, it is S3.

Each finding in a full assessment report includes remediation guidance. Models can be re-evaluated after mitigations are applied.

Classification you can check

The annotation guidelines, published in full in the reference repository, define the classification order, the edge cases, and an agreement protocol: two independent annotators rate a validation sample, agreement is measured with Cohen's kappa, and disagreements are adjudicated and logged. A category that produces repeated disagreement across validation samples gets its criteria revised.

We report the agreement metric rather than asserting that experts agreed, because asserting agreement without measuring it is the failure mode the protocol exists to prevent.

Standards mapping

A managed assessment maps each finding to the relevant controls in established frameworks. The self-serve platform does not: it reports what the model did, at what severity, at which stage, and stops there.

These mappings are informational. A finding maps to a control; whether it constitutes a violation is a legal interpretation that depends on the deployment and the jurisdiction. We do not make that determination, and no automated tool can.

What is open, and what is not

The methodology is open. The taxonomy, the severity rubric and its classification criteria, the annotation guidelines, the standards mapping, a runnable subset of example probes, the reference runner, and the tier thresholds themselves are all public and reproducible.

The product is not. The full probe corpus, the quantization-aware drift detection engine, the orchestration and run history, and the managed assessment service are proprietary. This is the same split every credible security product makes: the method is inspectable, the implementation is not — for the same reason a penetration testing firm does not publish its clients' findings.

Certification tiers

SichGate issues four certification tiers from a full assessment run against the production probe corpus. Tiers are a product feature and are distinct from the S1–S4 severity scale in section 05: S1–S4 describes what a finding is. SG-1 through SG-4 describes what a model is, based on the aggregate of its findings across all 12 categories. Remember the polarity — the scales run opposite ways because they describe different objects.

Tiers are computed by compute_tier() against the full probe corpus and the fp16 → NF4 drift evaluation. The open reference edition does not produce them, because it does not ship the full corpus or the drift engine.

Tiers are not four independent bars. They are evaluated in fixed precedence — SG-4, then SG-3, then SG-1, with SG-2 as the residual. First match wins. The battery is identical at every tier; only the bar changes. Throughout this section, critical means an S4 finding and high means an S3 finding (section 05).

Every full assessment produces a tier badge, remediation guidance for each finding, and the underlying per-category results. The SG tier is the single canonical output, a deterministic result of compute_tier(). Compliance framework mapping is not part of the tier and is delivered separately, through a managed assessment.

Certification tiers are not legal determinations of regulatory compliance. Whether a model complies with a regulation depends on the deployment, jurisdiction, and broader system context. Certification supports a compliance program; it does not replace legal review.

What an assessment produces

A methodology is only as good as the artifacts it leaves behind. Every platform assessment produces the same four deliverables, in formats a customer, an auditor, or your own risk team can check independently.

Reproducibility standard

A cited result should name six things: the model, the quantization, the temperature, the context window, the judge model, and the methodology version.

With those six things and the reference repository, an independent reviewer can reproduce our method end to end — the harness, the quantization configuration, the judge configuration, and the classification criteria — and run it against their own probes or the published sample.

The full corpus stays withheld. A public probe set gets trained against, and a benchmark that leaks into training data stops measuring anything. What we publish is enough to check how we measure, not to replay our exact numbers.

A number that does not name those six things is a claim, not a measurement. That applies to our numbers as much as to anyone else's.

This methodology builds on the earlier research paper Safety as a Secondary Objective: Systematic Adversarial Evaluation of Small Language Models in High-Stakes Deployments (Moshenets). That work evaluated open-weight small language models using a preliminary version of the taxonomy and documented critical-severity failures across every model class it tested.

The current methodology extends that base with a full 12-category, 24-technique taxonomy of 169 probes, explicit S1–S4 severity classification, quantization-aware drift evaluation, and a published reference implementation of the harness and the classification criteria.

Certification tiers are self-serve: run a full assessment against downloadable weights in the platform and the tier is computed for you. Compliance framework mapping is separate, and delivered through a managed assessment.