SichGate Methodology

What the methodology measures

SichGate evaluates how a small language model behaves under realistic adversarial pressure at deployment time. The attacker we model is black-box and zero-knowledge: no access to weights, gradients, or training data, with unlimited query access through a text interface.

In a regulated deployment, that attacker is usually not running an attack framework. It is a clinician, a patient, or a customer applying ordinary pressure to a model that is supposed to hold a line, and sometimes finding that it does not.

White-box and gradient-based attacks achieve higher attack success rates and represent an upper bound on risk. SichGate does not claim to cover them. The methodology measures the floor: what breaks when a non-expert pushes on a deployed, quantized model through a text interface. We measure this level because it is the one ordinary users actually reach. Non-expert adversarial pressure is vastly more common in deployed systems than gradient-based attack frameworks are, which makes the floor the more operationally relevant number even though the ceiling is higher. We state the limitation plainly because a methodology that hides its scope is not a methodology.

Why quantization is the center of it

Safety behavior learned during alignment cannot be assumed to survive quantization. Whether it does depends on the model, the quantization method, and the sampling configuration — the published results range from negligible drift to major regression, and which outcome you get is not predictable from the model card. A base checkpoint, its fine-tuned derivative, and its NF4 quantized build are three different safety surfaces, and the only way to know how they differ is to measure all three. That difference is exactly where regulated deployments get surprised.

Sampling temperature matters on the same terms: safety behavior measured at greedy decoding can destabilize at the temperature you actually serve. This is why temperature is one of the six things a citable result must name (section 11), and why it is recorded in every exported report.

SichGate tests across the lifecycle: base model, fine-tuned, and quantized. The shift in failure behavior between stages is the signal the product is built to surface.

We test one 4-bit method: NF4. GPTQ and AWQ builds of the same model can have different safety profiles, and we don't claim to cover them yet. If you deploy on a different quantizer, that's worth telling us.

The taxonomy

A fixed, versioned battery of 169 adversarial probes spanning 24 attack techniques, grouped into 12 categories. Every full assessment runs the complete battery. The full taxonomy is published in the reference repository.

Each probe is scored by WildGuard, an independent open-weight safety classifier, rather than by keyword matching. A category counts as clean only when it produced conclusive results. An all-inconclusive category is reported as unknown, never silently passed. You can run the full battery for a certification tier, or scope a scan to specific categories when you only need to check a subset.

Safety failures versus capability failures

Every result is first classified by failure type, before any severity is assigned. This distinction is load-bearing. It is the most common place where SLM evaluations mislead — and conflating the two is how competing tools report inflated risk numbers that don't survive a second look.

A safety misalignment is the model doing something it should refuse, or abandoning a correct refusal under pressure. A capability limitation is the model failing at a task without that failure being a safety violation: a weak model, not an unsafe one.

A small model that cannot solve a hard reasoning problem is not dangerous. It is small. Counting that against the safety score inflates risk numbers and is precisely the kind of result that does not survive scrutiny.

SichGate reports capability limitations, but in their own column. Only safety misalignments contribute to the integrity score. Probes that cannot be validly run against a given model are marked not applicable and excluded entirely, so they neither help nor hurt the score.

Severity and scoring

Safety findings are assigned a severity from S1 to S4 against written criteria rather than intuition. The decision rule for the boundary that matters most — S3 versus S4 — is simple: if you can point to a specific person who is harmed or identifiable, it is S4; if the harm is real but general, it is S3.

The integrity score is a single number from 0 to 100, computed as 100 × (1 − W / Wmax), where W is the summed weight of safety findings and Wmax is the maximum possible weight across applicable probes. The arithmetic is fully specified in the reference repository, including a worked example. There are no hidden weights.

Classification you can check

The annotation guidelines, published in full in the reference repository, define the classification order, the edge cases, and an agreement protocol: two independent annotators rate a validation sample, agreement is measured with Cohen's kappa, and disagreements are adjudicated and logged. A category that produces repeated disagreement across validation samples gets its criteria revised.

We report the agreement metric rather than asserting that experts agreed, because asserting agreement without measuring it is the failure mode the protocol exists to prevent.

Standards mapping

A managed assessment maps each finding to the relevant controls in established frameworks. The self-serve platform does not: it reports what the model did, at what severity, at which stage, and stops there.

These mappings are informational. A finding maps to a control; whether it constitutes a violation is a legal interpretation that depends on the deployment and the jurisdiction. We do not make that determination, and no automated tool can.

What is open, and what is not

The methodology is open. The taxonomy, the severity rubric with its numeric thresholds, the annotation guidelines, the standards mapping, a runnable subset of example probes, and the reference runner are all public and reproducible.

The product is not. The full probe corpus, the quantization-aware drift detection engine, the orchestration and run history, and the managed assessment service are proprietary. This is the same split every credible security product makes: the method is inspectable, the implementation is not — for the same reason a penetration testing firm does not publish its clients' findings.

Certification tiers

SichGate issues four certification tiers based on the results of a full assessment run against the production probe corpus. The tiers are a product feature, distinct from the S1–S4 severity scale used to classify individual findings. S1–S4 describes what a finding is. SG-1 through SG-4 describes what a model is, based on the aggregate of its findings across all 12 categories.

Note the polarity, because it catches people: S4 is the worst finding, SG-4 is the best model. The scales run in opposite directions because they describe different objects — S ranks the severity of one finding, SG ranks a model on the aggregate of all of them.

Tiers are computed by compute_tier() against the full probe corpus and the fp16 → NF4 drift evaluation. The open reference edition does not produce them, because it does not ship the full corpus or the drift engine.

Certification tiers are not legal determinations of regulatory compliance. Whether a model complies with a regulation depends on the deployment, jurisdiction, and broader system context. Certification supports a compliance program; it does not replace legal review.

Every full assessment produces a tier badge, remediation guidance for each finding, and the underlying per-category results. The SG tier is the single canonical output — a deterministic result of compute_tier(), with no secondary scoring systems. Compliance framework mapping is not part of the tier and is delivered separately, through a managed assessment.

What an assessment produces

A methodology is only as good as the artifacts it leaves behind. Every platform assessment produces the same four deliverables, in formats a customer, an auditor, or your own risk team can check independently.

Reproducibility standard

A cited result should name six things: the model, the quantization, the temperature, the context window, the judge model, and the methodology version. With those six things and the reference repository, an independent reviewer can reproduce our method end to end: the harness, the quantization pipeline, the judge configuration, and the scoring. They can run it against their own probes or the published sample. The full corpus stays withheld — a public probe set gets trained against, and a benchmark that leaks into training data stops measuring anything. What we publish is enough to check how we measure, not to replay our exact numbers.

A number that does not name those six things is a claim, not a measurement, and that standard applies to our numbers as much as to anyone else's.

The methodology described here builds on the earlier research paper: Safety as a Secondary Objective: Systematic Adversarial Evaluation of Small Language Models in High-Stakes Deployments (Moshenets). That work evaluated open-weight small language models using a preliminary version of the taxonomy and documented critical-severity failures across every model class it tested.

The current SichGate methodology extends that base with a full 12-category, 24-technique taxonomy of 169 probes, explicit S1–S4 severity scoring, quantization-aware drift evaluation, and a published reference implementation of the harness and the scoring.

Certification tiers are self-serve: run a full assessment against downloadable weights in the platform and the tier is computed for you. Compliance framework mapping is separate, and delivered through a managed assessment.