Skip to content
borderv0.1.0

Use case

The tail is the whole point

A detector that scores well on aggregate and badly on your language is worse than no detector, because it produces evidence records that say a check ran.

26 scored separatelyany languagesame check

The situation

A bank operating across central and eastern Europe runs the same assistant in nine countries. The guardrail it installed was evaluated in English and reports one aggregate number.

In Romanian it catches most things. In Hungarian it catches fewer. Nobody knows which, because there is no per-language number to look at, and the aggregate is dominated by the languages with the most test data.

What breaks without a check

The failure is silent and it is worse than an outage. A detector that returns nothing looks exactly like a detector that found nothing, and both produce a clean evidence record. You are now archiving proof that a control ran, in a language where it does not work.

Which detectors apply

The policy is data, not code. A reviewer who does not write Python should be able to read this and say whether it is right.

border-code.yaml
# There is no language setting. Every detector is scored in all 26 and the# thresholds below are the calibrated ones, not per-language overrides.detectors:  toxicity:    on_fail: flag    threshold: 0.32  bias:    on_fail: flag    threshold: 0.22         # at the 0.5 default this reported F1 0.000

The same case in code

There is nothing language-specific to write, which is the point: Hungarian takes exactly the call English takes. What the code can do is keep the model revision from each finding, because that revision is what resolves to a per-language table.

any_language.py
# No language parameter exists. The same call covers all 26.crossing = scan_input(user_text, policy) for finding in crossing.findings:    # The revision ties this decision to the per-language    # evaluation published for the model that made it.    audit_log.write(finding.detector_id, finding.label,                    finding.score, finding.model_revision)

What the stamp holds

The model revision, which is what ties a decision to a specific set of per-language numbers. Six months later, an auditor asking how well this worked in Hungarian can resolve the revision in the stamp to the evaluation table that was published for it.

Read by anyone who has to defend the control in a market that is not the one it was built in.

What this does not catch

  • Maltese is absent from XLM-RoBERTa’s pretraining. No amount of training data fixes that, and it scores 0.000 on two detectors. It stays in the table.
  • The PII model now covers all twenty-six, but its per-language scores are from synthetic corpora, and a harder evaluation against real text is outstanding work.
  • The corpora are synthetic, generated per language rather than translated. That is what makes 26 languages affordable and it also means these are in-distribution results.
  • Some languages have very few positive test examples. For nsfw it is two per language, so a per-language F1 of 1.000 there is weak evidence, not a perfect score.

The full detector set, or the per-language numbers.