Reference
Measured performance
Generated from the library's own benchmark run.
Generated from the library's own benchmark run.
Every score below carries the number of examples behind it, and the n means
column says what those examples are. A count of positives is not a count
of everything evaluated, and the two are not interchangeable. Where the
evaluation recorded no count at all the cell reads not recorded, which
means the score is unverified rather than good or bad.
Detectors
| Detector | Tier | Status | Metric | Macro | Median | Worst | Test cases | n means |
|---|---|---|---|---|---|---|---|---|
banned_terms | T1 | built | – | – | – | – | – | – |
bias | T2 | built | f1 | 0.977 | 1.000 | 0.824 | 264 | positive examples |
code_present | T1 | built | – | – | – | – | – | – |
disclosure | T0 | built | – | – | – | – | – | – |
encoded_payload | T1 | built | – | – | – | – | – | – |
gibberish | T1 | built | f1 | 0.966 | 0.958 | 0.870 | 276 | positive examples |
groundedness | T3 | not built | – | – | – | – | – | – |
injection | T2 | built | f1 | 0.970 | 0.983 | 0.727 | 357 | positive examples |
internal_domains | T1 | built | – | – | – | – | – | – |
invisible_text | T0 | built | – | – | – | – | – | – |
json_schema | T1 | built | – | – | – | – | – | – |
language_id | T1 | built | – | – | – | – | – | – |
markup_injection | T1 | built | – | – | – | – | – | – |
moderation | T2 | not built | – | – | – | – | – | – |
nsfw | T2 | built | f1 | 0.934 | 0.952 | 0.600 | 259 | positive examples |
output_format | T1 | built | – | – | – | – | – | – |
output_leakage | T1 | built | – | – | – | – | – | – |
pii | T1 | built | – | – | – | – | – | – |
politeness | T2 | built | f1 | 0.962 | 0.968 | 0.788 | 392 | positive examples |
postal_code | T1 | built | – | – | – | – | – | – |
regulated_advice | T2 | built | f1 | 0.995 | 1.000 | 0.957 | 622 | positive examples |
repetition | T1 | built | – | – | – | – | – | – |
secrets | T0 | built | – | – | – | – | – | – |
sql_injection | T1 | built | – | – | – | – | – | – |
summary_support | T1 | built | – | – | – | – | – | – |
system_prompt_leakage | T1 | built | – | – | – | – | – | – |
token_limit | T1 | built | – | – | – | – | – | – |
topic_scope | T3 | built | top1_accuracy | 0.865 | 0.857 | 0.375 | 175 | examples evaluated |
toxicity | T2 | built | f1 | 0.992 | 1.000 | 0.950 | 518 | positive examples |
url_reachability | T3 | built | – | – | – | – | – | – |
Caveats
bias: 12 of 26 languages have fewer than 10 positive examples: az, bg, da, de, el, fi, ga, hu, lv, mt, pl, sk. Their individual scores are indicative rather than measured.gibberish: 2 of 26 languages have fewer than 10 positive examples: bg, en. Their individual scores are indicative rather than measured.nsfw: 1 of 26 languages have fewer than 10 positive examples: ga. Their individual scores are indicative rather than measured.topic_scope: 26 of 26 languages have fewer than 10 examples evaluated: az, bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv, tr. Their individual scores are indicative rather than measured.topic_scope: no calibrated threshold recorded, so this detector runs at the policy default. Several detectors in this family reported nothing at 0.5 while separating positives from negatives well below it.
Latency
At a 396 character reference input, 1 thread, CPUExecutionProvider. Romanian prose with no entities in it, so this measures the cost of looking rather than the cost of finding.
| Detector | p95 ms | Budget ms | note |
|---|---|---|---|
banned_terms | 0.168 | 5.0 | – |
bias | 21.327 | 225.0 | – |
code_present | 0.009 | 5.0 | – |
disclosure | 0.038 | 5.0 | – |
encoded_payload | 0.205 | 5.0 | the unconfigured path |
gibberish | 27.528 | 225.0 | – |
injection | 27.622 | 225.0 | – |
internal_domains | 0.234 | 5.0 | – |
invisible_text | 0.038 | 5.0 | – |
json_schema | 0.001 | 5.0 | – |
language_id | 0.362 | 5.0 | – |
markup_injection | 0.239 | 5.0 | – |
nsfw | 27.544 | 225.0 | – |
output_format | 0.001 | 5.0 | – |
output_leakage | 27.991 | 225.0 | – |
pii | 27.636 | 225.0 | – |
politeness | 30.319 | 225.0 | – |
postal_code | 0.002 | 5.0 | – |
regulated_advice | 30.430 | 225.0 | – |
repetition | 0.497 | 5.0 | – |
secrets | 0.048 | 1.0 | – |
sql_injection | 0.238 | 5.0 | – |
summary_support | 0.921 | 5.0 | – |
system_prompt_leakage | 0.318 | 5.0 | the unconfigured path |
token_limit | 0.001 | 5.0 | the unconfigured path |
topic_scope | 46.298 | 300.0 | – |
toxicity | 27.224 | 225.0 | – |
url_reachability | 0.007 | 3000.0 | the unconfigured path |
This page is generated from docs/reference/performance.md in the library repository. Read it as markdown, or edit it at the source.