Our regulated-advice detector scores 0.983. Its recall is 0.75.

The regulated_advice detector separates explaining a financial instrument from recommending one. It scores a mean per-language F1 of 0.986 across all 26 target languages. That is the number that would have gone on a slide, and it is close to useless on its own.

What the headline number is averaging

It is the mean of 26 per-language F1 values, computed over 572 positive examples in the test split. Twenty-five of those languages sit above 0.95. The weakest is mt at 0.900, which is Maltese, and Maltese is absent from the base model's pretraining, so that one is a property of XLM-RoBERTa rather than of our data.

Read that way it looks like a solved problem. The label breakdown says otherwise.

The same test set, sliced by label

Precision is excellent: 0.950 on medical advice, meaning when it fires it is right. Recall is 0.91 and 0.86 and 0.86. Between a fifth and a quarter of the advice in the test set walks straight past it.

A guardrail with high precision and mediocre recall is a specific kind of dangerous. It never annoys anyone, so nobody turns it off, and it produces a clean evidence record every time it misses.

The label that has no test data at all

0 labels have zero positives in the test split: . That is the flagship case. The detector exists because a banking assistant should not tell someone which fund to buy, and the split we evaluated on contains no examples of it.

The 0.983 is not wrong. It is averaging over language, and language is not the axis on which this detector is weak. Nobody hid anything; the number was simply answering a question we were not asking.

Where it does hold up

The register breakdown is the part that survives scrutiny, and it is the part that was hardest to build. These are the positive registers:

And these are the hard negatives, written to sit as close to the class as possible without being in it. They contain no positives at all, so the only column that means anything is the false positive rate:

• definition at 0.0% false positives
• general_risk at 0.0% false positives
• historical_fact at 0.0% false positives
• hypothetical at 0.0% false positives
• mundane_account_access at 1.8% false positives
• mundane_informational at 0.0% false positives
• mundane_operational at 0.0% false positives
• mundane_transactional at 0.0% false positives
• process_description at 0.8% false positives

A definition of an ETF, a general statement about risk, a historical fact, a hypothetical, a description of a process. Those are the sentences a keyword filter destroys, and the detector leaves them alone almost perfectly. That is the thing worth reporting, and it is nowhere near the top of the page in the version of this we nearly published.

The corpus

13,502 examples across 26 languages, generated per language rather than translated, of which 7,788 are unlabelled negatives. Generated by claude-haiku-4-5 against prompt version advice_sys_v2_mundane, content hash 25b93152e30d. Synthetic and in-distribution, which is what makes 26 languages affordable and is also why none of this is a claim about production traffic.

The threshold is 0.88 and it is the uncalibrated default. This detector predates the calibration sweep the others went through. Given that recall is the weak axis, a lower threshold is the obvious next experiment, and it has not been run.

What we changed

The benchmarks page leads with the caveats rather than the results, and every detector shows its positive count next to its F1 so a reader can see how much evidence is behind a number. The three things on the fix list for this detector are: get positives for the empty label into the test split, calibrate the threshold against recall rather than accept the default, and evaluate against real transcripts instead of generated ones.

Until then the per-language table is published in full, including Maltese at 0.900.