Runs the flagged-term phrases through Chrome's on-device Decision API and through TypeSafe's Jev via OpenRouter, with the same question, and compares accuracy and latency (#11854). The default question is the one the Prohibited Language filter sends per matched term (#11927).
The Prohibited Language filter runs the Decision API only on phrases its keyword rule first matched, so this eval measures false positives on matched phrases. The eval asks the API about each phrase's own term, and the guard runs the API only for terms it matched, so ticking this hides phrases whose term the rule never fires on — buried in a word (Scunthorpe), split across words (“this hit the shelves”), or absent from the shipped corpus. (Such a phrase can still reach the API via a different term it does match.) The match is computed offline with the real guard (which normalizes leetspeak, symbols, and look-alikes), so the normalization itself is not shipped on this page.
Obfuscation phrases (for example @ss, s h i t, sh1t) disguise a
term to slip past a literal filter. Untick to exclude them: the extension normalizes text before
matching, so scoring them unnormalized here understates real-world accuracy. The
per-example results mark them, and the Scunthorpe false positives, in the
Challenge column.
A custom file is an array of {"text", "term", "expected"} where
expected is benign, inappropriate (or prohibited),
or ambiguous. Loading one discards the current runs.
The Decision API backends use the Gemma 3 SentencePiece tokenizer (262k vocabulary), the same one
Gemini uses. Load it to see how each phrase splits into tokens. The file is large, so it loads only
when you ask, from the public onnx-community mirror. A leading ▁ marks a word start.
Jev's tokenizer is not public, so this reflects the Decision API only.
Why this is the right tokenizer: EmbeddingGemma is built from Gemma 3, and Gemma 3 uses the 262k SentencePiece tokenizer shared with Gemini. The page loads the tokenizer from onnx-community/embeddinggemma-300m-ONNX, a Transformers.js conversion of that model that reuses its tokenizer files; Hugging Face's EmbeddingGemma announcement points to that mirror as the Transformers.js build. So the split matches the model, not a look-alike.
A phrase counts as positive when P(positive) is at or above the threshold. The built-in set has few positives, so precision and recall move a lot per phrase; ROC and PR AUC are threshold-free, and the step PR AUC is the most telling when positives are scarce.
Each curve sweeps every threshold, so its shape shows the accuracy-vs-false-alarm trade-off the AUC summarizes. The marked point is where the threshold slider above sits now; drag it to move the point along the curve. The dashed diagonal is chance.
Precision against recall as the threshold sweeps from high to low. Each example moves the line, so with few positives it zigzags. A true positive lifts precision, a false positive drops it. The step curve keeps those dips, and its area is the step PR AUC in the table. The interpolated curve lifts each point to the best precision at that recall or any higher recall, the PASCAL-VOC convention, which fills the dips in and gives a clean, always-descending curve whose area is the interpolated PR AUC. The dashed line is chance, the precision a no-skill classifier gets, which equals the positive rate. On the step curve the marked point is where the threshold slider above sits now.
Read the two together. The step curve shows the precision you actually get at each recall. The interpolated curve shows the precision you could get if you always picked the best threshold for that recall, which you cannot know ahead of time. So the interpolated AUC is always at least the step AUC, and the gap between them is how much the smoothing flatters the runner. To pick a classifier, trust the step number. The interpolated one is here to compare against VOC-style published results.
Loss is |P(positive) − truth|. A red cell is wrong at the threshold; an orange bar marks a phrase the two runners disagree on. The Challenge column marks Scunthorpe false positives (a term inside an innocent word), lexical ambiguity (a whole word used in a benign sense), and obfuscation attempts (a disguised term).