Decision API vs Jev eval

Disclaimer: this page benchmarks prohibited-language detection. PixieBrix does not condone the use of abusive language against others. The built-in dataset contains offensive words solely as test data.

Runs the flagged-term phrases through Chrome's on-device Decision API and through TypeSafe's Jev via OpenRouter, with the same question, and compares accuracy and latency (#11854). The default question is the one the Prohibited Language filter sends per matched term (#11927).

Dataset

A custom file is an array of {"text", "term", "expected"} where expected is benign, inappropriate (or prohibited), or ambiguous. Loading one discards the current runs.

Question

Each runner gets the phrase text as the subject of the decision. The default prompt calls it "this message", so it does not use {text}. Add {text} only if you want the phrase in the question itself, for example to try different wording. {category} is the term's prohibited-language category: profanity, slurs_hate, sexual, or threats. It is the corpus category. If the shipped corpus does not include the term, it is the closest match. {category} is empty only if a dataset you load does not set it. You can use the same placeholders in the option labels and descriptions below.

Choice-question options

Labels and descriptions apply only to choice questions. Boolean (Decision API) and noul (Jev) score the prompt alone. They send no options, so they ignore these fields. Put any per-option context in the prompt.

Runners

Chrome Decision API (on-device)

Jev via OpenRouter

Tokenization

The Decision API backends use the Gemma 3 SentencePiece tokenizer (262k vocabulary), the same one Gemini uses. Load it to see how each phrase splits into tokens. The file is large, so it loads only when you ask, from the public onnx-community mirror. A leading ▁ marks a word start. Jev's tokenizer is not public, so this reflects the Decision API only.

Why this is the right tokenizer: EmbeddingGemma is built from Gemma 3, and Gemma 3 uses the 262k SentencePiece tokenizer shared with Gemini. The page loads the tokenizer from onnx-community/embeddinggemma-300m-ONNX, a Transformers.js conversion of that model that reuses its tokenizer files; Hugging Face's EmbeddingGemma announcement points to that mirror as the Transformers.js build. So the split matches the model, not a look-alike.

Results

A phrase counts as positive when P(positive) is at or above the threshold. The built-in set has few positives, so precision and recall move a lot per phrase; AUC is threshold-free.

ROC curve

Each curve sweeps every threshold, so its shape shows the accuracy-vs-false-alarm trade-off the AUC summarizes. The marked point is where the threshold slider above sits now; drag it to move the point along the curve. The dashed diagonal is chance.

Per-example results

Loss is |P(positive) − truth|. A red cell is wrong at the threshold; an orange bar marks a phrase the two runners disagree on.