Sensitive action classification eval

Classifies committed agent actions (a button click or form submit, serialized as masked episode text) into the built-in contact-center fraud categories with Chrome's on-device Decision API, and compares it with TypeSafe Jev via OpenRouter on the same question (#12397). Try it live on the mock support console.

Dataset

View the episodes being tested as JSON

Open the raw file


  

The built-in set is a stratified sample of synthetic episodes from ai-behavior-detection (5 app markup styles × 4 phrasing families), serialized with its jevState masking, plus its real captured episodes. Classes outside the fraud set (send email, search, visit, transfer queue, …) are relabeled none: the monitor sees them constantly and must stay quiet. A custom file is an array of {"text", "expected"}, where expected is a category id or none; the console page exports one.

Question

Categories

Schema sent to DecisionModel.create()

    

Runners

Chrome Decision API (on-device)

Jev via OpenRouter (choice and effect questions; no field-kind pass)

Results

The veto and rescue apply to the answers a run collected, so untick them to score the same run without each. The prototype monitor runs both, at 0.3 and 0.6.

An event is what the monitor would record: a sensitive category fired. Precision is reported as measured and reweighted to 90% routine traffic, the live base rate the research assumes, because false events on routine actions are what would flood the security report.

Threshold sweep

Per category

Top confusions

Episodes