Toxicity
Configuration, evaluation scope, and practical limits for toxicity.
toxicity is the policy type used by the API and policy editor.
What it checks
Scores submitted LLM text with a toxicity classifier across configured categories.
Configure the policy
Create a policy under Dashboard → Security → Policies. Choose this policy type, set the agent scope, and start in warning mode while you measure results. Use blocking only after verifying the application’s decision gate.
The following is the config object, not a complete policy-creation request:
{
"threshold": 0.5,
"categories": [
"toxicity",
"severe_toxicity",
"identity_attack",
"insult",
"threat",
"obscene"
],
"check_output": true
}Scope and limits
Input and optional output are processed in bounded overlapping windows, with character and window budgets. Exceeding the coverage budget can make the evaluation degraded. A high score requires context review: quotations, counterspeech and clinical language can be false positives.
Verify before rollout
Test a representative safe input, a synthetic positive case, and an unavailable-detector case. Inspect evaluation_status, errors, policies_evaluated, findings and allowed. A clean empty finding list is useful only when the required checks completed.
The policy mode determines the decision: warning findings can coexist with allowed: true. Your application must act on a blocking decision before the protected operation. Evaluation does not automatically redact text or cancel an operation that already happened.