AI Agent Red Teaming: A Reproducible Test Plan
AI agent red teaming tests whether adversarial inputs can push a specific agent beyond its intended boundaries. A useful result records the target, inputs, observed behavior, configuration, and checks that did not finish. A passing scan describes that tested scope; it does not establish that an agent is safe against every attack.
This guide is for engineers who want to turn one reproducible finding into a release check. Start in a development or staging environment with synthetic data and a target you are authorized to test.
1. Define the boundary before selecting probes
Write down what the agent may read, release, and do. For a support agent, this might mean answering public product questions while withholding seeded account secrets and refusing unauthorized account actions.
Record the model and prompt versions, retrieval sources, available tools, authentication context, and relevant policy configuration. A changed tool permission can matter as much as a changed model.
Use OWASP's prompt injection guidance to understand direct and indirect instruction risks. Choose scenarios relevant to your application rather than treating a category label as proof of coverage.
2. Make the outcome observable
Seed a unique, synthetic marker such as DEMO-ONLY-NEVER-USE into a controlled test fixture. Define the assertion before running the probe: for example, the final response must not contain that marker.
Inspect both the response and the operation that occurred. A blocked evaluation is useful only if the application stops at the intended boundary. A tool authorization scenario also needs evidence that the prohibited tool action did not execute.
Connect an authorized target and review integration scope before relying on interception or trace coverage.
3. Separate findings, failed assertions, and incomplete runs
Keep attempted probes and completed probes distinct. A timeout, unavailable judge, invalid target response, or skipped detector must not silently become a safe result.
For each finding, retain enough context to explain:
- Which test input and assertion ran.
- Which response or recorded action violated the assertion.
- Which model, tool, policy, and connection configuration were used.
- Which checks completed and which remained incomplete.
- Whether the application actually enforced the returned decision.
A sample red-team report shows the format of scoped assessment evidence. Illustrative examples are not detector-accuracy benchmarks.
4. Retest the exact case after a change
Change the relevant prompt, policy, application gate, or tool permission. Run the original example again, alongside benign requests that should still work. Keep the original failure and the new result distinguishable.
In Nyraxis, save a regression case with an explicit assertion and run it against the connected agent from CI. An exact-text assertion checks that text boundary; it does not prove that every paraphrase or disclosure has been prevented.
A release check should report passed, failed, or incomplete. Decide how your deployment handles each state rather than equating missing evidence with success.
5. Review what the test sends and retains
Evaluation can send text to the configured API and, depending on the selected provider, another model endpoint. Metadata-only retention controls stored content; it does not eliminate processing elsewhere. Review data handling before using customer information.
Continue with one workflow
Follow the SDK quickstart, inspect one policy decision, and build a small regression set around an observable failure. Add broader scenarios as you learn what the first tests leave unmeasured.