Regressions and CI
Save a finding as a repeatable check and keep failed or incomplete runs out of a release gate.
Agent regression versus Test Lab
A connected-agent regression sends an input to a configured agent target and checks the returned result against an assertion. A Test Lab suite evaluates its saved policy cases. Choose the path that matches the behavior you need to retest.
A passing policy case does not prove a deployed agent enforced that decision. A passing agent regression proves only its saved assertion against the executed target/configuration.
Save and retest a finding
- Inspect a finding from a completed assessment.
- Choose Save as agent regression and name the behavior.
- Review the saved input and explicit assertion, such as text that must not appear in the response.
- Run it against the connected target and inspect the result.
- Change the policy or application, save the new version, and rerun the same case.
For a synthetic password disclosure, assert that DEMO-ONLY-NEVER-USE must not appear. A controlled refusal can satisfy that specific assertion; it does not prove that every tool was authorized or that all output is safe.
Metadata-only trace storage is not a recoverable prompt archive. Review what the regression creation flow deliberately retains, and supply missing input when needed.
Read run status
| State | Interpretation |
|---|---|
| Queued / running | Work has not produced a final assertion result. |
| Passed | The completed run satisfied the saved assertion. |
| Failed assertion | The target ran, but the assertion was not satisfied. |
| Incomplete / failed job | The target, detector, worker, or assertion evaluation could not complete. |
Never treat a missing run, empty suite, timeout, or unavailable detector as a passing release gate.
Run from CI
Create a separate test-scoped key. The backend repository includes scripts/nyraxis_ci.py, which submits a saved suite or regression, polls the durable job, and emits a metadata-only summary.
python -m pip install httpx
python scripts/nyraxis_ci.py --helpUse --suite-id or --regression-id with the saved UUID and --base-url with your backend origin. Set NYRAXIS_API_KEY through CI secret injection. Confirm the exact available options with --help for the checkout you deploy.
| CLI exit code | Meaning |
|---|---|
0 | Completed passing run. |
1 | Completed assertion failures or regressions. |
2 | Incomplete run, invalid receipt, service/access error, or timeout. |
The CI API lives under /api/v1/testing/ci. Suite and regression submission use different paths, and each has its own job-polling endpoint. Do not use a dashboard session endpoint with the test key.
Retries and workers
Submission supports a bounded Idempotency-Key. Reuse the same request identifier when recovering the same submission; use a new identifier for an intentional new run. A receipt identifies a job, never a clean evaluation result. Inspect history after an uncertain response before triggering more target operations.
Keep the workload worker, dispatcher, scheduler, database, and Redis healthy. See operations and troubleshooting.