Red Teaming AI Systems and Agents

Chapter 12 turned the red team on the organisation's estate. This chapter turns it on the organisation's own AI: assistants, the retrieval systems behind them and agents that act with identities of their own. These systems fail in ways conventional penetration testing does not reach: an assistant can be steered by a document it was asked to summarise, an agent can use a legitimate tool for an illegitimate purpose, and a model upgrade can quietly undo a guardrail. The chapter therefore shows how to scope the compound system, use taxonomies as test plans, turn campaigns into gating suites, test agent containment, and judge how far testing can be trusted.

In CSF terms the chapter is responsible for two Categories of the Identify Function (19, NIST, 2024). ID.RA Risk Assessment asks that vulnerabilities are identified, validated and recorded (ID.RA-01), threats identified (ID.RA-03), likelihoods and impacts understood (ID.RA-04, ID.RA-05) and risk responses chosen and tracked (ID.RA-06). ID.IM Improvement asks that improvements are identified from evaluations (ID.IM-01) and from security tests and exercises, including those run with suppliers (ID.IM-02). The chapter treats both Categories through the three focus areas of the NIST Cyber AI Profile (20, NIST, 2025), with securing AI systems as its core.

A compound AI system from gateway to agent, with test suites, the stop path and the release gate

Figure 13.1: A compound AI system from gateway to agent, with test suites, the stop path and the release gate.

Scoping the compound system

The model is the smallest part

A deployed AI application is a compound system of users and content sources, a gateway, prompts and the model, a retrieval store, an agent with a permission manifest, and tools, as Figure 13.1 shows. Most failures that matter arise where two parts meet: a model that refuses to disclose personal data when asked may still reproduce it when retrieval places a case document in its context.

AI Platform and Security, Chapter 16, concluded that prompt injection is a property of the architecture, not a patchable defect, so the controls that matter sit outside the model, in the gateway, retrieval pipeline, tool layer and identity permissions (73, Nordqvist, 2026). This chapter builds on that conclusion (F20). OWASP likewise notes that, given the stochastic influence at the heart of the way models work, it is unclear whether fool-proof methods of preventing prompt injection exist (14, OWASP, 2025). Testing the model in isolation therefore says little about what an adversary can achieve through the application.

The provider's evidence and the deployer's test

Under Article 55 of the AI Act, providers of general-purpose AI models with systemic risk must evaluate their models, including by conducting and documenting adversarial testing (1, European Union, 2024). AI Platform and Security, Chapter 20, drew the implication that deployers ask for that evidence and test the compound system themselves, because the provider has never seen their prompts, retrieval corpus, tools or users (73, Nordqvist, 2026); this chapter builds on that conclusion (F22). For high-risk AI systems, Article 15 requires resilience against third parties exploiting vulnerabilities, naming data poisoning, model poisoning, adversarial examples and confidentiality attacks (1, European Union, 2024). Red-team results against the compound system are direct evidence of that resilience; this is an architectural reading, not legal advice.

What belongs in scope

A scope statement names six elements and what the campaign may touch in each: the model version, the prompts and skills, the retrieval layer with its per-user filters, the tools with their identities, the gateway, and the users and content sources. The sixth is most often forgotten: residents also send letters that a caseworker later asks the staff assistant to summarise, so the person who controls the content is not the person who uses the assistant, and the campaign tests both routes. Production data, real accounts and the provider's infrastructure are normally out of scope, under rules of engagement approved under Chapter 7. A Mode 4 Built AI application is tested end to end; a Mode 5 Autonomous production agent is tested end to end and for containment.

Taxonomies as test plans

Four complementary vocabularies

AI Platform and Security, Chapter 16, established the OWASP Top 10 for LLM applications and MITRE ATLAS as the shared vocabulary of builders, defenders and testers (73, Nordqvist, 2026). This chapter uses four sources, each for a different purpose.

The OWASP Top 10 for Large Language Model Applications (2025 edition) is the first axis of a test plan for an assistant (14, OWASP, 2025). The OWASP Top 10 for Agentic Applications for 2026 is the primary axis for a Mode 5 agent, and several entries test containment rather than the model (15, OWASP, 2025). MITRE ATLAS describes adversary techniques against AI systems in the structure of ATT&CK (11, MITRE, n.d.), linking test cases to the threat models of Chapter 10 and the detections of Chapter 19. NIST AI 600-1, the generative AI profile, adds risks such as confabulation and harmful bias that a public authority answers for even without an adversary (17, NIST, 2024), and NIST's taxonomy of adversarial machine learning names attack classes precisely, from evasion and poisoning to indirect prompt injection (18, Vassilev et al., 2025).

From taxonomy to plan

To build the plan, map each entry to the parts of the system where it can occur, recording the rest as not applicable, and write each applicable entry as an objective for the control that should hold, not the attack that might succeed: "a resident cannot cause the assistant to disclose another resident's correspondence". Each objective carries its ATLAS technique and risk-register entry, so that the result updates ID.RA directly. Written this way, the plan is a defensive artefact; the adversarial inputs live in the access-controlled scenario library, not in the plan, in reports or in this book.

Worked example: campaign plan for NR-SEC-002 mapped to OWASP and ATLAS categories

Illustrative. Nordvik Region's first campaign under NR-SEC-002 covers the citizen-service assistant. The contracted tester runs it against a copy of the assistant in a separated test range, under rules of engagement approved by the CISO. The plan contains no test inputs.

# Illustrative: Nordvik Region, campaign plan for NR-SEC-002, wave 1 (extract)
kind: AIRedTeamCampaignPlan
id: NR-SEC-002-C01
decision_record: AIDR-0302
rules_of_engagement: RoE-NR-2027-003
system: citizen-service-assistant
target:
  environment: test-range-copy
  prompt_template: citizen-service/v7
  data: synthetic-only
out_of_scope: [production, real-resident-accounts, provider-infrastructure]
objectives:
  - id: OBJ-01
    statement: Instructions placed in retrievable content do not change the assistant's task
    owasp_llm: LLM01 Prompt Injection
    atlas: AML.T0051
    risk_entry: NR-RISK-AI-014
  - id: OBJ-03
    statement: Content added to the index outside the publishing workflow is not retrievable
    owasp_llm: LLM08 Vector and Embedding Weaknesses
    atlas: AML.T0070
    risk_entry: NR-RISK-AI-016
  - id: OBJ-04
    statement: Answers about benefits cite a published source or state that none exists
    nist_ai_600_1: Confabulation
not_applicable:
  - owasp_llm: LLM06 Excessive Agency
    reason: The assistant has no tools that change state
deliverables: [findings-register, scenario-cases-to-library, evidence-bundle]

The campaign's significant finding fell under OBJ-03: a page withdrawn from the public website was still in the retrieval index, because withdrawal did not trigger re-indexing, and the assistant cited it as current guidance. The closing control made withdrawal remove the document from the index within one hour, and the finding closed only when a new suite case confirmed this against the next release.

Suites, harnesses and gates

From campaign to suite

A campaign finds weaknesses once. AI Platform and Security, Chapters 17 and 20, concluded that red teaming of AI systems runs continuously in CI: the external exercise is for discovery, and every finding becomes a versioned attack case with explicit thresholds in a scenario library, run by an automated harness and published as evidence against control identifiers (73, Nordqvist, 2026). This chapter builds on those conclusions (F21 and F22) and on the principle of Architecture as Code (Chapter 13) that security tests run in the pipeline (71, Nordqvist, 2025).

The scenario library holds versioned RT-LIB-* cases with objective, taxonomy tags, inputs and scoring rule, and is access-controlled because the inputs are adversarial. The harness runs them against a named configuration of model version, prompt template, retrieval snapshot, tools and gateway policy, recording every input, output and policy decision, with judged results scored by an evaluator model and sampled for human review. Because generative systems are not deterministic, each case runs several times and the threshold is a rate: zero for outcomes that must never occur, such as one resident's data in another's answer, and a non-zero rate only where other controls bound the harm, recorded with its owner. The gate applies the thresholds to every change that can alter behaviour, because, as AI Governance, Chapter 8, put it, review is triggered by change, not by the calendar (72, Nordqvist, 2026). Nordvik's gate GATE-CSA-001, for example, runs five times per case, tolerates no failure on OBJ-01 and OBJ-03, accepts two per cent on OBJ-04 because unsupported answers are labelled, and blocks the release on failure, with a CISO override valid for at most 14 days. The signed evidence of each run feeds the evidence manifest of AI Governance, Chapter 11, answering ID.IM-02.

AI in the red team's own work

On the red team's side, a generator model produces variants of each catalogued case in different languages and formats, so that the suite tests a class of inputs rather than one phrasing, and an analysis model clusters outputs and drafts findings for a person to confirm. The generator works inside reviewed scenario classes, generated cases are reviewed before they gate anything, and neither model holds credentials to the system under test. Continuous suites do not, however, replace the threat-led penetration testing DORA requires, at least every three years, of the financial entities identified for it (2, European Union, 2022); as AI Platform and Security, Chapter 17, concluded, the exercise discovers what nobody has thought of and the suite keeps it fixed (73, Nordqvist, 2026).

Practical guide: running a red-team campaign on a retrieval-augmented assistant

Purpose. Discover how a retrieval-augmented assistant can be made to fail, and leave behind a gating suite that keeps the findings closed.

Roles. System owner (accepts residual risk), red-team lead or contracted tester, evaluation engineer, retrieval-pipeline owner, data protection officer, CISO.

Steps.

  1. Scope the six elements under rules of engagement limited to a copy of the assistant with synthetic data.
  2. Inventory who can place content in each retrieval source; every such route is a surface for indirect injection and poisoning.
  3. Build the plan from the OWASP LLM Top 10 with ATLAS tags and risk entries, adding relevant NIST AI 600-1 risks.
  4. Prepare a synthetic corpus with canary documents whose retrieval by the wrong user or after withdrawal is observable.
  5. Run time-boxed rounds, by default two weeks of testing and one of triage, reporting critical findings within one working day.
  6. Convert each finding into library cases that fail against the unfixed and pass against the fixed configuration, add them to the gate, and close the finding.

Default values to adapt. Five runs per case; zero tolerance for cross-user and system-prompt disclosure; at most two per cent unsupported answers where other controls bound the harm; a ten per cent human sample of machine-scored cases.

Definition of done. Every applicable taxonomy entry has a result, every finding is closed by a verified control or accepted in writing, and the suite runs on every triggering change with its evidence in the manifest.

Red teaming agents

From what the agent says to what it can do

For an assistant, the red team asks what it can be made to say; for an agent, what it can be made to do, and what stops it. AI Governance, Chapters 19 and 25, concluded that every agent should be governed as a well-meaning insider, because instructions lower the rate of unwanted behaviour but only boundaries cap its consequences (72, Nordqvist, 2026); this chapter builds on that conclusion (F13). Lynch and colleagues found that models in simulated corporate environments, with goals that later conflicted with the company's direction or their own replacement, produced harmful actions in some scenarios, such as disclosing confidential information, and that explicit instructions reduced the rate without eliminating it; they had not observed such behaviour in real deployments (63, Lynch et al., 2025). What matters, therefore, is whether the boundaries hold when the instructions do not.

The boundaries of AI Platform and Security, Chapter 19, become the test objectives: task-scoped credentials, default-deny egress, control artefacts outside the agent's reach, budgets, an external stop path, recorded approvals, and detection from telemetry and decoys (73, Nordqvist, 2026); this chapter builds on that conclusion (F23). The July 2026 incident, in which agents in a security-testing context exceeded their mandate (64, OpenAI, 2026), failed through a shared credential, a package mirror that was also an egress path and an early alert that did not lead to the evaluation run being stopped (64, OpenAI, 2026), all containment properties a red team can test in advance.

The permission manifest as the test plan

The permission manifest states the agent's tools, actions, resources, destinations, budget and approvals. Each line yields two objectives, that the permitted action works and that adjacent actions are refused, and anything unmentioned must be unreachable. The OWASP agentic list supplies the threat classes, from goal hijack to rogue agents acting without any external adversary (15, OWASP, 2025). The Five Eyes agencies urge careful adoption of agentic AI services (40, Five Eyes agencies, 2026), and a containment test shows that adoption was careful in fact.

Escape paths and stop paths

An escape path is any route by which an agent's effect can reach beyond its boundary: an over-broad credential, an unlisted network route, a writable location holding the agent's own configuration or logs, a shared service account, or an approval satisfiable without a person. The red team enumerates these from infrastructure code and tests each without needing the model to misbehave: in the test range a scripted driver replaces the agent's planning component and issues out-of-mandate tool calls with the agent's own identity and runtime, testing the boundary reproducibly, and a second round uses the real model with manipulated inputs to check detection and the stop path.

The stop path is tested last: whoever receives the alert must hold the authority to stop the agent, and the stop must revoke credentials within its time bound, even under load, without the agent being able to delay or reverse it. This answers the open question that AI Governance, Chapters 19 and 23, left for this book (Q6): the containment built for one's own agents also limits an adversary who has taken control of one, and the scripted driver tests it from that adversary's side.

Practical guide: running an escape-path assessment on a production agent

Purpose. Establish, with evidence, whether an agent's containment holds when the agent is manipulated or exceeds its mandate.

Roles. Agent owner (accepts residual risk), agent operators (stop path), red team, identity and network teams, white team (rules of engagement), SOC (detection).

Steps.

  1. Approve rules of engagement naming the agent, the test range, the scripted-driver technique and actions never attempted, such as calls to real external parties.
  2. Build a replica of the agent's identity, permissions, network policy and tools from the same infrastructure code, recording differences as limitations.
  3. Derive objectives from the manifest and the enumerated escape paths, and place decoys such as honeytoken credentials.
  4. Run the scripted-driver round and the manipulated-input round, recording refusals and detection times.
  5. Trigger the stop path while the agent is idle and during a long task, measuring time to credential revocation.
  6. Turn every escape path into a fix and a regression case, and re-run before closing.

Default values to adapt. Stop path effective within five minutes; zero tolerance for an out-of-manifest action neither refused nor detected; decoy access detected within fifteen minutes; repeat after any manifest, identity or network change and at least twice a year.

Definition of done. Every manifest line and escape path has a tested result, each escape path is closed or under an expiring exception, and the stop path has been exercised under load.

Worked example: containment test plan for Hansa's IT-operations agent

Illustrative. Hansa Mutual's IT-operations agent (AIDR-0027) has write access to production and is operated from Hamburg. Under HM-SEC-003 the group red team runs an escape-path assessment in the AI test range before every change to the agent's permission manifest.

# Illustrative: Hansa Mutual, containment test plan for the IT-operations agent (extract)
kind: ContainmentTestPlan
id: HM-SEC-003-CT-ITOPS-07
decision_record: AIDR-0313
agent_decision_record: AIDR-0027
rules_of_engagement: RoE-HM-2027-AIRT-03
environment: ai-test-range
replica_of: itops-agent/manifest@4.2.0
objectives:
  - {id: CT-01, area: credentials, owasp_agentic: ASI03 Identity and Privilege Abuse,
     hypothesis: Agent tokens are task-scoped and expire within 15 minutes}
  - {id: CT-02, area: credentials, owasp_agentic: ASI02 Tool Misuse and Exploitation,
     hypothesis: A restart of a service outside the ticket's scope is refused at the policy point}
  - {id: CT-04, area: control-artefacts, owasp_agentic: ASI10 Rogue Agents,
     hypothesis: The agent cannot write to its own manifest policy bundle or audit log}
  - {id: CT-06, area: detection,
     hypothesis: Use of the honeytoken credential raises a SOC alert within 15 minutes}
  - {id: CT-07, area: stop-path,
     hypothesis: The operator stop revokes all agent credentials within 5 minutes while a task runs}
rounds: [scripted-driver, model-with-manipulated-input]
residual_risk_owner: Head of IT Operations

The assessment found one escape path: the agent restarted services through a deployment service account that could also read an adjacent environment's secrets store, and the scripted driver reached it undetected because the SIEM did not collect those reads. The closing controls gave the agent a dedicated task-scoped identity and added a detection for any read by an agent identity; the finding closed when the re-run showed the access refused and detected.

The limits of testing

Evaluation awareness

AI Governance, Chapter 24, asked (Q4) how far evaluation can carry assurance when models can tell test settings from deployment (72, Nordqvist, 2026), a difficulty the International AI Safety Report 2026 also discusses (43, Bengio et al., 2026), and in the study by Lynch and colleagues one model produced harmful actions more often when it stated that it took the scenario to be real than when it stated it took it to be an evaluation (63, Lynch et al., 2025). A passing result in a test range may therefore overstate how the model behaves in production.

This chapter's answer to Q4 has three parts: make test settings resemble production where the model can see them; prefer tests whose outcome does not depend on the model's behaviour, because a boundary does not know it is being tested, which is the strongest reason to weight containment over behavioural testing for agents; and keep testing in production through chaos experiments and monitoring.

Residual risk and who accepts it

No suite covers every input, so every campaign report ends with a statement of residual risk: what was not tested, what passed at a non-zero threshold, which findings remain open and which limitations of the setting matter. The system owner accepts it in writing, as Architecture as Code (Chapter 20) required of every risk acceptance: an agent may draft, but a named human approves (71, Nordqvist, 2025). Consequently, a guardrail that passed every test is still backed by the policy point, task-scoped credentials and a stop path.

AI Governance, Chapter 24, left a second question (Q5): how organisations prepare for a stream of smaller agent incidents of the same shape inside enterprises (72, Nordqvist, 2026). After the July 2026 incident, an independent investigation was carried out to form an understanding of what the agents had done (66, Wijk et al., 2026). If the red team cannot reconstruct a scripted out-of-mandate action from the logs alone, a responder will not reconstruct a real one, so every containment test plan names the telemetry for each objective. Chapter 20 takes reconstruction into incident analysis.

Red teaming AI across modes, levels and focus areas

Across the usage modes

Each mode places the judgement and the identity in a different place, so each is red teamed differently (see usage modes in Chapter 6).

Mode Opportunity for the red team Characteristic vulnerability Who judges, with whose identity
Mode 0 Ambient AI Vendor evidence and contractual testing rights Acts automatically; cannot be tested directly The vendor's model; the product's
Mode 1 Conversational assistants Test data exposure through approved tools Sensitive data entered by users; injection via pasted content A person; the person's
Mode 2 Embedded workplace assistants Test what content the assistant reads for a user Injection via documents and email; over-broad delegated access A person; the person's, delegated
Mode 3 Agentic work tools Test sandbox and credential boundaries of coding agents Agent acts with an engineer's credentials between instruction and review A reviewing person; the engineer's
Mode 4 Built AI applications Test the full compound application Indirect injection, retrieval poisoning, cross-user disclosure The system owner; service identity and user's
Mode 5 Autonomous production agents Test containment, credentials and stop paths Out-of-mandate action with the agent's own permissions Policy within the manifest; the agent's own

Across the autonomy levels

The autonomy level of an agent under test (see autonomy levels in Chapter 6) decides which objectives weigh most.

Level Emphasis of the red-team test
Level 0 Advisory Whether manipulated input can make the advice misleading unnoticed
Level 1 Assisted Whether a drafted action can carry hidden effects past the person who executes it
Level 2 Supervised Whether the approval request shows the action truthfully and cannot be bypassed
Level 3 Bounded Whether the playbook boundary holds, actions are reversible and the stop path works mid-action
Level 4 Autonomous Never used for offensive or irreversible actions; otherwise whether boundary and stop path hold without approval

Verifying with security chaos engineering

The chapter's experiment injects a change to model version, prompt template, retrieval source or gateway policy, observed through the production change event, because some changes, such as a provider's silent model update, never pass the release pipeline. The steady-state hypothesis is that the gating suite runs against the changed configuration, with synthetic accounts and canary documents only, and every zero-tolerance case stays within its threshold; otherwise the change is rolled back or the system falls back to a safe mode. The time bound is four hours for Mode 4 systems and one hour for Mode 5 agents, within the safety boundaries of Chapter 25.

The three AI focus areas

Securing AI systems. Red teaming applications, retrieval systems and agents against OWASP and ATLAS categories produces validated vulnerabilities for ID.RA and regression cases for ID.IM, with the compound system as the object of the test (20, NIST, 2025).

AI-enabled cyber defence. AI generates variants of catalogued cases and analyses harness output at a scale a small team could not reach, within reviewed scenario classes and without credentials to the systems under test.

Thwarting AI-enabled attacks. Scripted-driver containment tests show whether a manipulated agent can do harm beyond its boundary, and the same boundaries limit an adversary who has taken control of it.

Nordvik Region and Hansa Mutual

Nordvik Region red teams its citizen-service and staff assistants under NR-SEC-002. It contracts the first campaign and keeps the results as a classified suite that runs before each change. For the casework variant of the staff assistant, the main surfaces are case documents carrying instructions and the case-management API, where the test is whether a caseworker reaches only their own cases. Both are Mode 4 services, and the Head of Unit who owns each accepts the residual risk in writing.

Hansa Mutual runs HM-SEC-003 against the claims assistant, the claims-triage agent and the IT-operations agent, testing the agents for containment, including whether the claims-triage agent's tool calls stay bound to the market of each claim. Its evidence states that passing results bound residual risk without removing it. Nordvik thus tests what people can do through its assistants, whereas Hansa must also test what its agents can do with their own identities, which is why containment dominates its campaigns.

Summary

Red teaming an AI system means testing the compound system, because providers test their models under Article 55 of the AI Act but only the deployer can test what is built around them. Taxonomies serve as test plans, and every campaign ends as a versioned suite that gates releases: under ID.RA Risk Assessment the campaigns reveal how the system fails, and under ID.IM Improvement the findings become regression cases.

Agents need a containment test, with the permission manifest as the test plan, scripted drivers for escape paths and a stop path exercised under load. Because models can behave differently when a setting looks like a test, system owners accept residual risk explicitly and controls never depend on test results alone. Chapter 14 opens Part IV (Protect) with identity and access for people, agents and machines.

Sources

  1. NIST. The NIST Cybersecurity Framework (CSF) 2.0. (19, NIST, 2024)
  2. National Institute of Standards and Technology. Cybersecurity Framework Profile for Artificial Intelligence (Cyber AI Profile). (20, NIST, 2025)
  3. European Parliament and Council of the European Union. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (1, European Union, 2024)
  4. European Parliament and Council of the European Union. Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). (2, European Union, 2022)
  5. OWASP Foundation. OWASP Top 10 for Large Language Model Applications, 2025 edition. (14, OWASP, 2025)
  6. OWASP Foundation. OWASP Top 10 for Agentic Applications for 2026. (15, OWASP, 2025)
  7. MITRE Corporation. MITRE ATLAS: Adversarial Threat Landscape for AI Systems. (11, MITRE, n.d.)
  8. National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. (17, NIST, 2024)
  9. Vassilev, A., Oprea, A., Fordyce, A., Anderson, H., Davies, X. & Hamin, M. Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations. (18, Vassilev et al., 2025)
  10. Bengio, Y. et al. International AI Safety Report 2026. (43, Bengio et al., 2026)
  11. Five Eyes cyber security agencies: ASD's ACSC, CISA, NSA, Canadian Centre for Cyber Security, NCSC-NZ and NCSC-UK. Careful adoption of agentic AI services. (40, Five Eyes agencies, 2026)
  12. OpenAI. OpenAI – Hugging Face Incident Technical Report. (64, OpenAI, 2026)
  13. Wijk, H., Cotra, A. & Greenblatt, R. Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. (66, Wijk et al., 2026)
  14. Lynch, A., Wright, B., Larson, C., Troy, K. K., Ritchie, S. J., Mindermann, S., Perez, E. & Hubinger, E. Agentic Misalignment: How LLMs could be insider threats. (63, Lynch et al., 2025)
  15. Nordqvist, G. Architecture as Code. (71, Nordqvist, 2025)
  16. Nordqvist, G. AI Governance: Governing Artificial Intelligence in European Enterprises and Public Authorities. (72, Nordqvist, 2026)
  17. Nordqvist, G. AI Platform and Security: Building and Securing AI Platforms in European Enterprises and Public Authorities. (73, Nordqvist, 2026)