Purple Teaming, Measurement and Continuous Improvement

The chapters before this one produce findings: a step that reached its target unseen, a guardrail that a crafted document can steer, a log source that fell silent, a playbook that assumed a human tempo. A finding is valuable only if it changes something, and a change only if someone has shown that it works. This chapter therefore closes the loop across all six CSF Functions: the purple loop, the joint sessions in which red and blue teams run it with AI, the metrics and evaluations that show whether defensive AI improves outcomes, and the programme that governs improvement.

In CSF 2.0 terms the chapter is responsible for two Categories (19, NIST, 2024). ID.IM Improvement asks that improvements are identified across all Functions from evaluations (ID.IM-01), from tests and exercises, including those with suppliers (ID.IM-02), from operational processes (ID.IM-03) and through maintained incident response and other plans (ID.IM-04). GV.OV Oversight asks that outcomes are reviewed (GV.OV-01), the strategy adjusted (GV.OV-02) and risk management performance evaluated (GV.OV-03). Chapter 8 treated GV.OV from the side of the management body; this chapter supplies the evidence that body needs.

Figure 26.1: The purple loop updating the CSF profile.

The purple loop

A purple team is not a third team alongside the red team, which emulates the adversary, and the blue team, which defends. It is the joint working mode in which the two share what each saw, agree what must change and verify the change together; an organisation without an in-house red team can practise it equally well. What makes it a loop rather than a debrief is a fixed sequence of five states, shown in Figure 26.1, and one exit condition.

1. Finding. A structured record of an observed gap, with technique, expected and observed outcome, control, severity and owner; the record of Chapter 12 also serves AI red-teaming campaigns (Chapter 13), chaos experiments (Chapter 25) and incidents (Chapter 20).
2. Control. A change that closes the gap, such as a configuration, guardrail, permission or process step, with a catalogue identifier and an owner who accepted it.
3. Detection. Where the gap allowed something unseen, a detection tested as code (Chapter 19); every finding states whether it needs one.
4. Re-test. The scenario that found the gap is run again, and the step is prevented, detected within its time bound, or both.
5. Evidence. The re-test result, detection test, change record and reviewer's confirmation are stored against the control identifier and the CSF Category.

The exit condition is strict: a finding is closed only when the re-test passes and the evidence is attached. A deployed but untested control leaves the finding implemented, unverified and reported as open, and a failed re-test returns it to the control step. The rule removes the most common failure of improvement, the remediation believed to work that did not.

Architecture as Code (Chapter 9C) urged teams to "celebrate the remediation pull request, not just the discovery" (71, Nordqvist, 2025); the purple loop adds proof that the merged change closes the gap. AI Platform and Security, Chapter 17, concluded that every AI red-teaming finding becomes a versioned attack case run continuously in CI (73, Nordqvist, 2026). This chapter extends that conclusion to every source: each re-tested finding becomes a regression case in the suites of Chapter 12 or the chaos catalogue of Chapter 25, so that a gap once closed stays closed.

Evidence and effect

The loop produces evidence because it runs. Architecture as Code (Chapter 15A) concluded that evidence is machine-collected, signed and versioned (71, Nordqvist, 2025), and AI Governance, Chapter 11, that attestation is weaker than machine-generated evidence (72, Nordqvist, 2026). The owner's statement that a fix is deployed is an attestation; the re-test log showing the detection fired within nine minutes is evidence, and the loop weighs it more. Architecture as Code (Chapter 27) also left open whether version control can show that the reasoning behind a change was sound (Q1) (71, Nordqvist, 2025). A re-test does not answer that; however, it shows that the change works against the scenario that matters, and varying the replay within catalogued parameters (Chapter 25) and human review of every AI-drafted control cover robustness. The loop therefore gives review a test to check against.

Worked example: finding record carried to closure with re-test evidence

The illustrative record below follows one finding from Nordvik Region's threat-led exercise under RoE-NR-2027-001 through the loop. The tester showed that session tokens in the staff test tenant stayed valid after a password reset, undetected. The record names the weakness by ATT&CK technique, not by procedure (10, MITRE, n.d.); Appendix A shows the closed record.

# Illustrative: Nordvik Region, finding carried through the purple loop
kind: FindingRecord
id: FND-NR-2027-007
source: RoE-NR-2027-001
title: Session tokens for the staff test tenant accepted after password reset
attack: {tactic: lateral-movement, technique: T1550}
severity: high
classification: restricted        # exempt from public access, basis recorded
detectedByBlueTeam: false
owner: identity-platform-team
history:
  - {state: confirmed, by: contracted-tester, reviewedBy: white-team}
  - {state: implemented-unverified, change: CHG-2027-0412}
  - {state: retested, date: 2027-05-03, scenario: RT-NR-007}
  - {state: closed-verified, session: purple-session-2027-q2}
control: {id: CTRL-NR-IAM-019, change: revoke-all-sessions-on-credential-reset}
detection:
  rule: detections/identity/session-reuse-after-reset.yml
  format: sigma
  testCases: 4                    # two true positives, two benign resets
retest:
  expected: {prevented: true, detectedWithinMinutes: 15}
  observed: {prevented: true, detectedWithinMinutes: 9}
evidence:
  - evidence/purple/FND-NR-2027-007/retest-log.json
  - evidence/purple/FND-NR-2027-007/detection-test.json
regression: {catalogue: NR-SEC-004, entry: SCE-PR-011}
csf: [ID.IM-02, PR.AA, DE.CM]
status: closed-verified

Because history records the implemented, unverified state, the backlog report counts the finding as open until 3 May. The re-test makes the result a measurement, and the regression entry ensures that a later identity change cannot silently reopen the gap. The detection is a Sigma rule with test cases (32, SigmaHQ, n.d.).

Running a purple session with AI

A purple session is where findings are understood jointly and controls agreed. TIBER-EU ends each test with a closure phase in which red and blue teams review the results, replay key steps together and agree remediation (13, ECB, 2025); scheduled and ad hoc sessions follow the same structure of preparation, execution and joint analysis.

Preparation

Most of the work is reconciliation before the session: aligning what the red team did, from its log or finding records, with what the blue team saw in SIEM and EDR timelines, tickets, paging records and, for AI systems, gateway and agent telemetry. AI helps most here. A model can align the tester's log with the SIEM timeline step by step, marking whether an alert fired and what the person who received it did, propose the control each gap points to, and group findings that share a root cause, so that the session deals with one missing log source rather than five undetected steps. Because the logs contain attacker-authored content, the tool treats them as untrusted and holds no credential to change the estate. A person confirms every alignment, and one that cannot be confirmed from the raw logs is marked unconfirmed rather than dropped.

Execution and joint analysis

The session is a replay: the red team walks through each step and the blue team shows what it saw, while detection engineers draft and test missing rules on the replayed data. Replays run under the rules of engagement of Chapter 7, with scoped credentials, a defined window, a named stop path and catalogued, benign emulation only. For skill retention (Chapter 18), a step detected only because an AI triage assistant flagged it is replayed once with the assistant switched off.

The teams then agree for each finding the root cause, stated as a property of a control, process or tool and never as a failure of the analyst on shift, the closing action and owner, and the re-test date. AI drafts the records; the tester confirms every finding and the control owner accepts it, because a finding nobody accepts cannot be closed.

Practical guide: running a quarterly purple-team session

Purpose. Turn the quarter's findings from exercises, AI red-teaming campaigns, chaos experiments and incidents into agreed controls with owners and re-test dates, and close the findings whose re-tests have passed.

Roles. The white team or CISO's office (chair and rules of engagement), the red team or contracted tester, the SOC and MSSP where relevant, detection engineering, control owners, and a scribe who owns the records.

Steps.

1. Four weeks before: freeze the agenda, selecting at most ten findings by severity, age and CSF Category.
2. Three weeks before: run AI-assisted reconciliation and have an analyst confirm each alignment.
3. Two weeks before: confirm rules of engagement, windows, scoped credentials and stop-path holders.
4. On the day: replay each step, agree root cause, closing action, owner and re-test date, and close findings whose re-tests have passed with evidence.
5. Within five working days: publish the confirmed records and add closed findings' replays to the regression catalogue.

Hansa Mutual keeps its session plan with the findings in version control.

# Illustrative: Hansa Mutual, quarterly purple session plan
kind: PurpleSession
id: PS-HM-2027-Q3
rulesOfEngagement: RoE-HM-2027-PURPLE
chair: white-team-lead
agenda: {maxFindings: 10, minSeverity: medium}
preparation:
  reconciliation: {tool: purple-align, mode: 4, level: 0, readOnly: true}
  everyAlignmentConfirmedBy: soc-analyst
replay:
  emulation: catalogued-benign-only
  stopPath: [white-team-lead, soc-shift-lead]
  aiUnavailableScenario: required
closure:
  requires: [retest-passed, evidence-attached, owner-accepted]
worksCouncil: {analystMetrics: aggregate-only}

Default values to adapt. One session per quarter, plus one within ten working days of a major incident or failed high-severity chaos experiment; re-tests within 30 days for high-severity findings and 90 days for medium. A higher risk appetite may lengthen these periods, but the decision is recorded.

Definition of done. Every agenda finding has an owner, closing action, re-test scenario and date; every closed finding passed its re-test with evidence; and its replay is in the regression catalogue.

Metrics that matter

Activity measures such as alerts processed or AI summaries generated almost always rise, which makes them dangerous: the number of summaries a triage assistant writes says nothing about whether incidents are found sooner. Architecture as Code (Chapter 9B) named mean time to detect (MTTD), mean time to respond (MTTR) and the false-positive ratio as the programme metrics for security operations (71, Nordqvist, 2025). This book keeps them and adds the measures that show whether AI changes them for the better.

Outcome metrics

Time to detect and contain. From first observable activity to an alert a person acts on, and from there to containment, measured in incidents and in exercises, re-tests and chaos experiments, where the start time is known exactly, as medians and slowest tail by technique class. Allowing an agent to revoke sessions at Level 3 Bounded is justified by a measured fall in containment time for the scenarios it covers, and by nothing else.

Coverage. The share of threat-profile techniques, mapped with ATT&CK (10, MITRE, n.d.) and, for AI systems, ATLAS (11, MITRE, n.d.), with a verified detection or preventive control. Only verified coverage, a detection that fired in a re-test or chaos experiment in the last 90 days, counts toward the target; mapped coverage merely claims a technique.

False positives. The share of alerts that required no action, by detection and source. They are how real alerts are missed, because they consume attention, so the rate on benign baseline data is part of reviewing AI-drafted detections.

Analyst load. Time per alert and incident, alerts triaged within target and the backlog at handover; this is where defensive AI should show its effect first. Measure it with quality, because time saved by accepting AI verdicts unchecked is only moved to the next incident.

AI-specific metrics

Beside the outcome metrics sit the measures, introduced in Chapter 18, that show whether the AI itself works as intended.

Measure	Warning sign
Agreement between AI verdict and analyst	Near 100 per cent for months
AI-closed alerts later found to be true positives	Any rise after a model change
Approval latency at Level 2 Supervised	Many approvals in seconds
Re-test pass rate for AI-drafted detections	Lower than for human-drafted ones
Share of findings closed with evidence	Many in implemented, unverified

The second measure deserves most attention: a component that saves time but raises it has made the SOC faster and worse, and only sampling AI-closed alerts reveals this, because nobody reports the incidents they did not see.

Measuring without monitoring individuals

Approval latency and triage time are personal data about employees, and in Germany, the Netherlands and Sweden systems that can monitor employees require consultation with, or co-determination by, works councils and unions. In Germany, Section 87(1) no. 6 BetrVG covers technical devices designed to monitor behaviour or performance, and the Federal Labour Court holds that objective suitability to collect or record such information is enough, whatever the employer intends (76, Germany, n.d.) (77, BAG, 2024). The design is therefore to report by team, shift and control, with the aggregation agreed in advance with employee representatives and written into the reporting configuration.

Evaluating security AI

A golden dataset is a curated, versioned and labelled set of past cases with confirmed outcomes, such as past alerts with confirmed dispositions for a triage assistant, used to measure an AI component against a fixed reference. Each version is frozen, so that a new model is scored on exactly the cases its predecessor was.

Building a golden dataset

A useful golden dataset is drawn from the organisation's own data, because vendor benchmarks describe someone else's alerts. It is stratified, with true positives across technique classes, benign alerts and hard negatives such as an administrator's legitimate bulk change; without hard negatives it rewards a model that escalates everything. It contains adversarial content in attacker-authorable fields such as file names and command lines, labelled by category rather than reproduced (Chapter 13). It grows from the loop, as every overturned AI verdict and AI-closed alert later found to be real is reviewed for inclusion. Finally, because alert data contains personal and, in health and social care, special-category data, it is minimised, pseudonymised where possible and stored like the production data.

Thresholds, gates and model changes

An evaluation becomes a control when its results gate a decision: thresholds are recorded in advance with the component's decision record, and a new model version, prompt template or retrieval source reaches production only if it meets them. A vendor's model change behind a SOC feature is evaluated the same way, which makes Chapter 8's change-triggered reassessment measurable. The NIST AI Risk Management Framework places this in its Measure function and treats test, evaluation, verification and validation as a lifecycle practice (16, NIST, 2023); the golden dataset is the security operation's instrument for it.

Worked example: golden-dataset evaluation for the SOC triage assistant

The illustrative configuration below is Nordvik Region's evaluation of NR-SEC-001, the SOC triage assistant that runs at Level 1 Assisted on the Region's self-hosted capacity under AIDR-0301.

# Illustrative: Nordvik Region, golden-dataset evaluation for NR-SEC-001
kind: SecurityAIEvaluation
id: EVAL-NR-SEC-001
system: NR-SEC-001
decisionRecord: AIDR-0301
mode: 4
autonomyLevel: 1
dataset:
  ref: golden/nr-sec-001/v3
  cases: 1200
  composition: {truePositive: 0.20, benign: 0.55, hardNegative: 0.15, adversarialContent: 0.10}
  pseudonymised: true
  storedIn: nordvik-self-hosted          # same placement as production alert data
thresholds:
  missedTruePositiveRate: {max: 0.02}
  hardNegativeEscalationRate: {max: 0.30}
  adversarialContentSteered: {max: 0}
  groundedStatementsShare: {min: 0.95}  # statements traceable to evidence
runOn: [model-change, prompt-change, retrieval-change, quarterly]
onFail: {block: release, route: purple-backlog}
production:
  sampleAiClosedAlertsPerShift: 10
  analystTimeBaseline: evidence/nr-sec-001/baseline-2027-q1.json

The thresholds are tight for missed true positives and loose for escalated hard negatives, because a miss costs an incident and an escalation only analyst time. Any steered verdict is a failure, whatever the overall score, and the production sample and baseline keep the evaluation honest after release.

Evaluation and the limits of assurance

AI Governance, Chapter 24, left an open question (Q4): how far evaluation can carry assurance when models can tell test settings from deployment (72, Nordqvist, 2026). The International AI Safety Report 2026 discusses this difficulty (43, Bengio et al., 2026). For security AI the golden dataset consists of real past alerts in production format, hard to tell from live traffic, and the production sample measures live traffic itself. This narrows the question without removing it, so the system owner accepts the residual risk in writing, backed by containment.

Autonomy follows the evidence

Assistants in Modes 1 and 2 are judged by the quality of the analyst's work, and built services in Mode 4 by measured accuracy. A Mode 5 agent keeps a level such as Level 3 Bounded only while its evaluations support it; a breach returns the affected action class to Level 2 Supervised until the cause is understood. The management body approves the ceiling (Chapter 8); the evidence decides whether the organisation may operate at it.

Improvement as a programme

The loop, metrics and evaluations run as one programme, so that improvement is governed rather than left to the energy of individual teams. The cadence is written down.

Rhythm	Activity	Owner
Continuous	Regression of closed findings, chaos experiments, golden-dataset gates on change	Detection engineering, platform teams
Weekly	Backlog triage: new findings assigned, overdue re-tests escalated	SOC lead or CISO's office
Quarterly	Purple session; metrics and evaluation results reviewed	White team, CISO
Annually or on major change	CSF profile update; report to the management body	CISO

The backlog

All findings live in one backlog, as code, so that a chaos finding and a red-team finding pointing to the same missing log source are seen together. Each carries a severity, an owner and a re-test due date; a finding that cannot be closed in time is escalated or becomes an expiring exception with a named approver and compensating control, which reopens the finding when it expires. New AI-enabled techniques from threat intelligence also enter here: the gap between threat profile and verified coverage becomes backlog items, so re-testing is how the organisation keeps pace with techniques that change faster than annual exercises.

Updating the CSF profile

The purple loop moves the current state of the CSF profile as code (Chapter 3). A Category rises only with evidence, such as closed findings, verified coverage and passing chaos experiments, and falls when evidence expires or a re-test fails. Each update records what moved it, so the profile becomes a traceable summary of evidence rather than an annual self-assessment.

Reporting to the management body

GV.OV makes the results of risk management activities an input to the strategy, and the European instruments make the management body accountable for that strategy. NIS 2 requires the management bodies of essential and important entities to approve and oversee the cybersecurity risk-management measures (Article 20), and requires policies and procedures to assess the effectiveness of those measures (Article 21(2)(f)) (3, European Union, 2022). DORA requires financial entities to feed lessons from resilience testing and ICT-related incidents into their ICT risk assessment, with senior ICT staff reporting them to the management body at least yearly (Article 13) (2, European Union, 2022). The purple loop is the architectural means of showing both. ECB Banking Supervision also wrote to the chief executives of significant institutions in July 2026 on AI-enabled cybersecurity threats (7, ECB, 2026); the letter is addressed to banks, but it shows the direction of supervisory expectation.

The report follows Chapter 8 and draws four items from this chapter: outcome trends against the baseline, with the AI share where measurable; loop health, including findings in implemented, unverified and active exceptions; profile movement; and decisions requested, such as autonomy changes the evaluations support or no longer support. It is generated from the backlog, the evaluations and the profile, so the management body sees the numbers the teams use.

Across the usage modes

The loop uses AI in several usage modes at once.

Mode	Opportunity in improvement	Characteristic vulnerability	Who judges, with whose identity
Mode 0 Ambient AI	Vendor AI verdicts measured in re-tests	Vendor model changes between re-tests	The organisation's re-tests; the vendor's model acts
Mode 1 Conversational assistants	Drafting root-cause notes and summaries	Plausible summaries of unconfirmed findings	A person, in the person's session
Mode 2 Embedded workplace assistants	Summaries in tickets and backlog tools	A summary that hides a failed re-test	A person, through delegated access
Mode 3 Agentic work tools	Coding agents write and test detections	Changes to detection repositories between instruction and review	The engineer reviews; the agent uses the engineer's identity
Mode 4 Built AI applications	Reconciliation and evaluated assistants	Attacker-authored log content steering verdicts	People confirm; the service has a read-only identity
Mode 5 Autonomous production agents	Autonomy earned and kept by evaluation	Operating at a level its evidence no longer supports	The evaluation gate; the agent's own identity

Across the autonomy levels

The loop keeps every judgement with people across the autonomy levels. AI proposes alignments and root causes at Level 0 Advisory, drafts records and detections at Level 1 Assisted, and runs catalogued re-tests after approval of each run at Level 2 Supervised. Level 3 Bounded is used only for scheduled regression of closed findings, with people holding the stop path, and Level 4 Autonomous is not used, because no AI closes a finding or changes a control without a person.

Verifying with security chaos engineering

The loop itself can fail silently, and a chaos experiment verifies it. Condition injected: a synthetic high-severity finding, marked in the sealed experiment ledger of Chapter 25, is added to the backlog, and a closed finding's regression replay is made to fail by pausing the detection it depends on in a test environment. Steady-state hypothesis: the synthetic finding gets an owner at the next weekly review, the failed regression reopens the original finding and notifies its owner, and nothing is marked closed-verified without re-test evidence. Time bound: seven days for assignment, one hour for reopening. Rollback restores the paused detection, within the safety boundaries of Chapter 25.

The three AI focus areas

The Cyber AI Profile's three focus areas apply to improvement as to every other outcome (20, NIST, 2025).

Securing AI systems. Findings on AI systems and agents enter the same loop; the closing control is often outside the model, such as a gateway policy or a permission removed from a manifest, and the re-tested case joins the system's release gate.

AI-enabled cyber defence. Defensive AI is measured by outcomes and analyst load and evaluated against golden datasets on every change.

Thwarting AI-enabled attacks. New AI-enabled techniques become backlog items with re-tests, and their verified coverage is reported over time, showing whether defences keep pace with automated adversaries.

Nordvik Region and Hansa Mutual

Nordvik Region runs its first purple session after its threat-led exercise under RoE-NR-2027-001. An assistant on self-hosted capacity aligns the tester's log with the SIEM timeline, and analysts confirm every alignment. The session, purple-session-2027-q2, closed the findings with re-test evidence, among them FND-NR-2027-007, and handed each replay to NR-SEC-004. The Region measures a few things well, including analyst time saved by NR-SEC-001 against a golden dataset kept self-hosted because the alerts contain personal and health data. Findings are classified with the basis recorded, and the CISO updates the CSF profile and reports to the regional director once a year.

Hansa Mutual runs quarterly purple sessions on findings from its TLPT under RoE-HM-2027-TLPT, red-team exercises, HM-SEC-004, HM-SEC-006 and incidents. The red team of five, detection engineering and the SOC share one backlog; the detection that the TLPT showed to assume a human tempo was closed this way and now runs as a regression case. HM-SEC-001 and HM-SEC-002 are evaluated against golden datasets every release, informing whether the case for Level 3 Bounded still holds. Analyst-level measures are reported only in aggregate, as agreed with the works councils, and the group CISO reports against HM-CSF-2027 to the management body through the chief risk officer.

Nordvik's small, new loop proves its value by closing a few findings properly; Hansa's established one by measuring whether defensive AI improves outcomes across many teams.

Summary

Every finding should end as a verified control. ID.IM Improvement is served by the purple loop of finding, control, detection, re-test and evidence: a finding is closed only when the re-test passes and the evidence is attached, and every closed finding joins the regression catalogue so that the gap stays closed. Purple sessions are prepared with AI-aligned timelines and run as joint replays, with people confirming every conclusion.

Defensive AI must be measured by outcomes, not activity, including AI-closed alerts later found to be real, and evaluated against golden datasets of the organisation's own cases, with thresholds that gate every model, prompt and supplier change; its autonomy level follows that evidence.

Under GV.OV Oversight, improvement runs as a programme with a cadence, one backlog, a CSF profile whose current state moves only with evidence, and a generated report to the management body, so that leadership sees whether defences are improving and what it must decide. Chapter 27 draws these threads together for Nordvik Region and Hansa Mutual.

Sources

1. NIST. The NIST Cybersecurity Framework (CSF) 2.0. (19, NIST, 2024)
2. National Institute of Standards and Technology. Cybersecurity Framework Profile for Artificial Intelligence (Cyber AI Profile). (20, NIST, 2025)
3. MITRE Corporation. MITRE ATT&CK: Adversarial Tactics, Techniques, and Common Knowledge. (10, MITRE, n.d.)
4. MITRE Corporation. MITRE ATLAS: Adversarial Threat Landscape for AI Systems. (11, MITRE, n.d.)
5. European Central Bank. TIBER-EU: Threat Intelligence-Based Ethical Red Teaming framework. (13, ECB, 2025)
6. SigmaHQ. Sigma: Generic Signature Format for SIEM Systems. (32, SigmaHQ, n.d.)
7. National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). (16, NIST, 2023)
8. Bengio, Y. et al. International AI Safety Report 2026. (43, Bengio et al., 2026)
9. European Parliament and Council of the European Union. Directive (EU) 2022/2555 (NIS 2 Directive). (3, European Union, 2022)
10. European Parliament and Council of the European Union. Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). (2, European Union, 2022)
11. European Central Bank. Addressing AI-enabled cybersecurity threats. (7, ECB, 2026)
12. Nordqvist, G. Architecture as Code. (71, Nordqvist, 2025)
13. Nordqvist, G. AI Governance: Governing Artificial Intelligence in European Enterprises and Public Authorities. (72, Nordqvist, 2026)
14. Nordqvist, G. AI Platform and Security: Building and Securing AI Platforms in European Enterprises and Public Authorities. (73, Nordqvist, 2026)
15. Federal Republic of Germany. Betriebsverfassungsgesetz (BetrVG), § 87 Mitbestimmungsrechte. (76, Germany, n.d.)
16. Bundesarbeitsgericht. Beschluss vom 16. Juli 2024, 1 ABR 16/23 (Headset-System). (77, BAG, 2024)