Awareness and Training: Resilience to AI-Generated Deception

Security awareness has long meant an annual e-learning module, a quarterly simulated phishing email and a click-rate chart, and AI has broken the assumptions behind that routine. A lure no longer betrays itself through clumsy grammar, a voice on the telephone can belong to nobody, and a video call can show a colleague who is not there. Meanwhile, the people inside the security function have a new task that no awareness programme prepared them for: supervising AI that triages alerts, drafts detections and proposes containment, which they must learn to check, override and, where it acts on its own, stop.

This chapter therefore treats two disciplines that AI has drawn together: resilience to AI-generated deception, built into processes and tested on processes rather than individuals, and AI literacy for security staff, so that the people who work with security AI keep the judgement it is supposed to support. In CSF terms the chapter is responsible for PR.AT Awareness and Training, whose outcome is that personnel receive awareness and training to perform their cybersecurity-related tasks. PR.AT-01 covers general personnel and PR.AT-02 individuals in specialised roles (19, NIST, 2024). With security AI in the estate, both audiences change: general staff face AI-generated deception, and specialised roles now include those who supervise AI.

The chapter builds on two earlier conclusions. AI Governance, Chapters 10 and 21, concluded that human oversight must be designed, not declared, and that approvals given in seconds signal rubber-stamping (72, Nordqvist, 2026). AI Platform and Security, Chapter 18, committed to identity and approvals that resist synthetic media and to AI for defence under oversight, where human verification is the bottleneck (73, Nordqvist, 2026). Training is how the organisation makes both commitments real.

Figure 15.1: Approval process with synthetic-media checkpoints and training loop.

Deception at scale

The most cited case moved deepfakes from a conference topic to a board topic. In 2024 the engineering firm Arup confirmed that it had lost about 25 million US dollars after an employee in its Hong Kong office was deceived by a video call in which the participants, including a senior finance officer, were AI-generated imitations of real colleagues (55, Magramo, 2024). The employee was not careless: the request came through an everyday channel from apparently familiar faces, and the payment process accepted the call as sufficient authority. Spotting suspicious links does nothing against that attack, whereas a call-back to a known number and a second approver does.

Google's threat-intelligence group found that state-backed actors used a generative model mainly for productivity, such as target research, translation and drafting, and had not observed it giving them fundamentally new capabilities (46, Google GTIG, 2025). OpenAI described scam operations using models to draft and translate messages at volume, adding AI to existing playbooks rather than inventing new forms of attack (54, OpenAI, 2025). Europol's 2026 assessment names AI, alongside encryption and proxies, among the factors expanding cybercrime (35, Europol, 2026).

These sources support a sober conclusion. AI has not created a new class of social engineering; it has removed three frictions that used to limit it: language, because a lure can be written fluently in Swedish, Polish or German by an actor who speaks none of them; personalisation, because tailoring a message costs almost nothing; and presence, because a voice or face can be synthesised well enough to carry a short call. The ATT&CK techniques, T1566 Phishing, including its voice variant, and T1684.001 Impersonation, are familiar (10, MITRE, n.d.); what has changed is their cost and quality. Three consequences follow for training.

1. Recognition is no longer a reliable control. People should still report what feels wrong, but the organisation cannot rely on them to distinguish a real voice from a synthetic one.
2. Processes must carry the load. The controls that work are those that do not depend on how convincing a request is: an expected channel, a call-back to a registered number, a second approver and a cooling-off period for payee changes. Chapter 14 designs these approvals under PR.AA; this chapter trains people to apply them and tests whether they do.
3. The target is the approval, not the inbox. Training and exercises should concentrate on payment changes, treasury transfers, privileged-access grants, service-desk resets and urgent requests from apparent executives.

Test the process, not the person

The classic simulated phishing campaign tests individuals covertly and records who clicked. It damages trust, measures the difficulty of the lure as much as the organisation's resilience, and with AI-generated material becomes ethically harder still, because a synthetic voice of a manager can cause real distress that a debrief cannot undo.

The alternative makes the process the subject of the test. An exercise asks whether the payment-change process, the service desk's reset procedure or the treasury's approval chain behaves as designed when a realistic, AI-generated request arrives, and the result is a finding about the process: whether the call-back was applied, whether the second approver refused, and how quickly the request was reported. People take part knowingly, and their individual performance is not measured, just as Chapter 25 treats a failed hypothesis in the SOC as a finding about a detection or process, never about the person on shift.

Ethics, labour law and data protection

Four principles keep exercises ethical. First, announce the exercise at the level of the unit: staff know that a resilience exercise may take place within a stated period, typically two to four weeks, without knowing when or in which form. This does not remove the test's value, because a process that works only when people are warned has already failed. Second, obtain specific consent from anyone whose likeness is used, preferring a fictitious persona or a consenting voice actor. Third, exclude themes that exploit personal vulnerability, such as health, family emergencies, bereavement, redundancy and pay. Fourth, debrief everyone involved promptly and make clear that the result concerns the process.

An exercise that observes how employees respond can monitor employees, and in Germany, the Netherlands and Sweden works councils or unions must be consulted on, or agree to, such activities under differing national rules. In Germany, Section 87(1) no. 6 BetrVG covers technical devices designed to monitor behaviour or performance, and the Federal Labour Court holds that objective suitability to collect or record such information is enough, whatever the employer intends (76, Germany, n.d.) (77, BAG, 2024). Exercises are therefore designed with employee representatives, not presented to them. An awareness exercise is an offensive activity in the book's sense, so it runs under written rules of engagement as Chapter 7 sets out.

Exercises also process personal data, so the GDPR requires a legal basis, a defined purpose, minimisation and a retention limit, and a large exercise may call for a data protection impact assessment (9, European Union, 2016). The ethical design choices also satisfy these: measure outcomes from systems that already record them, do not record calls, aggregate results, delete raw records after 30 days by default, and destroy synthetic media at the end. For public authorities, public-access rules add a further consideration: an exercise design that becomes public loses its value, and a list of weak approvals is a gift to an adversary, so the design record is classified before it is stored.

Practical guide: designing a deepfake-resilience exercise for payment approvals

Purpose. Test whether the approval process for supplier payment changes or high-value transfers holds against a realistic AI-generated request, and improve it where it does not, without testing individuals covertly.

Roles. The process owner (head of finance or treasury), who owns the findings; the CISO, who approves the rules of engagement; a white team of two to three people, who design the scenario and can stop it; the data protection officer; employee representatives where labour law requires; and the SOC shift lead, who can confirm with the white team whether a report concerns the exercise.

Steps.

1. Choose the process, not the population. Select one high-impact approval flow and write down its designed control: expected channel, call-back to a registered number, second approver, cooling-off period.
2. Write the hypothesis, for example that a synthetic voice request to change a supplier's bank details meets a call-back before any change and is reported to security within 60 minutes.
3. Design the scenario within ethical limits, with a fictitious or consenting persona, an ordinary business theme and realistic channels, described at the level of pretext and channel; the synthetic media are produced inside the white team's environment and never reused.
4. Agree the design in an exercise design record, with employee representatives' agreement where required, the data protection officer's review and the CISO's signature on the rules of engagement.
5. Put safety mechanisms in place, holding payee changes during the window for white-team verification so that no real money moves, and letting any distressed participant stop at once.
6. Announce the window without date or form, and run the exercise within it.
7. Measure from systems, not people, in aggregate: whether the call-back was placed, whether the second approver refused, and when the request was reported.
8. Debrief within two working days and close findings as process changes, re-running the same scenario to verify each change, in line with the purple loop of Chapter 26.

Default values to adapt. A window of two to four weeks; at most two scenarios per unit per year; no reporting of any group smaller than ten people; raw records deleted after 30 days.

Definition of done. The design record is signed and classified, no real payment was affected, results are reported in aggregate, every failed step has a finding with an owner, and the re-run has verified the control change.

Worked example: exercise design record with consent, scope and success criteria

The illustrative record below is Nordvik Region's design for its first announced process test in the finance unit, filed under the exercise's rules of engagement and classified before storage.

# Illustrative: Nordvik Region, exercise design record (classified, extract)
kind: ResilienceExerciseDesign
id: NR-AWR-2027-01
rulesOfEngagement: RoE-NR-2027-002
csf: [PR.AT-01]
process:
  name: supplier-payment-change
  owner: head-of-finance
  controlUnderTest: [expected-channel, call-back-registered-number, second-approver]
hypothesis:
  condition: synthetic voice request to change a supplier's bank details
  expected: call-back applied before change; request reported to security
  reportWithinMinutes: 60
scenario:
  persona: fictitious-supplier-contact     # no real person's likeness
  themesExcluded: [health, family, pay, redundancy, bereavement]
  syntheticMedia: {producedBy: white-team, reuse: false, destroyAfter: window-end}
announcement: {audience: finance-unit, window: {from: 2027-03-01, to: 2027-03-19}}
consent: {employeeRepresentatives: consulted-and-agreed, dataProtectionOfficer: reviewed}
dataHandling:
  sources: [payment-system-audit-log, reporting-button, service-desk-tickets]
  callRecording: false
  minimumGroupSize: 10
  rawRecordRetentionDays: 30
safety: {paymentHoldDuringWindow: true, stopAuthority: [white-team-lead, ciso]}
successCriteria:
  callBackAppliedShare: 1.0
  reportRateMin: 0.7
  realPaymentsAffected: 0
classification: restricted

The record deliberately contains no call script and no synthetic media, which stay with the white team and are destroyed at the end of the window. It contains what an auditor, a works council or the data protection officer needs to judge whether the exercise was proportionate.

Measuring resilience

The click rate is driven by the lure, counts individual failures and says nothing about whether anyone noticed a successful deception and told those who could act. Against AI-generated deception, where some requests will deceive any reasonable person, the decisive question is how quickly the organisation learns that a deception is under way. Four aggregate measures answer it.

Measure	What it shows	Source
Process adherence	Share of exercise requests where the designed control was applied	Payment-system or service-desk audit log
Report rate	Share of recipients or requests reported to security	Reporting button, tickets
Time to report	Median and 90th-percentile time from first contact to first report	Reporting button, tickets
Time to containment	Time from first report to a warning or a blocked channel	SOC tickets and broadcast log

Process adherence is the primary measure for approvals, while report rate and time to report treat staff as a distributed sensor network whose early reports let the SOC warn others and block a sender, linking PR.AT to the Detect Function. Two practices keep the measures honest. Every report is welcomed, including reports of genuine requests, because people who fear reporting a false alarm will not report a real one. Furthermore, results are compared over time for the same process, not between units: a unit's report rate rising from 40 to 70 per cent over three exercises is evidence of resilience, whereas a league table of departments shows only the exercise design.

The results feed the evidence manifest of AI Governance, Chapter 11 (72, Nordqvist, 2026). In Nordvik's first exercise, 23 finance staff received four requests; the call-back was applied in three, the report rate was 0.74 and the median time to report 38 minutes, with no real payment affected. The hypothesis nevertheless failed, because the payment system allowed a payee change to be saved before the call-back field was completed. Finding NR-FND-2027-014 is therefore closed by a system change that enforces the call-back field, not by a training module.

Training analysts for AI-assisted work

AI literacy as a duty and as a skill

For deployers in the European Union, AI literacy is a legal obligation as well as a security need. Article 4 of the AI Act, as replaced by Regulation (EU) 2026/1744 (the Digital Omnibus on AI, in force since 27 July 2026), requires providers and deployers to take measures to support the development of AI literacy among staff and other persons who operate or use AI systems on their behalf, taking into account their knowledge and the context of use, without obliging them to guarantee any specific level of AI literacy for any individual (75, European Union, 2026). Security AI is AI in the sense of the Act, so the duty reaches the SOC. For high-risk AI systems, Article 26 requires deployers to assign human oversight to people with the necessary competence, training and authority, and Article 14 expects those people to remain aware of automation bias (1, European Union, 2024). Security AI is rarely high-risk, but the book adopts these provisions as the standard for all security AI.

NIS 2 requires cybersecurity training for members of management bodies and lists cyber hygiene and training among the risk-management measures (3, European Union, 2022), and the implementing regulation adds training requirements for the digital-infrastructure and managed-service entities in its scope, including managed security service providers (4, European Commission, 2024). DORA requires ICT security awareness programmes and digital operational resilience training, and sufficient knowledge of ICT risk in the management body (2, European Union, 2022). The architectural implication is that this training must now cover the AI used for defence and the AI adversaries use for attack.

AI literacy for security staff is concrete: what each security AI system reads, how it fails, why attacker-controlled content can steer it, its autonomy level and who can stop it. Following AI Governance, Chapter 14, which located this competence in roles (72, Nordqvist, 2026), the SOC uses short modules per system, built from its failure modes and red-teaming findings, with completion recorded per role.

Verification against the evidence

The first skill is verification. An AI triage proposal or draft detection is a claim about the evidence, and the analyst checks it against the evidence, not against its plausibility. Training teaches a fixed habit: before accepting a proposal, open the telemetry and confirm the two or three facts it rests on, such as the process tree, source address, user and time; where it cites no event, treat it as unsupported.

Verification also has a security-specific dimension, because much of what security AI reads, such as email bodies, file names and log fields, is attacker-controlled and can be crafted to steer a model's proposal (14, OWASP, 2025), which AI Platform and Security, Chapter 16, treats as a property of the architecture (73, Nordqvist, 2026). Analysts therefore learn to scrutinise proposals that are unusually convenient for an attacker, such as closing a case, allow-listing a sender or taking no action. Training uses cases from the organisation's AI red-teaming programme (Chapter 13), described by behaviour and outcome, without the steering content.

Automation bias and approval quality

The second skill is resisting automation bias: when a system is usually right, people stop checking it. Training makes the tendency visible by teaching the conditions in which it grows, such as high volume, fatigue and interfaces where accepting takes one click and rejecting five. The conclusion of AI Governance, Chapters 10 and 21, becomes operational here (72, Nordqvist, 2026): the organisation measures approval quality from the override log, through time to decision, the share of proposals rejected or modified, and agreement with later review. A rejection rate near zero, or a median approval time of a few seconds on proposals that require reading telemetry, is a finding about the oversight design, not the analyst, and may call for a different interface, lower volume or a lower autonomy level. Analysts also rehearse rejecting, modifying and recording reasons, so that the override path is familiar before it is needed.

Supervising and stopping agents

Where security AI acts with its own identity, training extends from verification to command. Agent operators and shift leads learn to read the agent's permission manifest, recognise behaviour outside its mandate and use the stop path, which AI Platform and Security, Chapter 19, placed outside the system it stops (73, Nordqvist, 2026). Because an unpractised stop path exists in name only, stopping an agent is drilled like any emergency procedure, with named holders, a documented sequence, a time target and a quarterly drill in a test environment; Chapter 6 defines the roles and authorities.

Skill retention

The third skill is keeping the skill the AI performs, because analysts who never triage unassisted lose the ability to judge the assistant and to work when it is unavailable or compromised. The book recommends deliberate practice: by default, ten per cent of alerts, chosen at random, are worked without the AI's proposal and compared afterwards, which trains the analyst and produces evaluation data for the AI. Exercises also include degraded-mode shifts without the security AI; Chapter 18 describes how analysts stay in command of the AI-assisted SOC day to day.

Across the usage modes

Training differs by mode because the person's role differs: verifying suggestions in Modes 1 and 2, supervising agents in Mode 3, and operating and stopping Mode 5 agents (Chapter 6).

Mode	Opportunity	Characteristic vulnerability	Who judges, with whose identity
Mode 0 Ambient AI	Vendor AI filters lures first	Staff assume the filter catches everything	The vendor's model, then staff; the product's
Mode 1 Conversational assistants	Quick checks of suspicious messages	Sensitive data pasted in; fluent but wrong answers	The person; their session
Mode 2 Embedded workplace assistants	Summaries in mail, tickets and SOAR	A summary hides the detail that reveals a fake	The person; delegated access
Mode 3 Agentic work tools	Detection engineers build faster	The engineer reviews the result, not the agent's steps	The engineer at review; their credentials
Mode 4 Built AI applications	Triage and drafting at SOC scale	Automation bias; attacker-controlled alert content	The approving analyst; a read-only service identity
Mode 5 Autonomous production agents	Containment at machine speed	Rubber-stamped approvals; an untrained stop-path holder	Operators by approval and stop; the agent's own identity

Across the autonomy levels

Each autonomy level (Chapter 6) changes what the supervising person must be trained to do.

Level	What training must cover
Level 0 Advisory	Verification against evidence; recognising attacker-convenient suggestions
Level 1 Assisted	Verification of each draft; unassisted practice to keep the skill
Level 2 Supervised	Approval quality, override and recorded reasons; automation bias
Level 3 Bounded	Reading the playbook and the agent's manifest; stop-path drills
Level 4 Autonomous	Not used for irreversible or offensive actions; monitoring and post-hoc review where used at all

Verifying with security chaos engineering

Training is verified like technical controls, with an injected condition, a steady-state hypothesis and a time bound, under the safety boundaries of Chapter 25, announced at unit level and reported in aggregate. In the approval-resilience experiment, an announced synthetic-media payee-change request arrives within a stated window, and the hypothesis is that the call-back is applied before every change, at least 70 per cent of requests are reported and the median time to report is under 60 minutes, with results within five working days. In the SOC verification experiment, during an announced practice shift, a catalogued case presents an AI proposal that contradicts the telemetry, and the hypothesis is that it is overridden with a recorded reason before the case closes. A third experiment triggers a Mode 5 agent's stop path in a test environment and checks that the named holder halts it within the target time. Every failed hypothesis becomes a finding about the process, interface or training.

The three AI focus areas

Securing AI systems. Everyone who builds or operates AI systems needs AI literacy proportionate to their role, which for EU deployers is also the duty of Article 4 of the AI Act (1, European Union, 2024); in the security function this includes those who supervise security AI, itself a system to protect (20, NIST, 2025).

AI-enabled cyber defence. Analysts verify security AI against the evidence, recognise automation bias, override and record why, stop agents, and keep their own skills. Because human verification is the bottleneck of AI for defence, training widens it without letting it become a formality.

Thwarting AI-enabled attacks. Resilience to AI-generated phishing, vishing and deepfakes is built into approval processes and exercised on them, measured by process adherence, report rate and time to report rather than click rates.

Nordvik Region and Hansa Mutual

Nordvik Region combines the AI literacy that Article 4 of the AI Act requires of it as a deployer with announced process tests in the finance unit, whose first exercise led to the system-enforced call-back field. Its six analysts receive specific training for NR-SEC-001: checking a Level 1 Assisted triage proposal against the telemetry, recording an override, and working a share of alerts without assistance. Because out-of-hours monitoring is contracted, the Region asks its MSSP to show that the analysts who handle its alerts receive equivalent training.

Hansa Mutual tests claims-payment and treasury approvals in each market and language, only in entities where the works council has agreed the design. SOC analysts in Hamburg and at the Polish service centre are trained to supervise HM-SEC-001 at Level 2 Supervised, including stopping it, and detection engineers review drafts from HM-SEC-002 as carefully as a colleague's. Some Level 2 approvals were given in seconds, which Hansa treats as a finding about its oversight design, answered with interface changes and refresher training on automation bias; training material also uses the red team's finding that crafted alert content could steer the agent's proposals. Both organisations test processes rather than people, but Hansa's programme is larger because it spans languages, labour-law regimes and agents with greater authority.

Summary

AI has made social engineering cheaper, more fluent and more personal by removing the frictions of language, personalisation and presence, so recognition can no longer carry the load and processes must. Under PR.AT Awareness and Training, the organisation tests processes rather than people, through announced, lawful exercises agreed with employee representatives where required and documented in a classified design record, and measures resilience by process adherence, report rate and time to report rather than click rates.

For specialised roles, analysts verify AI output against the evidence, resist automation bias, override and record why, drill the stop path of agents and keep their skills by working without assistance, while approval quality from the override log makes oversight measurable. Across the focus areas, training thus supports role-based AI literacy, real supervision of security AI and approval processes that hold however convincing the request. Chapter 16 turns from people to the data and platforms that security AI processes and protects.

Sources

1. NIST. The NIST Cybersecurity Framework (CSF) 2.0. (19, NIST, 2024)
2. National Institute of Standards and Technology. Cybersecurity Framework Profile for Artificial Intelligence (Cyber AI Profile). (20, NIST, 2025)
3. Magramo, K. Arup revealed as victim of $25 million deepfake scam involving Hong Kong employee. (55, Magramo, 2024)
4. Google Threat Intelligence Group. Adversarial Misuse of Generative AI. (46, Google GTIG, 2025)
5. OpenAI. Disrupting malicious uses of AI: an update. (54, OpenAI, 2025)
6. Europol. The evolving threat landscape. How encryption, proxies and AI are expanding cybercrime – Internet Organised Crime Threat Assessment (IOCTA) 2026. (35, Europol, 2026)
7. MITRE Corporation. MITRE ATT&CK: Adversarial Tactics, Techniques, and Common Knowledge. (10, MITRE, n.d.)
8. European Parliament and Council of the European Union. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (1, European Union, 2024)
9. European Parliament and Council of the European Union. Directive (EU) 2022/2555 (NIS 2 Directive). (3, European Union, 2022)
10. European Commission. Commission Implementing Regulation (EU) 2024/2690 laying down rules for the application of Directive (EU) 2022/2555 as regards technical and methodological requirements of cybersecurity risk-management measures. (4, European Commission, 2024)
11. European Parliament and Council of the European Union. Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). (2, European Union, 2022)
12. Nordqvist, G. AI Governance: Governing Artificial Intelligence in European Enterprises and Public Authorities. (72, Nordqvist, 2026)
13. Nordqvist, G. AI Platform and Security: Building and Securing AI Platforms in European Enterprises and Public Authorities. (73, Nordqvist, 2026)
14. European Parliament and Council of the European Union. Regulation (EU) 2016/679 on the protection of natural persons with regard to the processing of personal data (General Data Protection Regulation). (9, European Union, 2016)
15. OWASP Foundation. OWASP Top 10 for Large Language Model Applications, 2025 edition. (14, OWASP, 2025)
16. European Parliament and Council of the European Union. Regulation (EU) 2026/1744 amending Regulations (EU) 2024/1689, (EU) 2018/1139 and (EU) 2023/1230 as regards the simplification of the implementation of harmonised rules on artificial intelligence (Digital Omnibus on AI). (75, European Union, 2026)
17. Federal Republic of Germany. Betriebsverfassungsgesetz (BetrVG), § 87 Mitbestimmungsrechte. (76, Germany, n.d.)
18. Bundesarbeitsgericht. Beschluss vom 16. Juli 2024, 1 ABR 16/23 (Headset-System). (77, BAG, 2024)