AI-Assisted Threat Modelling and Risk Assessment

Most organisations still model threats in a workshop: a security architect draws the data flows on a whiteboard, walks through a checklist and files the result. Within weeks a retrieval source is added, a service account gains a permission or an agent is given a new tool, and the model, and the risk scores built on it, describe a system that no longer exists. When adversaries use AI to find and exploit weaknesses faster, that drift becomes the gap through which they arrive.

This chapter shows how AI makes threat modelling continuous and risk assessment evidence-based: threat models generated from architecture code and regenerated on every change, threat intelligence fused with asset and exposure context, attack-path analysis across identity, cloud and AI components, and machine-assisted risk scoring reviewed by named people for soundness. It is responsible for one CSF 2.0 Category, ID.RA Risk Assessment in the Identify Function, whose outcome is that the organisation understands the cybersecurity risk to the organisation, its assets and individuals (19, NIST, 2024), and treats it through the three focus areas of the NIST Cyber AI Profile (20, NIST, 2025).

Figure 10.1: The threat-modelling loop from architecture change through AI-assisted candidates and review to treatment.

ID.RA Risk Assessment and what it asks for

ID.RA gathers ten Subcategories that read like a description of the loop in Figure 10.1 (19, NIST, 2024). This chapter concentrates on ID.RA-02 to ID.RA-07: threat intelligence is received, threats are identified, impacts and likelihoods are recorded and used to understand inherent risk, responses are prioritised and tracked, and changes and exceptions are assessed for their risk impact. Vulnerabilities and disclosures (ID.RA-01 and ID.RA-08) are the subject of Chapter 11, the integrity of acquisitions and supplier assessment (ID.RA-09 and ID.RA-10) belong to Chapter 8, and the asset inventory and risk appetite on which every score depends come from Chapters 9 and 5.

The Category maps onto European obligations. NIS 2 names policies on risk analysis first among the risk-management measures of Article 21(2)(a) (3, European Union, 2022). DORA requires financial entities to identify sources of ICT risk on a continuous basis and to perform a risk assessment upon each major change in their network and information system infrastructure (Article 8) (2, European Union, 2022). For high-risk AI systems, Article 15(5) of the AI Act names attack classes against which they must be resilient, among them data poisoning, model poisoning, adversarial examples and confidentiality attacks (1, European Union, 2024). An annual assessment cannot show that each significant change was assessed, whereas a threat model regenerated on every change can.

Threat models that keep up with change

Threat modelling is a practice, not a document. The Threat Modeling Manifesto frames it around four questions: what are we working on, what can go wrong, what are we going to do about it, and did we do a good enough job; among its values are "a journey of understanding over a security or privacy snapshot" and "continuous refinement over a single delivery" (25, Threat Modeling Manifesto Working Group, 2020). A workshop answers the four questions once, but the estate asks them again every time it changes.

Architecture as Code (Chapters 9 and 9C) applied STRIDE to infrastructure and pipelines as well as applications and concluded that threat handling belongs in the codebase: threat models are generated from architecture code and regenerated when it changes, automated checks flag missing trust boundaries, findings return as pull requests, and risk is a versioned record with an owner and expiring exceptions (71, Nordqvist, 2025). This chapter builds on those conclusions (F1 and F2 in SERIES_FOUNDATIONS.md) and asks what changes when AI takes part.

AI is good at the first draft: given the data flows, trust boundaries, identities and catalogue vocabularies, it can propose candidate threats for every pull request in minutes and show only what differs from the accepted model. It does not decide, however. A candidate becomes part of the organisation's threat model only when a named person accepts it, and a rejected candidate stays rejected, with a reason, until the architecture changes, so the loop in Figure 10.1 sends accepted threats to the register and rejected ones back to the generator as feedback.

The loop needs the architecture described as code, a threat model beside it with a validated schema, and pipeline triggers for regeneration; AI Platform and Security, Chapter 17, named these review_trigger paths, with a code-ownership rule routing every such change to the security architect (73, Nordqvist, 2026). AI adds only the draft.

AI-assisted STRIDE and ATLAS analysis

The quality of AI-generated threats depends less on the model than on its vocabulary. Free prose is plausible but hard to compare between versions, whereas entries classified by STRIDE, by ATLAS or ATT&CK technique and by OWASP identifier are searchable, stable and shared with the defenders who will test and detect them.

Classical boundaries with STRIDE and ATT&CK

For classical boundaries, STRIDE remains a serviceable checklist: spoofing, tampering, repudiation, information disclosure, denial of service and elevation of privilege (71, Nordqvist, 2025). The generator proposes, for each applicable category at each trust boundary, a threat expressed as an ATT&CK technique (10, MITRE, n.d.): a public endpoint invites exploitation of a public-facing application (T1190), and a service account crossing into production invites the use of valid accounts (T1078). Expressing threats by technique lets detection engineers check coverage and the red team of Chapter 12 choose scenarios.

AI systems with ATLAS and OWASP

For AI systems, STRIDE alone is insufficient, because it names the harm but not the technique by which an adversary reaches a model or extracts what it holds. AI Platform and Security, Chapter 17, therefore extended it with MITRE ATLAS and the OWASP Top 10 for LLM applications (73, Nordqvist, 2026). ATLAS describes adversary techniques against machine-learning systems, with case studies (11, MITRE, n.d.); the OWASP Top 10 names the principal risks, from prompt injection (LLM01) and sensitive information disclosure (LLM02) to excessive agency (LLM06) (14, OWASP, 2025); and the OWASP Top 10 for Agentic Applications adds the risks of tools, delegated identity and multi-step autonomy (15, OWASP, 2025).

The generator applies these vocabularies where documents enter a retrieval index, a prompt is assembled, a model is called, an agent uses a tool credential, and output lands in a system of record. AI Platform and Security, Chapter 16, concluded that prompt injection is a property of the architecture, not a patchable defect, so controls must sit outside the model (F20) (73, Nordqvist, 2026); the OWASP guidance on prompt injection likewise notes that, given the stochastic influence at the heart of how models work, it is unclear whether fool-proof methods of prevention exist (14, OWASP, 2025). A candidate that proposes "improve the system prompt" as mitigation is therefore unsound, whereas least privilege for tools, allow-listed retrieval, output validation or human approval are in the right family.

What the generator is given and what it is not

The generator itself reads sensitive material, so it reads only architecture definitions, the accepted model and the catalogues; runs on trusted, for a public authority perhaps self-hosted, infrastructure; writes only to a pull request comment; and treats external threat reports as untrusted input, because a report crafted to make a path look harmless is an attack on the risk assessment itself.

Threat intelligence in risk assessment

ID.RA-02 asks that cyber threat intelligence is received, and ID.RA-03 that threats are identified and recorded (19, NIST, 2024). Most organisations receive more intelligence than they can read; the difficulty is relevance to their own assets, exposure and sector.

Architecture as Code (Chapter 9C) concluded that threat intelligence is ingested as STIX and TAXII through pipelines with auditable provenance (71, Nordqvist, 2025). STIX 2.1 separates attack patterns, intrusion sets and vulnerabilities, links them through relationship objects, and lets objects carry external references to entries in other catalogues, which is the bridge to ATT&CK and ATLAS techniques in the threat model (24, OASIS Cyber Threat Intelligence (CTI) TC, 2021). AI extracts that structure from unstructured reports, linking each object to its source passage so that an analyst can check it, and joins it to the asset inventory of Chapter 9, the exposure data of Chapter 11 and the detection coverage held as code. The output is a proposed change to likelihood: this technique, used by an intrusion set that targets insurers, is relevant to these three internet-facing services, and no detection covers one of its steps.

Adversary capability as an explicit input

The most consequential intelligence for the coming years is a shift in adversary capability. The United Kingdom's National Cyber Security Centre assessed that AI will almost certainly increase the frequency and intensity of cyber threats up to 2027 (37, NCSC, 2025), and the ENISA Threat Landscape 2026 records AI being incorporated into the playbooks of cybercriminals, State-nexus intrusion sets and information manipulation sets, mostly in a supporting role so far (34, ENISA, 2026). Mandiant's analysis found that the average time to exploit a vulnerability had fallen to five days in 2023 (44, Charrier & Weiner, 2024), and the case of the Arup employee in Hong Kong who transferred money after a video call in which colleagues were impersonated with deepfakes showed what convincing impersonation can achieve against a process that relies on recognising a face and a voice (55, Magramo, 2024).

These findings belong in the risk model as parameters, not as a paragraph in an annual report: a likelihood that assumes weeks between disclosure and exploitation is wrong when the interval is days. The practical form is a small set of named, versioned assumptions, such as time to exploit, the credibility of voice and video impersonation, and the volume of targeted phishing, each with its source and review date, so that changing one rescores every dependent risk in a visible pull request. Chapter 2 describes the evidence in full.

Attack-path analysis

Serious incidents rarely follow from a single threat. Adversaries chain small weaknesses, each below the register's threshold, across the boundaries between identity, cloud, network and AI platform teams.

Attack-path analysis therefore represents the estate as a graph built from identity-provider configuration, cloud policies, network rules, the asset inventory and agents' permission manifests. Nodes are identities, resources and the content channels that feed AI systems; edges are relationships an adversary could traverse, such as can authenticate as or can reach. AI components need edge types classical graphs lack: an agent holding a tool credential is an edge to everything the tool can change, a retrieval index ingesting external content is an edge from whoever can write that source to every prompt it feeds, and a coding agent is an edge from anything it reads to everything its engineer can do. Such edges show how a document upload in one system can reach a production write in another.

Path-finding is a deterministic computation, and it should stay one: the graph engine enumerates paths from entry points such as internet-facing services and externally writable content to crown-jewel assets such as the identity provider, claims systems and the AI platform's control plane. AI contributes around it, proposing edge types and entry points from threat models and intelligence, ranking paths with checkable explanations from likelihood, detection coverage and criticality, and proposing the cheapest cut, the single edge whose removal closes the most high-ranked paths, in the vocabulary of MITRE D3FEND (12, MITRE, n.d.).

The ranking is where machine assistance is most useful and most dangerous, because a poisoned inventory or misleading report could push a real path down the list. The red team of Chapter 12 therefore tests the highest-ranked paths under rules of engagement, and the chaos experiment below checks that the graph notices a new edge. The graph stays at the level of relationships and techniques, never the operational detail an intruder would need.

Reviewing machine-generated risk

Architecture as Code (Chapter 27) left an open question for this book: version control captures what changed, but not whether the reasoning was sound (Q1 in SERIES_FOUNDATIONS.md) (71, Nordqvist, 2025). An AI-proposed threat or score can be well formed and wrong: a missed threat is invisible in the diff, and a deflated likelihood is accepted because it looks like all the others.

Soundness checks

Soundness can be reviewed only against something, so the organisation defines in advance what a sound candidate must show, and the pipeline checks the mechanical parts before a person sees it.

Check	Question the reviewer answers	Automated part
Grounding	Does the threat refer to a component, boundary, identity or flow that exists in this version?	Every reference resolves in the architecture definition
Classification	Are the STRIDE category, technique and OWASP identifier right for the vector?	Identifiers exist in the pinned catalogue versions
Mitigation family	Does the control act outside the model where the threat concerns model behaviour?	Controls resolve to deployed configuration
Likelihood basis	Which assumption or intelligence drives the likelihood, and is it current?	Every likelihood cites a named assumption or dated STIX object
Impact basis	Is the impact tied to the inventory's criticality and classification?	Impact matches or explains its deviation
Completeness	Does every boundary changed in this pull request have threats considered?	Changed boundaries without candidates are flagged
Stability	Why did the score change since the last version?	Diff against the previous accepted model

Each question can be answered in a minute, and the completeness check catches the failure a reviewer reading only the candidates would never see: a boundary that changed without any candidate threat.

Provenance and reviewer accountability

Every AI-proposed threat and score carries its provenance, the model and prompt versions, its inputs and the time, so that a reviewer sees its basis, a past assessment can be explained to a supervisor, and every proposal from a later-faulty model version can be found, as AI Platform and Security, Chapter 21, does with its incident impact query (73, Nordqvist, 2026).

Architecture as Code (Chapter 20) concluded that agents draft and humans approve, with exceptions and risk acceptances carrying a named human approval in the audit trail (71, Nordqvist, 2025). For risk assessment, the reviewer is therefore named, not a group alias, and records a reason for each decision; acceptance of a residual risk is reserved to the risk owner in the register, with an expiry date on every exception; and the reviewer's workload is watched, because a reviewer who accepts every candidate in seconds is not reviewing. The organisation samples accepted candidates for a second review and tracks how often first decisions are overturned. The book's answer to Q1, for risk, is thus not merely that AI output is reviewed: soundness is defined in advance, its mechanical part is automated, a named person judges against provenance, and the quality of that judgement is itself measured.

Practical guide: running an AI-assisted threat-modelling review on a pull request

Purpose. Ensure that every change to an architecture definition that alters its threat surface is reflected in a reviewed threat model before it merges.

Roles. The change author; the security reviewer, named per system in code ownership; the risk owner for residual-risk acceptance; the platform team that runs the generator.

Steps.

1. Define trigger paths per system: data flows, trust boundaries, retrieval sources, tools and credentials, model versions, permissions and data classifications.
2. Run the generator on approved infrastructure with the architecture definitions, the accepted threat model and pinned catalogues as its only inputs.
3. Publish the candidate diff as a pull request comment, including boundaries changed without any candidate, and never write to the accepted model.
4. Run the automated soundness checks, failing the job if a candidate fails grounding or a changed boundary has no candidate.
5. The reviewer marks each candidate accepted, rejected or amended with a reason; rejected candidates are stored so that they are not raised again for the same architecture state.
6. Accepted threats name controls that resolve to deployed configuration, or a treatment item with an owner and due date; residual risk above appetite needs the risk owner's approval before merge.
7. On merge, version the accepted model, update the risk register and open any new detection or test case in the purple backlog of Chapter 26.

Default values to adapt. A reviewer decision on every candidate within two working days; no merge with an unreviewed candidate; a ten per cent quarterly re-review sample; and generator model changes treated as changes to the generator's own threat model.

Definition of done. Every pull request that touches a trigger path shows a candidate diff, every candidate has a named reviewer's decision, every accepted threat resolves to a control or treatment item, and the merged commit links the accepted model's version.

Worked example: threat-model.yaml for the citizen-service assistant

The illustrative threat model below belongs to Nordvik Region's citizen-service assistant, the Mode 4 system that answers residents' questions from published information (AIDR-0014). The triggering pull request adds a retrieval source, the disruption feed of a contracted public-transport operator, which creates a new trust boundary because third-party content now enters the index that steers the assistant's answers. The generator ran on the Region's self-hosted capacity, and one of the six analysts reviewed its candidates.

# Illustrative: Nordvik Region, threat model excerpt after PR 412
kind: ThreatModel
system: citizen-service-assistant
decisionRecord: AIDR-0014
version: 3.4.0
architectureCommit: 9f3c2e1
classification: classified-security    # public-access rules
trustBoundaries:
  - {id: TB-05, from: transport-operator-feed, to: retrieval-index}   # new
threats:
  - id: T-031
    boundary: TB-05
    stride: tampering
    owasp: LLM01
    atlas: AML.T0051.001               # indirect prompt injection
    vector: Instructions placed in disruption notices by whoever can write the feed
    impact: Residents receive misleading answers about services and contacts
    proposedBy: {generator: tm-generator@3.2, inputs: [9f3c2e1, atlas-pinned, owasp-2025]}
    review: {decision: accepted, by: nr-analyst-04, reason: feed is third-party writable}
    controls: [CTRL-CSA-021, CTRL-CSA-022, CTRL-CSA-007]
    residual: low
  - id: T-033
    boundary: TB-05
    stride: information_disclosure
    owasp: LLM02
    vector: Personal data of residents leaks into the operator feed
    review: {decision: rejected, by: nr-analyst-04, reason: feed flows inbound only}
controls:
  CTRL-CSA-007: Answers cite their source page; no actions on residents' behalf
  CTRL-CSA-021: Feed content indexed as untrusted data, never as instructions
  CTRL-CSA-022: Feed limited to schema fields; free text length-capped
closure:
  finding: T-031
  verifiedBy: NR-SEC-002 suite case csa-ind-017   # re-run passes

The rejected candidate stays in the file with its reason, so the next regeneration does not raise it again and an auditor can see that it was considered. The accepted threat ends, as every finding in Part III does, with the control that closes it and the test that verifies it: the AI red-teaming suite of NR-SEC-002 gains a case that plants benign marker instructions in a test copy of the feed and checks that the answers are unaffected, as Chapter 13 describes.

Across the usage modes

The usage mode decides who judges and with whose identity the AI acts, as defined in Chapter 6, and therefore the dominant threat to model.

Mode	Opportunity in risk assessment	Characteristic vulnerability to model	Who judges, with whose identity
Mode 0 Ambient AI	Vendor AI scores risk without effort	Automatic actions and scores the organisation cannot inspect	The vendor's model, the product's identity
Mode 1 Conversational assistants	Faster threat drafting and reading	A misled person acting on a plausible wrong answer	A person, own session
Mode 2 Embedded workplace assistants	Risk summaries in everyday tools	A summary that hides a critical path; injected content	A person, delegated access
Mode 3 Agentic work tools	Quick updates to architecture and threat models	An agent acting with an engineer's credentials before review	A person after the fact, engineer's identity
Mode 4 Built AI applications	Generator and path ranker at estate scale	Poisoned inputs steering scores	Named reviewers, read-only service identity
Mode 5 Autonomous production agents	Machine-speed risk models for machine-speed operations	A manipulated or compromised machine insider	Policy outside the agent and its operators, own identity

Mode 5 agents are modelled as well-meaning insiders that can be manipulated or compromised, so their permissions are scored as if an adversary held them.

Across the autonomy levels

The generator and the ranker are kept at low autonomy levels (Chapter 6), because their output is an organisational statement of risk.

Level	Use in threat modelling and risk assessment
Level 0 Advisory	An analyst asks for candidate threats or a path explanation and decides what to use
Level 1 Assisted	The generator drafts candidates and scores; a named reviewer decides each one; the default
Level 2 Supervised	The pipeline writes accepted items to the register after approval of each change
Level 3 Bounded	Reversible housekeeping only, such as rescoring after an approved assumption change, with the owner informed
Level 4 Autonomous	Not used: no AI accepts a threat, a risk or an exception

Verifying with security chaos engineering

The loop in Figure 10.1 can fail silently: a trigger path is renamed, an upgraded generator stops proposing AI-specific threats, or reviewers stop opening the comments. The experiment injects a condition, not an attack. In a test repository mirroring a production system's architecture definitions, the platform opens a pull request that adds a new trust boundary, such as a synthetic externally writable retrieval source or a new tool credential for an agent. The steady-state hypothesis is that the review job starts, the generator proposes at least one candidate with a valid STRIDE category and ATLAS or ATT&CK technique, the completeness check passes, and a named reviewer decides every candidate; a second variant adds a boundary to the identity graph and checks that the new edge appears and affected paths are re-ranked. The time bounds are candidates within 30 minutes, decisions within two working days and the new edge within 24 hours. A failed hypothesis goes to the purple backlog until the experiment passes again; Chapter 25 sets the safety boundaries.

The three AI focus areas

Securing AI systems. AI systems are compound systems whose threat models use STRIDE extended with ATLAS and the OWASP catalogues at the boundaries classical models omit (11, MITRE, n.d.) (14, OWASP, 2025). For high-risk systems, the model shows entry by entry that the attack classes of Article 15(5) of the AI Act were considered (1, European Union, 2024), and the generator and ranker have threat models of their own, led by poisoned intelligence and manipulated architecture inputs (20, NIST, 2025).

AI-enabled cyber defence. AI maintains threat models from architecture code on every relevant change, joins structured intelligence to asset context and ranks attack paths with checkable explanations, which makes ID.RA continuous rather than annual.

Thwarting AI-enabled attacks. Adversaries' AI capabilities enter the risk model as explicit, versioned assumptions about the time to exploit, the credibility of impersonation and the volume of targeted deception (37, NCSC, 2025) (34, ENISA, 2026), and paths that an AI-enabled adversary could traverse faster are ranked accordingly.

Nordvik Region and Hansa Mutual

Nordvik Region starts with the benefits-triage service and the citizen-service assistant, which carry the most personal data and consequence for residents. Their threat models live as code, candidates are generated on self-hosted capacity because the architecture's weaknesses are themselves sensitive, and one of the six analysts reviews each. For the procured benefits-triage service, the Region models the boundaries it controls as deployer and asks the provider for the rest, and it classifies accepted models because of public-access rules.

Hansa Mutual runs attack-path analysis across all seven markets, with its three-person threat-intelligence cell supplying AI-enabled adversary capability as explicit inputs. The paths that matter most run through agents: the IT-operations agent, a Mode 5 system with write access to production, is modelled as a possible machine insider, and a path from attacker-controlled alert content through HM-SEC-001 into its containment proposals is one of the first the red team takes up under HM-SEC-003. Named owners review paths and scores before they enter the group risk register and the HM-CSF-2027 profile, and each major change is assessed as DORA Article 8 requires. The two differ in breadth, not method: Nordvik models two services in depth because its reviewers are few, while Hansa needs the graph because no person can see every path across a group estate.

Summary

ID.RA Risk Assessment cannot rest on threat models drawn once in a workshop. Threat models belong in code beside the architecture and are regenerated whenever a data flow, trust boundary or component changes. AI proposes candidate threats using STRIDE with ATT&CK for classical boundaries and STRIDE extended with ATLAS and OWASP for AI systems, joins structured intelligence to asset and exposure context, and ranks attack paths through a graph of identities, resources and AI components for the red team to validate and the owners to cut.

Machine-generated risk enters the register only after a soundness check defined in advance: grounding, classification, mitigation family, likelihood and impact basis, completeness and stability, with the mechanical part automated and a named reviewer deciding each candidate against its provenance. The generator works at Level 1 Assisted, and no AI accepts a threat, a risk or an exception.

Securing AI systems means modelling them as compound systems, AI-enabled cyber defence means AI-maintained models and path analysis, and thwarting AI-enabled attacks means treating adversaries' AI capabilities as explicit inputs; the chaos experiment verifies that a new trust boundary triggers a reviewed model. Chapter 11 turns from assessing risk to reducing exposure, showing how vulnerability discovery and patching keep pace with exploitation at machine speed.

Sources

1. NIST. The NIST Cybersecurity Framework (CSF) 2.0. (19, NIST, 2024)
2. National Institute of Standards and Technology. Cybersecurity Framework Profile for Artificial Intelligence (Cyber AI Profile). (20, NIST, 2025)
3. Threat Modeling Manifesto Working Group. Threat Modeling Manifesto. (25, Threat Modeling Manifesto Working Group, 2020)
4. MITRE Corporation. MITRE ATLAS: Adversarial Threat Landscape for AI Systems. (11, MITRE, n.d.)
5. MITRE Corporation. MITRE ATT&CK: Adversarial Tactics, Techniques, and Common Knowledge. (10, MITRE, n.d.)
6. MITRE Corporation. MITRE D3FEND: A knowledge graph of cybersecurity countermeasures. (12, MITRE, n.d.)
7. OWASP Foundation. OWASP Top 10 for Large Language Model Applications, 2025 edition. (14, OWASP, 2025)
8. OWASP Foundation. OWASP Top 10 for Agentic Applications for 2026. (15, OWASP, 2025)
9. OASIS Cyber Threat Intelligence. STIX Version 2.1. (24, OASIS Cyber Threat Intelligence (CTI) TC, 2021)
10. European Parliament and Council of the European Union. Regulation (EU) 2024/1689 (Artificial Intelligence Act). (1, European Union, 2024)
11. European Parliament and Council of the European Union. Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). (2, European Union, 2022)
12. European Parliament and Council of the European Union. Directive (EU) 2022/2555 (NIS 2 Directive). (3, European Union, 2022)
13. European Union Agency for Cybersecurity. ENISA Threat Landscape 2026. (34, ENISA, 2026)
14. National Cyber Security Centre. Impact of AI on cyber threat from now to 2027. (37, NCSC, 2025)
15. Charrier, C. & Weiner, R. How Low Can You Go? An Analysis of 2023 Time-to-Exploit Trends. (44, Charrier & Weiner, 2024)
16. Magramo, K. Arup revealed as victim of $25 million deepfake scam involving Hong Kong employee. (55, Magramo, 2024)
17. Nordqvist, G. Architecture as Code. (71, Nordqvist, 2025)
18. Nordqvist, G. AI Platform and Security: Building and Securing AI Platforms in European Enterprises and Public Authorities. (73, Nordqvist, 2026)