AI RiskWhere could the AI deviate — either way?
AI Risk

AI risk is now the hard part, and it is wider than the model.

Deploying AI is no longer the challenge. The challenge is knowing where it creates exposure: where it could breach regulation or go beyond your own risk appetite, and where it could fail in ways traditional software doesn’t. But exposure is only half of what risk means, and it’s the more familiar half.

Written against
ISO 31000:2018 · ISO/IEC 23894:2023 · ISO/IEC 42001:2023 · ISO/IEC 38507:2022 · NIST AI RMF 1.0 · Regulation (EU) 2024/1689
Last updated
Answered first

How do we assess and rate the risk of the AI we already use?

Start with an inventory of what is running and what it touches, then assess each system against criteria the board has approved. The assessment has three moves: identify what is running, analyze how each system can fail across seven risk domains, and evaluate the results against the criteria set at the start. Each risk is then avoided, reduced, shared or accepted.

Are we a deployer or a provider of the AI we use?

If you use an AI system under your own authority you are a deployer, even when the AI came inside software you bought. Your role can shift to provider under Article 25 if you put your name on a high-risk system, substantially modify one, or repurpose it for a high-risk use it was not sold for. Role matters more than dates in deciding what the EU AI Act asks of you. The EU AI Act and its dates

What is a risk threshold that acts?

A threshold is the level an indicator may reach before something has to happen, so each one has an owner, a notification window and a decision it triggers. Set thresholds from the risk appetite the board approves (ISO/IEC 38507), and set them before the assessment. Each also has a direction, because some indicators cut both ways: a falling override rate may mean better AI or less attentive reviewers.

The principle underneath

Risk management exists to create and protect value

Risk runs in both directions. ISO 31000 states the purpose of risk management in one line: the creation and protection of value. That choice of words changes what the work is for.

ISO 31000 defines risk as the effect of uncertainty on objectives. ISO/IEC 42001 uses the harmonised form, the effect of uncertainty. Note 1 to the entry keeps the part that matters most: an effect is a deviation from the expected, in a positive or negative sense (clause 3.7). Its planning clause carries the same idea in its title: actions to address risks and opportunities (6.1). So upside is part of the definition, not an optimistic add-on to a threat register.

Take an AI vision model in quality inspection. It can miss defects or drift as the products change, and it can also cut scrap and speed up inspection. A programme that counts only the first side will under-invest in the AI that’s working. It hands the board a list of threats with no view of where the return sits. The same evidence answers both questions, so we treat the risk work and the value work as one engagement, seen from two directions.

AI enters an organization through many doors: sanctioned platforms, features inside vendor software, tools employees bring in, and, increasingly, autonomous agents. Each one can shape a decision, touch personal data or trigger an action. When you buy a tool that arrives with AI switched on, nothing in the purchase triggers a classification, so the review has to be designed into the process.

So the first failure comes before the model: not knowing where the risk is. A system that looked harmless in a demo becomes a high-risk system the moment it informs an employment, credit or safety decision. Exposure hides in tools nobody catalogued and in data flows nobody scoped as AI, until an auditor, a regulator or an incident brings it out.

AI risk spans seven domains. Each has its own failure modes, owners and controls, and each becomes expensive in a different way. A credible risk picture covers all seven, not just the accuracy of the model. Bias appears twice on purpose: in the model domain it’s a failure mode, and in the human domain it’s a consequence. That difference decides the remedy, because the first needs engineering and the second needs recourse.

  • 01

    Technical & model risk

    Hallucination, drift, bias and brittleness. Models fail probabilistically and degrade quietly as the world moves away from their training data, because a degraded model answers with the same confidence as a healthy one. So the typical failure isn’t an outage. It’s a slow decline in decision quality that nobody is watching for, found when a customer complains or an auditor samples the output.

  • 02

    Data risk

    Quality, provenance, privacy, poisoning, and the copyright exposure carried in training data. Data failures don’t show up in testing: a model trained on flawed or improperly licensed data passes its benchmarks and fails where it matters. The consequence usually arrives from outside. A regulator asks where the data came from, a rights holder asks under what licence, or a person asks to be removed from something that can’t easily forget.

  • 03

    Operational risk

    Shadow AI, vendor concentration and integration failure. The largest uncontrolled exposure is often the AI nobody approved, such as a feature switched on inside a tool you already pay for, running outside sanctioned workflows. It carries the same obligations as a deliberate deployment, but none of the controls. Vendor concentration adds to it: when one provider sits behind six of your processes, their outage, price change or deprecation becomes your continuity problem. ISO 22301 is the management system for business continuity, and an AI dependency belongs in its scope like any other operation the organization cannot afford to lose.

  • 04

    Governance risk

    No clear owner, no inventory, no audit trail. You can’t govern what you can’t see. Naming every AI system in use is harder than it sounds, and naming who is accountable for each is harder still. When this domain is weak, failures in the other six arrive unannounced. Without an owner, it’s nobody’s job to notice, and without records, nobody can reconstruct what happened once someone does.

  • 05

    Human & societal risk

    Bias and discrimination in their effect on people, loss of meaningful human review, over-reliance, manipulation, deskilling, workforce impact, and harm to the people affected by automated decisions. The other domains describe how AI fails. This one describes where the failure lands: an applicant screened out, a claim declined, an employee assessed, a customer priced. So it asks questions the model domain can’t. Can a person find out that AI was involved, contest the outcome and reach somebody with the authority to change it? Do the reviewers still have the standing to disagree with the system? And what capability does the organization lose when work moves to a model it can no longer operate without? ISO/IEC 42001 clause 6.1.4 requires a process for assessing the potential consequences of AI systems for individuals, groups of individuals and societies, and ISO/IEC 42005 supplies the method. The deployer, not the vendor, answers for the result.

  • 06

    Legal & regulatory risk

    Penalties, liability split between providers and deployers, and conflicts between jurisdictions. The fine is rarely the whole exposure. There’s also the obligation to suspend a system mid-process and the burden of proving how a decision was reached. And there’s the liability that passes on through your own contracts to customers who relied on the output.

  • 07

    Agentic risk

    Autonomous agents that plan, call tools and pass context to one another. Traceability breaks where context gets compressed, and boundaries blur where one agent calls another. The decision path a control was written for is put together at runtime, not at design time. There’s no adequate control catalogue for agents yet, so the controls have to be derived from the risk itself.

A board asks what happens if this risk is real. Likelihood on its own doesn’t answer that, and for AI the answer is rarely a single event.

The first cost is wrong decisions at scale. A system that has drifted doesn’t stop. It keeps deciding, at volume, until somebody notices. The damage builds up in the gap between degradation and detection, and the gap stays open until monitoring is in place to close it.

The second cost is remediation. Re-running decisions, contacting the people affected and correcting downstream records is manual work, and it scales with exactly the volume the system was bought to handle. Automation that saved a year of effort can cost more than a year to unwind.

The third cost reaches beyond your organization, because your contracts promise outcomes to your own customers and a model failure doesn’t stay in-house. Then there’s the copyright and IP exposure carried in generated output. And adding provenance, logging and oversight to a system already in production always costs more than designing them in.

What the incident record shows
  • —Only around 2% of reported AI incidents occurred pre-deployment. Almost all harm materialises in operation, not in development.
  • —Roughly a third of reported incidents involve systems that would fall in the EU AI Act’s high-risk tier; a small minority in the prohibited tier.
  • —About half were assessed as intentionally caused, which means about half were not.
  • —The largest domain is malicious use, dominated by fraud, scams and targeted manipulation. The next is system safety, failures and limitations.

MIT AI Risk Initiative incident tracker, classifying reports from the AI Incident Database. Treat the proportions as indicative rather than representative: reporting is voluntary, so the sample skews toward incidents somebody chose to publicise, and classification is automated, with spot-checks rather than case-by-case expert review.

The 2% figure matters most here. If almost all harm materialises after deployment, then almost all of it materialises inside the deployer’s process. That’s the organization running the system rather than the one that built it, whoever ultimately answers for it.

Every finding needs a yardstick. Without one, an assessment produces observations, and nobody is obliged to act on an observation.

Risk appetite is the amount and type of risk an organization is willing to pursue or retain. For AI, it has to be stated in terms you can actually measure the portfolio against. How much model uncertainty is acceptable, and in which decisions? How much autonomy may a system exercise unaided? How much opacity is tolerable where an outcome affects a person? And how much dependence on a single provider will the business carry?

ISO/IEC 38507 places that decision with the governing body, and that matters more than it sounds. A threshold set by the team that built the system is a preference. The same threshold approved by the board is a control, with an owner and a consequence when it’s breached.

Tolerance is the narrower question: how far a specific system may deviate before somebody has to act. The indicators further down are measured against it, so tolerances are set before the assessment, not after. If you choose thresholds after you know the findings, it’s tempting to choose them so that nothing has breached.

Finding exposure isn’t a one-off exercise. The work follows the risk management process defined in ISO 31000, which NIST’s AI RMF re-cuts for AI. There are four stages, one foundation, and a governance layer that runs throughout.

Govern — across all four

Communication, consultation, recording and reporting aren’t a stage. ISO 31000 runs them alongside the whole process, and NIST lifts the same idea into a cross-cutting function that informs the other three. Governance that only happens at the end is exactly the failure this page is about.

MAP

Scope, context and criteria

What the assessment covers, which systems and processes fall inside it, and the criteria they will be judged against. Those criteria include the risk appetite approved at board level under ISO/IEC 38507. Findings without criteria are observations. Findings against criteria are decisions.

MAP · MEASURE

Risk assessment

This stage has three moves. Identify what is running and what it touches. Analyse how each system can fail across the seven domains. Then evaluate the result against the criteria set in stage one. Classification under the EU AI Act happens here, with assessment against NIST AI RMF and ISO/IEC 42001.

MANAGE

Risk treatment

Decide what changes: design and build controls, formally accept the exposure, stop an activity, or share the risk with a third party. This is the stage where findings become decisions.

MEASURE · MANAGE

Monitoring and review

Indicators are measured against the thresholds set in stage one, so movement shows up before an audit does. When the portfolio grows, the regulation changes or the appetite shifts, the process goes back to stage one.

Foundation — the AI inventory

Every use case and every AI system in one place, proposed as well as running. ISO/IEC 42001 never uses the word inventory, but its Annex A asks for the resources behind each AI system to be identified and documented (A.4.2). NIST AI RMF asks for the inventory outright (GOVERN 1.6). Without it, the four stages above run on memory. And the inventory reaches one level higher. In our method, the top-level object is the business process, not the AI system. Each AI system connects to the process inventory: the business process and its information first, where the value lives, then its owner and objectives, the supporting assets and AI models, suppliers, risks, opportunities, impacts on the people affected, controls, evidence and the value realized. Quality, security and AI governance can each read it from their own direction.

NIST AI RMF: how we use it

ISO 31000 is explicit that the process is iterative, and AI portfolios change faster than most risk registers were designed for.

Stage 03 in full — the four decisions
Avoid

Don’t run the use case, or stop running it. It’s the easiest option to leave off the table, because stopping can look like giving up when it’s a decision like the other three.

Reduce

Design and build controls until the residual risk sits inside appetite. This is where the architectural guardrails come from, and where most of the effort in an engagement usually goes.

Share

Move part of the exposure to a third party by contract or insurance. A deployer’s regulatory duties do not transfer, whatever the contract says.

Accept

Retain the risk deliberately: recorded, owned and revisited. A documented accepted risk is a governance artifact, not a failure. Without a way to accept risk deliberately, it gets accepted by default, silently, by whoever happens to be closest to it.

Regulation is one of the seven domains of risk. It gets more attention than the others because it’s the one that arrives with a deadline and an enforcer attached.

Two things change your exposure more than any date. The first is your role: the EU AI Act assigns duties by role, and an organization using an AI system under its own authority is a deployer. Buying software with AI inside and using it makes you one, whether or not procurement noticed. The second is that the line between roles moves. Article 25 of the EU AI Act covers responsibilities along the AI value chain. Under it, the full set of provider obligations can attach to you instead. Three things can trigger that: putting your own name on a high-risk system, modifying one substantially, or repurposing a system into a high-risk use it was not sold for.

Some obligations already apply. Transparency duties and enforcement against general-purpose AI providers have applied since August 2026. The deferral of the high-risk tiers changed when those obligations begin to apply, not what they require. The standards page sets out the full timetable, the obligations role by role, and the supervisory authorities for the DACH region and Canada.

A risk function that counts only the downside is doing half its job.

Under-investment is exposure too, and so is a control posture so cautious that value never ships. A governance function that can only say no is creating risk of its own. And falling short of the expected outcome is as real a deviation as an incident.

The competitive version is sharper. If a comparable organization automates a process you left manual, the gap grows while your register records nothing, because nothing went wrong. A programme that only looks for threats can’t see this kind of risk.

The inventory that maps exposure also shows where a system is under-used and where a proven capability could be extended to a related decision. It shows where a manual process sits beside an automated one doing similar work. The same rows are a risk register when you read them for downside, and a pipeline when you read them for upside. That’s where the AI Value work picks up.

These indicators measure exposure, not whether the compliance work got done. Whether classifications are complete and assessments have been filed is control effectiveness, and that sits with the governance indicators. The ones below move when the risk itself moves.

A · Exposure — how much is riding on it
  • —Consequential decisions influenced by AI per period, and the direction of travel.
  • —Share of those taken without human review before they take effect.
  • —People subject to an AI-influenced decision per period.
  • —Value flowing through AI-influenced decisions: credit extended, claims settled, candidates screened, spend approved.
  • —Systems running in production that fall in the high-risk tier.
  • —Share of business-critical processes dependent on a single model or provider.
B · Early warning — signals that move before harm does
  • —Human override and correction rate on AI output. It’s the strongest signal available without access to the model, and it rises before quality failures become visible.
  • —Share of outputs falling below an agreed confidence threshold.
  • —Drift against baseline, where the model is yours to instrument.
  • —Share of inputs failing validation, and retrieval corpus past its freshness limit.
  • —Shadow-AI discoveries per period, and time to bring each into the sanctioned portfolio.
  • —Vendor AI features activated without review: the bought-software exposure, made measurable.
  • —Systems whose intended purpose has changed since classification: the EU AI Act Article 25 trigger that can turn a deployer into a provider, watched for instead of discovered later.
  • —Agent actions outside the defined action space, and the depth of agent-to-agent delegation.
  • —Challenges raised against AI-influenced decisions, and the share overturned. A contest rate near zero is a finding in itself: it usually means the route to challenge is too hard to use.
C · The other direction — opportunity, measured
  • —Decisions still made manually where an approved capability already exists.
  • —Use cases stalled between gate approval and production, and for how long.
  • —Provisioned capacity paid for and unused, where no decision has been taken either to use it or release it.
  • —Time from an opportunity being identified to a decision being taken on it.

These are illustrative. A usable set comes from the appetite set out earlier on this page. The threshold makes an indicator actionable, and the approved appetite makes the threshold defensible. Several need instrumentation that has to be built first, and building it is usually part of the work itself.

What happens when one breaches

Without an escalation path, an indicator is just a dashboard tile. Each threshold needs three things before it’s useful: who is told, within what period, and what decision it triggers. That decision might be re-assessment, suspension, or acceptance at a level with the authority to accept. Each threshold also needs a stated direction, because several of these indicators cut both ways. A falling override rate can mean the model improved, or that the reviewers stopped looking.

Foundation

ISO 31000 and ISO/IEC 23894

AI risk management is a specialization of established practice. ISO 31000 supplies the principles, framework and process, including the definition this page opens with. ISO/IEC 23894 layers the AI-specific guidance on top. It mirrors the same clause structure and extends the principles where AI behaves differently, particularly around dynamism, the quality of available information, and human and cultural factors. It cannot be implemented on its own, because it assumes the general baseline underneath. So where a mature risk function already runs, the machinery is largely in place and only the AI-specific extension is new. NIST’s AI RMF re-cuts that process as Govern, Map, Measure, Manage, so governance runs continuously instead of as a precondition. If you already run an ISO 31000 practice, adopting it is an extension, not a second programme.

More questions

Why is AI risk wider than the model?

Exposure also comes from sanctioned platforms, vendor features, employee tools and autonomous agents. The risk surface covers seven domains, each with its own failure modes, owners and controls: technical and model, data, operational, governance, human and societal, legal and regulatory, and agentic.

Does AI risk only mean downside?

No. Under ISO 31000, risk management exists to create and protect value, so upside is part of the definition. Under-investment and over-cautious controls are exposure too, which is why risk work and value work belong in one engagement.

What do we measure to watch AI risk?

Three groups of indicators: exposure, early warning and opportunity, each with an escalation path. That keeps the monitoring tied to decisions the board has already agreed to take.

Get a defensible view of your AI risk.

Tell us where AI operates in your organization. We will respond with a read on where the exposure sits, where the upside sits alongside it, and a sensible first step — usually a bounded AI risk assessment.