AI Risk Management

AI introduces a distinct and in many ways more complex risk profile than previous generations of enterprise technology. The risks are not merely larger versions of familiar IT risks — they include new categories of failure, new mechanisms of harm, and new governance challenges that traditional risk management frameworks are not designed to address. Executives who apply conventional IT risk thinking to AI consistently underestimate the risks that matter most and over-invest in mitigating risks that are relatively manageable. This module provides a systematic framework for understanding and managing AI risk.

The AI Risk Landscape

AI risks can be organized into six major categories, each with distinct characteristics, manifestation patterns, and mitigation approaches. Understanding the full landscape — rather than focusing narrowly on the risk category most familiar from IT or regulatory experience — is the starting point for effective AI risk governance.

Accuracy and Reliability Risk

AI systems make errors. This is not a defect to be eliminated — it is a structural property of probabilistic systems. The risk management question is not "how do we achieve zero errors" but "what error rates are acceptable for what applications, and what happens when errors occur?" A 95% accurate fraud detection system operating on a billion transactions will generate 50 million false positives annually. Whether that is acceptable depends on the cost of false positives, the alternatives, and the harm of the true positives it catches. These tradeoffs must be explicitly analyzed, not assumed.

Reliability risk extends beyond error rates to system stability: does the model perform consistently over time, across different input distributions, under different loads? AI systems can fail in ways that are more subtle and harder to detect than conventional software failures. A model can degrade gradually — performing worse and worse as the world it was trained on diverges from the world it is operating in — without any error occurring in the conventional sense. Production monitoring systems specifically designed to detect performance drift are a necessary component of AI reliability management, not optional.

Bias and Fairness Risk

AI systems trained on historical data can encode, replicate, and in some cases amplify historical patterns of discrimination. This is not a hypothetical concern: there are well-documented cases of AI systems producing biased outcomes in consequential domains including hiring (Amazon's discontinued resume screening tool), lending (the Consumer Financial Protection Bureau's investigations of algorithmic credit scoring), healthcare (a widely studied algorithm that systematically underestimated care needs for Black patients), and criminal justice (the COMPAS recidivism scoring system).

The mechanism of algorithmic bias typically involves one of three patterns. First, training data bias: if historical hiring decisions were biased against women, a model trained on those decisions will learn to discriminate against women. Second, proxy variable bias: even if protected characteristics (race, gender) are excluded from the model, the model can learn to use correlated variables (zip code as a proxy for race, name as a proxy for gender) to achieve similar discriminatory effects. Third, measurement bias: if the outcome variable used to train the model is itself measured differently for different groups, the model will reflect that measurement inequality.

Bias risk management requires proactive evaluation — testing AI system outputs for differential impact across protected groups before deployment, not reactively after harm has occurred. It also requires ongoing monitoring: a model that is fair at deployment can develop unfair patterns over time as the world changes or as the model is exposed to different operational distributions.

Security and Adversarial Risk

AI systems introduce new attack surfaces that did not exist in conventional software. Three categories of adversarial AI attack are most practically relevant for enterprise risk management.

Adversarial inputs are carefully crafted inputs designed to cause AI systems to produce incorrect outputs. In computer vision, minimal perturbations to an image that are invisible to the human eye can cause image classifiers to confidently misclassify the image. In natural language processing, carefully crafted prompts can cause large language models to bypass safety filters or produce outputs they were designed not to produce. For AI systems deployed in security-critical contexts — fraud detection, malware identification, content moderation — adversarial robustness is a primary design requirement.

Data poisoning attacks involve injecting malicious data into training datasets to cause the resulting model to behave incorrectly in specific, attacker-defined ways. This is particularly concerning for systems that continuously learn from operational data — a model that updates itself based on user interactions can be poisoned by an adversary who manipulates those interactions.

Model extraction and inversion attacks allow adversaries to reconstruct either the model itself or the training data from which it was derived, through repeated queries to the model's API. This can expose proprietary intellectual property (the model architecture and weights) or sensitive training data (private information about individuals who appeared in the training set).

Regulatory and Compliance Risk

The regulatory landscape for AI is evolving rapidly, and the trajectory is clearly toward more, not less, regulation. The EU AI Act — the most comprehensive AI regulatory framework currently in force — creates binding requirements for AI systems in high-risk categories including credit scoring, hiring, educational evaluation, law enforcement, and critical infrastructure. US federal agencies have issued sector-specific AI guidance across financial services (OCC, CFPB), healthcare (FDA), and other regulated industries. The UK's sector-based approach is producing similar requirements through existing regulators.

The compliance risk for AI is amplified by two factors. First, many organizations have deployed AI systems without conducting the formal risk assessments, documentation, and testing that emerging regulations require — creating retroactive compliance exposure. Second, the regulatory requirements for AI systems are more process-oriented than most technology regulations: they require not just that the system produces certain outcomes, but that it was developed, tested, and monitored using defined governance processes. Organizations that lack the documentation of their AI development and testing processes face significant remediation costs as regulations take effect.

Reputational Risk

AI failures generate reputational consequences that are disproportionate to their operational impact, for two structural reasons. First, AI failures often involve harms to identifiable individuals that are easy to narrate compellingly — an AI hiring system that discriminates against a specific candidate, an AI lending tool that denies a loan to a specific small business owner, an AI content moderation system that silences a specific voice. These individual stories generate public and media attention that aggregate statistical harms (affecting many people less severely) often do not. Second, AI failures are perceived as intentional in a way that conventional software failures are not — the organization is seen as having made an affirmative choice to deploy a system that caused harm, rather than having experienced a technical accident.

The reputational risk calculus requires scenario planning for plausible failure modes in your specific AI applications. What is the worst-case story a journalist or regulator could write about your AI system's behavior? What evidence would be cited? What would the organizational response be? Organizations that have thought through these scenarios — and that have governance structures capable of responding rapidly — handle reputational incidents dramatically better than those that encounter them unprepared.

Operational Dependency Risk

As organizations integrate AI more deeply into critical processes, they create new forms of operational dependency that introduce concentration and fragility risks. A financial services firm that deploys AI-assisted trading across multiple strategies becomes dependent on that system's continued correct functioning. A healthcare organization that integrates AI diagnostic support into clinical workflows creates a dependency that must be managed if the AI system is unavailable or performing unreliably. An organization whose customer service is substantially AI-powered must plan for scenarios in which the AI system is unavailable, performing poorly, or subject to adversarial attack.

Operational dependency risk management requires explicit continuity planning: what is the fallback if the AI system fails? Can processes revert to pre-AI approaches? Are the human capabilities required for those fallbacks being maintained, or has the organization allowed them to atrophy? The organizations most exposed to operational dependency risk are those that have reduced human capacity in processes where AI is deployed without maintaining adequate fallback capability.

Risk Assessment Frameworks

The NIST AI Risk Management Framework (AI RMF), published in 2023, has emerged as the most widely adopted voluntary framework for enterprise AI risk assessment in the United States. Its four core functions — Govern, Map, Measure, Manage — provide a practical structure for building organizational AI risk management capability.

The EU AI Act's risk-tiering approach provides a complementary framework particularly relevant for organizations with EU market exposure: categorizing AI applications by risk level (unacceptable, high, limited, minimal) and applying proportionate governance requirements accordingly. The practical value of this framework for non-EU organizations is in forcing explicit risk categorization of AI applications — a discipline that many organizations lack even when they are not subject to the regulation directly.

For organizations seeking a more quantitative risk assessment approach, the ISO 31000 risk management standard adapted for AI provides a framework for estimating probability and impact of AI risks across the categories described above, enabling portfolio-level risk prioritization.

Governance Structures for AI Risk

Effective AI risk governance requires clear accountability structures. In practice, AI risk governance involves three tiers.

Policies Every AI-Deploying Organization Needs

A minimum viable AI governance policy framework includes the following documents, each with clear ownership, defined scope, and regular review cycles.

• AI Use Policy — defines acceptable and unacceptable uses of AI within the organization, including both internally built systems and commercially procured AI tools used by employees.
• AI Development Standards — defines required practices for AI systems built internally: data governance requirements, bias testing standards, documentation requirements, performance benchmarks, and monitoring obligations.
• AI Procurement Policy — defines required vendor due diligence, contract provisions, and ongoing vendor management standards for commercially procured AI systems.
• AI Incident Response Plan — defines how the organization detects, escalates, investigates, and responds to AI system failures, with clear escalation thresholds, defined response teams, communication protocols, and regulatory notification obligations.
• AI Acceptable Use Policy for Employees — defines what AI tools employees are authorized to use, what data can be input into external AI systems, and what human review is required before AI-generated content is used in consequential decisions or external communications.

Incident Response for AI Failures

When an AI system produces a significant failure — incorrect outputs that cause material harm, biased behavior that generates legal or reputational exposure, security incident, or unexpected operational impact — the response quality depends entirely on preparation. Organizations that are encountering their first significant AI incident for the first time, without a practiced response plan, consistently perform worse than those that have planned and rehearsed.

Effective AI incident response has five phases: detection (systematic monitoring that identifies failures before they cause significant harm, rather than learning about them from external parties), containment (defining the scope of the failure and limiting further harm, including potential system suspension), investigation (understanding the root cause — data problem, model failure, deployment issue, adversarial attack), remediation (addressing the root cause, not just the immediate symptom), and communication (transparent communication with affected parties, regulators, and the public according to defined protocols).