AI incident response plans: Not just for security anymore

It's a given that artificial intelligence systems will fail. What shouldn't be inevitable is a catastrophic outcome to that failure. For organizations that design, develop, sell or operate AI systems, preparing for the day a failure happens is a must — and can make all the difference between a timely, controlled response and chaos.

Types of failures and risks unique to AI systems

Based on a breakdown of the most common failure modes among reported incidents on the AI Incident Database, AI failures cluster around several categories: security, unauthorized outcomes, discriminatory outcomes, privacy violations, physical safety, and lack of transparency and accountability.

These failures may face inward, directly affect users or consumers, or have even a more widespread impact. The extent to which a company is prepared to effectively and efficiently respond can limit both legal consequences and public outcry. While databases like the AI Incident Database offer invaluable insights into the types of incidents that have occurred, they also underscore the importance of updating the data that underpins your AI system. Such inaccuracies can have serious repercussions, from financial losses to compromised health and safety, and can result in both regulatory penalties and reputational damage.

If an incident happened today, could your employees or affiliates answer the following?

• What constitutes an "incident"?
• What policies apply when an incident has occurred?
• Have these policies been tested or practiced?
• What roles and responsibilities attach to identified individuals or groups?
• Who should be notified, who should do the notifying, and at what stages?
• Who can make the "no-go" decision to pause or stop productions or operations?

Answering these questions is the start of an incident response plan. And they are necessary because the risks associated with AI systems are different from those in traditional software. While concerns around data and model bias have received significant attention, other unique aspects are less well known. One of the major risks is model decay, even more than with static computer programs.

AI models are often trained on historical data and their accuracy can diminish over time if not updated. For instance, if the data an AI system was trained on predates the COVID-19 pandemic, its continued operations may not accurately represent current realities, leading to flawed predictions or recommendations. This makes continuous monitoring and recalibration essential.

Another concern is the exceptionally complex nature of many AI systems, particularly those implementing various forms of deep learning, neural networks or generative AI. With multiple layers of interconnected nodes and parameters, it can be difficult to pinpoint where an error occurred or how a faulty decision was made. Unlike deterministic systems where outputs are predictable, AI systems deal in probabilities and uncertainties. When errors do occur, their effects can multiply exponentially, compounding minor inaccuracies into major failures or harms.

Finally, basic security for AI systems differs from traditional network security. Model extraction, data poisoning, membership inference, adversarial attacks and other vulnerabilities specific to machine learning pose extra levels of security risk that can result in dangerous, or at best embarrassing, outcomes.

Building an AI incident response plan

AI incident response plans need to go beyond the scope of traditional cybersecurity measures, which are primarily oriented toward meeting threats such as hacking or data breaches. While these remain concerns for AI systems, they are only part of the picture. AI brings a host of intrinsic vulnerabilities based on the varied and additional failure modes discussed above.

As a foundation for building an incident response plan, a complete inventory of AI systems is a critical first step. Furthermore, evaluating "incidents" requires some kind of measurement — having a baseline for model operations helps warn when things go astray. Finally, internal oversight resources are always limited, so they should be used to focus on the most material models to prevent widespread or damaging harms.

With this foundation in place, following the pattern of traditional incident response processes as commonly implemented in information security programs can be a useful framework, if revised to address these additional threats. Traditional response plans typically consist of six stages: preparation, identification, containment, eradication, recovery and lessons learned. Below is a brief map of how each phase might be applied to AI incidents.

Preparation

• Design policies and procedures addressing internal operations around an AI incident response, defining terms and thresholds, and assigning clear roles and responsibilities.
• Establish categories and rankings of foreseeable harms and devise general processes for dealing with the "unforeseeable" outcomes.
• Ensure inclusion of events in which unintended model outputs occur (inaccuracy, bias, etc.), as well as results from external attacks and malicious actors.
• Ensure policies and procedures encompass failures at all stages of the model life cycle.
• Institute initial and recurring training (individual as well as organizational) to operationalize policies, ensuring sufficient awareness and competence after an incident.

Identification

• Follow procedural standards to detect and verify that an incident has occurred.
• Monitor models and systems for the full range of harms potentially generated by AI failures.
• Activate or access channels to accept notices, inputs and real-time feedback from affected business partners, consumers or other third parties.
• Establish directed procedural controls for creating relevant logs, notifications and information sharing.

Containment

• Address the immediate damage first, then stop or pause operations, engage temporary alternatives where available and begin documentation, tracking and backup procedures as appropriate.
• Implement procedural directives to scope, evaluate, elevate, respond or otherwise address near- and long-term harms as anticipated from the incident.
• Apply technical fixes as identified by the engineering, project or programming teams to mitigate harms and minimize further problematic performance.

Eradication

• Formally remove operational systems or pull systems in development until sufficient reviews and mitigation can confirm no further harms will occur relative to the specific incident.
• Employ extensive and documented testing on revised or replacement systems, particularly related to the harms from the known incident.
• Review for any other corollary performance errors in the affected system, and look for any "upstream" or "downstream" dependencies that may be affected.

Recovery

• The newly revised or replacement model should be hardened before deployment.
• Benchmark and document the new/repaired model performance metrics and outputs before resuming development or returning to production-level operations.

Lessons learned

• Review documentation generated in response to the incident, create summarized reports and analyses of problematic outcomes as well as organizational actions in response, and identify gaps or areas of particular success or effectiveness.
• Share such documentation institutionally with those responsible for the incident response processes and incorporate it into future training and testing scenarios as appropriate. Review and update policies as required.

There is sometimes a notion that incidents or failures involving AI are purely technological challenges, a perspective which can be reductive and misleading. AI systems do not exist in a vacuum. They are integrated into complex organizational, ethical and legal frameworks. As such, when AI systems go rogue, the implications are multidimensional. This reality should encourage us to think of responses to incidents in a holistic manner.

One best practice to address this is to foster interdisciplinary teams. Identified risk managers can play a critical role in coordinating among technologists and legal professionals and can contribute to strategies that encompass multiple dimensions of AI deployment. Each role contributes distinct, yet complementary, expertise, working collectively to manage the complex risks that AI presents.

Implementing and managing the AI incident response plan

In terms of accountability and governance, designating a specific individual or team as the entity responsible for overseeing the AI incident response plan should start the process. This could be a chief risk officer (within the financial services sphere), a chief policy officer, an AI ethics board or a dedicated AI governance committee. Their role extends beyond merely drafting the plan; they are also responsible for its periodic review, testing, training and auditing.

Accessing resources such as the U.S. National Institute of Standards and Technology's AI Risk Management Framework and the Organisation for Economic Co-operation and Development's AI Principles can be helpful in incident response planning, as in all other aspects of responsible AI governance. Also consider the recently published AI Harm Framework by the Center for Security and Emerging Technology's, which provides a systematic approach to defining, tracking and understanding harms caused by AI, and complements the risk-management perspectives offered by NIST and the OECD.

To start with a basic review, Luminos.Law, in collaboration with the Future of Privacy Forum, offers a set of ten questions aimed at gauging potential vulnerabilities. These questions help identify the scope of AI models, their intended outputs and reaches, while focusing on the procedural rigor around auditing and continuous monitoring of AI-related liabilities. One of the guide's key aspects prompts organizations to consider sociological biases in their models and to adhere to established standards for trustworthy AI.

The process of systematically cataloging and taxonomizing AI incidents has been underway for a while now. The AI Incident Database mentioned above has collected over 560 separate incidents, some of which include responses enacted by the deployers and developers of those systems. Similarly, other databases which may provide useful examples are the AI Vulnerability Database, the AIAAIC, and George Washington University Law School's AI Litigation Database.

Crafting an AI incident response plan is not an optional undertaking. It is a mandatory component for any entity that interacts directly or indirectly with AI systems. From acknowledging the unique risks associated with AI systems to emphasizing the importance of cross-disciplinary realms of expertise, preparation is key. Responding to tomorrow's problems will only be successful if planning starts today.