Skip to content
Olivier Reuland

Your Incident Response Playbook Wasn't Built for GenAI

GenAI incident response breaks deterministic assumptions, and multi-agent systems make containment and reconstruction harder by spreading bad context across tool chains and agent handoffs.

A chatbot giving you a different answer twice is annoying. A chain of agents doing the same thing while calling tools is an incident response problem.

That is the gap a lot of teams still have not absorbed. Traditional incident response assumes that if something goes wrong, you can reproduce it, trace it, fix it, and verify the fix against a stable baseline. GenAI systems break that assumption, and multi-agent systems break it harder.

The OWASP GenAI Incident Response Guide 1.0 is useful because it starts from the right place: security practitioners need a way to investigate systems whose outputs vary, whose dependencies are wide, and whose reasoning is partly opaque. Pair that with OWASP's Multi-Agentic System Threat Modeling Guide v1.0 and the operational picture gets sharper very quickly.

The problem is not only non-determinism. It is propagation across tools and agents. One agent reads a poisoned document. Another agent trusts its summary. A third agent takes action based on the summary. By the time the bad outcome is visible, the original trigger is buried inside multiple handoffs.

Incident response for GenAI is not just about preserving prompts and outputs. It is about preserving the chain of context, tool calls, and agent-to-agent handoffs that made the bad action look reasonable at the time.

Why the old playbook feels thin

That is why the old playbook feels thin.

If the same input can produce different outputs, you cannot rely on simple reproduction as your main investigative move. If tools are chained, the attack surface is no longer just your application code. It includes plugins, upstream data sources, internal APIs, external services, orchestration layers, and the trust relationships between agents. If one compromised or misled component can feed another, your incident is not isolated. It is contagious.

The NIST incident response lifecycle still gives you the right scaffolding: preparation, detection, containment, eradication, recovery, and lessons learned. But each phase needs a different emphasis in a GenAI system.

  1. Preparation: log the right things before you need them. Not just final outputs. You need prompts, retrieved context, tool inputs and outputs, identity, configuration state, model version, and enough workflow trace to reconstruct who handed what to whom. If you are running multiple agents, you also need to capture the hops between them. Otherwise you will know the last bad act and miss the first bad input.
  2. Detection: "unexpected output" is not a clean indicator when variability is normal. You need to watch for output drift, unusual tool sequences, access patterns outside task scope, and agents invoking paths they do not usually touch. In a multi-agent system, you also need to notice when trust propagation starts looking odd. One agent suddenly becoming the source of too many sensitive downstream decisions should make you uncomfortable.
  3. Containment: full shutdown is sometimes right, but scope reduction is often the faster and safer first move. Disable a tool. Narrow data access. Break an agent-to-agent path. Force a human approval checkpoint. Move the system back to recommendation mode while you investigate. In agentic systems, the ability to degrade gracefully is part of containment.
  4. Eradication: removing a poisoned tool or fixing a prompt path only solves the forward problem. You still need to work out what the agent already touched, what outputs were acted on, and which downstream artefacts now contain bad data or bad decisions. Multi-agent systems make that worse because one wrong answer can become another agent's trusted input.
  5. Recovery: this is not just "redeploy and retest". You need confidence that the system's outputs are back inside expected bounds, that the dangerous interaction path is closed, and that the restored workflow is not quietly rebuilding the same failure through a different route.

What the first hour needs

If I were tightening an IR playbook for agentic systems, the first-hour checklist would be simple:

  • freeze relevant logs, prompts, retrieved context, tool I/O, and model configuration
  • identify whether the incident involves one agent or a chain
  • narrow scope before full shutdown if safe to do so
  • disable or isolate external tools and integrations that could be feeding the problem
  • identify which outputs or actions have already been consumed downstream by humans or other systems

That last point matters more than most teams realise. The incident is not over because the model stopped producing the bad output. It is over when you understand where that output travelled and what it influenced.

GenAI incident response is not a brand new discipline. It is traditional incident response forced to deal with systems that improvise, integrate widely, and sometimes pass each other bad judgement at machine speed.

If your playbook still assumes one system, one root cause, one reproducible failure, it was not built for this. Address that before the workflow grows more complex.

Related

Olivier Reuland