Incident Response When AI Sends the Wrong Email

What to do when AI sends the wrong email splits into two problems that share nothing but the timeline. The next hour is technical: contain the agent, work out who got what, decide if any of it can still be pulled back. The days after are procedural: notify the people affected, decide if it is a reportable breach, and fix the setting that let it happen.

This runbook assumes the sender was an AI agent with some level of autonomy — a Copilot draft that got auto-approved, an Autopilot reply that fired without a review, or a scheduled send that used a template with the wrong merge field. Handle a genuine human mistake the same way from step two onward; step one changes.

We build AI Emaily, an AI email client with Manual, Copilot and Autopilot modes plus a logged, reversible action trail — so the containment step below is our territory. The recall step is not, and we say so.

The short answer#

Do these five things, in this order, before you draft the apology. The order matters because each step preserves the evidence and options for the next one.

First, flip the agent to Manual on the account that sent the message, and on every other account it manages. A wrong send at 09:14 followed by an autonomous follow-up at 09:22 is a bigger incident than the first send alone, and takes twice as long to explain.

Second, pull the audit log for a wide window around the send — an hour before, an hour after — and export it. Save the export. This is your one clean picture of who received what, when, and which rule or reasoning produced it.

Third, attempt recall where it is realistic. Native Gmail Undo Send catches the message before it leaves your outbound server if you act inside the delay you configured. Microsoft 365 Message Recall can pull a message from another M365 mailbox under narrow preconditions. Beyond those, treat the mail as delivered and move to notification.

Fourth, notify the affected recipients directly and, if it involves personal data, notify your privacy owner. Fifth, decide inside 72 hours whether the incident meets your regulator's breach threshold — the GDPR clock in particular is fast and starts when you know, not when you finish investigating.

Criteria that actually matter#

Most incident-response templates treat an AI mistake as a single category. It isn't. The response splits along four axes, and getting the axis wrong is how a fixable mistake turns into a reportable one.

• Where the message is now. Held in your outbound server (recall works), delivered to a mailbox inside your tenant (Exchange recall may work), or delivered to an external mailbox (it is gone, and only the recipient can delete it).
• What is in it. A tone problem is embarrassment. A wrong attachment or a third party's personal data is a breach question, and the definitions in GDPR Article 33 and the HHS Breach Notification Rule both catch a misdirected email that exposes protected information.
• Who received it. The wrong person entirely, the right person with wrong content, or a group that mixes both. Each needs a different corrective message, and a bulk apology to the group is often worse than five individual notes.
• Which autonomy mode was on. Manual is a human mistake — the AI drafted, a person clicked send. Copilot with auto-approval enabled is a policy failure — the human gate existed and was disabled. Autopilot is a system failure — the rule fired inside its scope, and the scope was wrong.

The first-hour runbook#

Run the table top to bottom. The realistic-outcome column is what actually happens in a normal mail stack; the exceptions are named where they exist. Do not skip a row because a later one looks more decisive — each one preserves the evidence the next one uses.

Worked example: an Autopilot reply on the wrong thread#

The most common shape of this incident, in the tickets we see: an Autopilot rule replies to a contract negotiation email with a paragraph intended for a different client, because the two threads share a subject line and a shared sender domain. Walk through the runbook against that scenario.

1. 1

   T+0: The mistake

   Autopilot fires at 09:14 with confidence above the account's threshold. The reply names Client A's discount and quotes the wrong renewal date. It goes to Client B. The rule was scoped to the sender domain, not the specific thread, and the domain is shared.

2. 2

   T+45 seconds: Contain

   You notice the Sent-folder entry. Undo Send is already past. Open the AI agent's settings and switch the sending account to Manual. Do the same for the other three accounts on the same rule scope. Confirm the rule is paused, not just the mode.

3. 3

   T+3 minutes: Establish scope

   Pull the audit log for 09:00–09:20. Confirm one recipient, one send, no auto-forwarded follow-up, and that the rule cited Client A's context as the source. Export the log. Note the confidence score and the reasoning field — you will need both.

4. 4

   T+8 minutes: Attempt recall (and stop)

   Client B is on their own Google Workspace tenant, not yours. There is no server-side recall path. Do not try to send a follow-up that quietly overwrites the first; it will not, and it makes the audit trail worse.

5. 5

   T+20 minutes: Notify the recipient

   One short message to Client B from a human. Name the error plainly, name what was wrong in it, name what the correct information is. Do not blame the tool. Offer a call if the wrong information is commercially material.

6. 6

   T+2 hours: Notify internally

   Tell the account owner for Client A that their pricing was disclosed to Client B, and tell your privacy owner. If Client A is under a confidentiality clause with you, this is a contract question as well as a policy one.

7. 7

   T+24 hours: Breach assessment

   Personal data of a third party was disclosed to the wrong recipient. Under GDPR that meets the definition of a personal data breach. Assess likelihood of risk to Client A's individuals; if it is more than negligible, notify the supervisory authority inside 72 hours from the moment you became aware at 09:14.

8. 8

   T+1 week: Post-incident review

   The rule scope was the bug, not the model. Move that rule from sender-domain to a stricter matcher — thread ID, or a client label the Context Brain owns. Lower the Autopilot confidence threshold on that account, or move Client A's category to Copilot until you trust the new scope.

Between recall and notification: the log is your evidence#

The gap between the second and the fourth row of the runbook is where the incident is either resolved cleanly or reopened in a month by someone who was not there. Whatever you did in the first hour, the audit log is what proves it.

A useful log — the one you need for both the notification and the review — names the actor (which agent run, or which user), the target (message ID, recipient), the change (sent, replied, archived, unsubscribed), the reason (the rule that matched, or the reasoning behind an autonomous action), and the timestamp. If any of those columns are missing, you will guess in the meeting a week from now.

Red flags that turn one wrong email into two incidents#

These are the moves that look sensible in the moment and produce a worse outcome. Every one of them we have seen after the fact.

• Sending a follow-up that pretends the first one didn't happen. It arrives out of order in the recipient's client, both messages sit in their inbox, and the correction looks evasive rather than prompt.
• Using Outlook Message Recall on an external address. It cannot work — the mechanism is intra-tenant — and the failed recall notice tells the recipient a message they might have missed exists. Now they read it.
• Deleting the offending message from your own Sent folder to hide it. It stays in the recipient's mailbox, it stays in provider server logs, and it makes the internal review look like a cover-up.
• Editing or clearing the audit log while the incident is live. Some tools let you clear an agent run's history; do not. If the log format lets you annotate rather than delete, use that.
• Switching back to Autopilot the same day, on the same rule, because you diagnosed the mistake as a one-off. Diagnose the scope, not the run. The rule that fired on the wrong thread will fire again on the next wrong thread until the scope changes.
• Announcing the incident to a wider audience than the affected group. A company-wide note about a two-recipient send creates a bigger disclosure than the incident and often trips notification thresholds you had not otherwise met.
• Adding an AI-generated apology. If you would not have sent the first message without a human read, do not send the second one without one either.

Is a misdirected email a data breach?#

Often, yes. Both the main frameworks that matter here — GDPR in the EU and the HHS Breach Notification Rule under HIPAA in the US — treat unauthorised disclosure of personal or protected information as a breach, and neither of them carves out mistakes made by an autonomous system. The AI agent doing the sending does not lower the threshold.

GDPR Article 33 requires notification of the supervisory authority within 72 hours of becoming aware, unless the breach is unlikely to result in a risk to individuals. That is a real exemption — a misdirected message with no personal data, or a message that only reached another employee inside your organisation, may not require notification — but the assessment has to be documented either way. Absence of a report is not absence of an obligation.

The HHS rule, for covered entities under HIPAA, treats an impermissible disclosure of protected health information as presumed a breach unless a four-factor risk assessment concludes low probability of compromise. Send a wrong reply to a patient thread that includes another patient's information and you are inside that presumption.

This is where the audit log earns its keep. The assessment needs to say what data was disclosed, to whom, how many individuals, what mitigation you applied, and how quickly. The log is where those numbers come from — and if the log doesn't have them, an incident that might have been unreportable becomes reportable simply because you cannot prove it wasn't.

What we'd pick and why (honest)#

We build AI Emaily. What you saw above — Manual, Copilot, Autopilot per account, a logged audit trail with the reasoning behind each run, undo on destructive local actions, an Undo Send delay for outbound — is deliberately shaped for this incident. The containment step and the evidence step are the ones we take responsibility for, and both are one click and one export respectively. Autonomy is a per-account setting, not a global one, so containing a runaway rule on one mailbox does not disable the agent everywhere.

The honest concession is on the third step. Undo is not retroactive on any mail client we know of, ours included. Our Undo Send holds the mail in the seconds window you configure before it leaves the outbound server; once it is on a recipient's server, we cannot pull it. Microsoft 365's Message Recall is the one server-side path that exists in mainstream mail, and it is intra-tenant only. If your incident-response requirement is a recall mechanism that actually removes delivered mail from external mailboxes, that mechanism does not exist — buy the process that reduces incidents, not the tool that promises to undo them.

The other honest concession is legal-hold and e-discovery. Google Vault and Microsoft Purview are built for that, and every third-party AI email client — us included — sits on top of, not beside, those systems. If your organisation's evidence workflow runs through Vault or Purview, use them as the source of truth and treat the AI client's log as a supplement.

AI Emaily is right for a team or founder who wants to run mail through an agent, wants the containment and the evidence trail to be one click each, and can live with the same recall constraints that apply to every other mail client. See how Copilot and Autopilot are separated in our features page at /features/copilot-autopilot, and pricing (7-day free trial on Pro and Autopilot plans, card required, no permanent free tier) at /pricing. If your evaluation criterion is enterprise-grade Message Recall or a native e-discovery hold, keep looking — an add-on on top of your Microsoft 365 or Google Workspace tenant will beat any third-party client on those two dimensions.

Post-incident review: the questions that stop it recurring#

Run this at T+7 days, not T+2. Some of the answers show up only after the follow-up mail has arrived and the recipient has responded. Two people in the room minimum, one of whom did not touch the incident.

• Which mode was on, and was that the intended mode for that account? A Copilot account with auto-approval enabled is often the real bug, not the send.
• What triggered the rule? Read the reasoning column in the log; the model's own account of why it acted is the fastest way to see whether the scope was wrong or the input was wrong.
• What would have caught this before send? A confidence threshold change, a stricter rule scope, a Context Brain profile update, a client label — write down which one and set a date.
• Did the audit log contain what you needed? If any of actor, target, change, reason, timestamp was missing, that is the first fix, not the last.
• How long did each step actually take? Time-to-contain and time-to-notify are the two numbers to trend. If either grew since the last incident, the runbook is slipping.
• Did any regulator or customer notification obligation get triggered, and did it happen inside the required window? Write the answer down even when it is no.