Decision Matrix for Choosing an AI Email Tool (With Weights)

A weighted decision matrix decides the winner before you sit down to the demos. That is the point. You set what matters and how much it matters, then the scoring is mechanical — the vendor with the highest total wins because your criteria said so.

The failure mode is the mirror image: criteria set after the demos, weights nudged toward the tool the loudest person in the room already liked, and everything scored four out of five because nobody wants to defend a two in front of a vendor. This post is how to build a matrix that survives a decision meeting instead of getting quietly rewritten in it.

The short answer#

Four moves, in this order. Reversing any of them is how a matrix stops being decisive.

1. 1

   Split criteria into gates and scoring

   Gates are pass or fail. Miss one — no SSO, no audit trail, no BAA when you need one — and the vendor is out before the numbers start. Scoring criteria are weighted numerics for everything else.

2. 2

   Set weights before you score anything

   Weights sum to 100. Do this in a room, in writing, before the first demo. Get initials next to each number. Weights that move after a vendor has answered are advocacy in a spreadsheet.

3. 3

   Score on live data, not on slides

   Every vendor gets the same inbox sample, the same three-day trial window, and the same three real users. A score from a scripted demo is a rating for the demo, not the tool.

4. 4

   Publish the tie-break rule before the totals arrive

   Two vendors within five points of each other are a tie, and the tie-break is a named human — the person accountable for the outcome. Not a re-vote, not a re-weight, not another demo.

Criteria that actually matter#

Most matrices fail because they weight what is easy to score (feature counts, UI polish) over what actually decides whether the tool survives contact with your inbox. Five to seven scoring criteria is the sweet spot; more than eight and the weights become noise. The list below is what we would score an AI email tool on today. Steal or adapt.

• Authority and undo — does the vendor separate Manual, Copilot (approval before send) and Autopilot (bounded automatic action), with an undo window and an audit log an admin can read? An AI that only offers 'send' or 'don't send' has no middle setting for the mail you would normally handle in three seconds.
• Provider coverage — real support for Gmail, Microsoft 365 / Outlook, and IMAP under one account, not one provider plus a roadmap. Test the least-loved provider first; that is where the seams show.
• Triage accuracy on your inbox — false-positive rate matters more than false-negative. A tool that misses two newsletters a week is annoying. A tool that archives one paying customer's reply is a firing offence.
• Voice and drafting quality — measured on live threads with your own past correspondents. Watch for the tell that the model is guessing at 'polite email' rather than sounding like you.
• Data handling and posture — where drafts and read content flow, whether the vendor trains on user mail (they should not), envelope encryption on OAuth and BYOK keys, and the vendor's answer when you ask what happens on export or account deletion.
• Deployment shape — SSO / SCIM if you need them, admin controls that scope the agent to specific senders or labels, per-seat versus usage-metered pricing, and rollout mechanics that do not require a security exception.
• Escape hatch — what an export actually contains, whether rules, contexts and drafts come with you, and how much of the value is stuck inside the vendor if you leave in eighteen months.

The scoring table (template with weights)#

Weights below are a starting point for a mid-market team of ten to fifty seats. Move them to fit your situation — a solo founder should push voice and triage up; a regulated team should push data posture up and add a gate. The rule that does not move: weights sum to 100.

A worked example#

Two vendors, same brief, same three-day trial on the same inbox sample. Both cleared the gates (SSO available, audit log present, BAA on the paid plan). Below is what the totals look like when you actually multiply weight by score.

Two things to read out of that. First, Vendor B beat A on the single most visible dimension — triage accuracy — and still lost by 65 points, because A won three other categories that mattered nearly as much. That is what weights are for: they stop the loudest signal from deciding.

Second, the gap is well outside the five-point tie band, so the winner is A and the meeting is over. If the totals had come in at 430 and 428, you would not re-score. You would go to the named tie-breaker, who applies the pre-agreed rule — usually 'whichever tool the three trial users would keep using tomorrow' — and moves on.

Red flags that break a matrix#

Most matrices are undone by one of five moves, all of them done in good faith. Watch for them and name them out loud when they happen.

• Weights changed after scores are in. The moment a weight moves in response to a total, the matrix is a preference in disguise. Freeze weights before scoring; if a weight is genuinely wrong, throw the whole matrix out and restart.
• Everything scored a four. If no vendor gets a two or a five, the matrix has no discrimination. Force a spread — the highest and lowest on each criterion cannot be the same number.
• Demos scored as if they were trials. A demo shows what the vendor wants to show. Score only on the live-inbox trial; leave demo notes as qualitative context.
• Gates smuggled in as scoring criteria. If SSO is required and a vendor lacks it, that vendor is out, not scored a two on 'deployment'. Otherwise a strong showing elsewhere will paper over a hard blocker.
• One person scored everything. The three trial users score independently, then average. A single reviewer scoring both vendors is auditioning their own taste, not the tools.

Anchoring the criteria to something citable#

For the data-posture and admin rows, ground the scoring in a published framework rather than in your own taste. The NIST AI Risk Management Framework (AI RMF 1.0) gives you a shared vocabulary for measurable AI risk (Govern, Map, Measure, Manage). The Cloud Security Alliance's Cloud Controls Matrix and CAIQ give you a standard set of vendor security questions and a response format buyers already recognise.

You do not need to score against every control. Pick the six or eight that matter for AI email — data handling, key management, logging, incident notification, sub-processor list — and require the vendor to point to a documented answer. 'We take security seriously' scores a one. A CAIQ response and a link to a policy scores higher.

What we'd pick and why (honest)#

Disclosure: we build AI Emaily, so treat this section as advocacy and check us against the same trial the other vendors get. The reason we score well on the weights we recommend is that we built the product against those weights — approve-before-send Copilot, a bounded Autopilot with undo and an audit log, Gmail / Microsoft 365 / IMAP under one account, envelope-encrypted OAuth and BYOK, no training on user mail, and an export that includes rules and context rather than just a mailbox dump. Packaging is a 7-day free trial on Pro/Autopilot (card required, no charge if cancelled before day seven), not a permanent free tier.

Where we lose, and it is worth saying by name: if your whole team lives inside Gmail and the decisive weight is the fastest keyboard-driven search across a large archive, Shortwave has built harder on that than we have. If your triage need is entirely bulk-newsletter clean-up rather than agentic action, SaneBox does that job well and is cheaper. If you want a native Mac binary with a fully local archive, Mimestream is the honest recommendation — our Mac app is a real downloadable app, but it is an Electron shell that shares the web codebase.

Everywhere else — mixed providers, a team that wants approval-first automation, a data posture that will survive a security review — we are the recommendation, and the weighted matrix above is the argument for it rather than a slogan.