Evaluation · November 10, 2026

Conqr Mail Guard Benchmark Report

Conqr Mail Guard checks outgoing email and attachments on the device before Send in Outlook. We tested it across five suites, including anonymised deal documents and 800 public legal documents.

Tests passed
91.1%
6,491 of 7,123 tests
Real-document accuracy
90.3%
5,797 of 6,417 tests
External pipeline requests
0
Observed in the tested browser runs
Failure conditions handled safely
23 / 23
12 separate Send gaps documented

Test outcomes

6,491 of 7,123 tests passed. The two real-document suites account for 6,417 tests. Across the browser runs, the checking pipeline made zero external requests.

All 7,123 tests by outcome
Pass rate by suite
Results across all five suites
SuiteTestsPassedPass rate
Anonymised deal documents77166586.3%
Core legal scenarios300300100%
Odd and broken files364364100%
Send button safety423071%
Public legal documents5,6465,13290.9%
All tests7,1236,49191.1%

Where the 632 failures came from

False alarms accounted for 72.3% of failures. Of the 156 misses, 88 involved non-English files, 22 had too little readable text, and 46 involved English files that were close in topic.

Failed tests by outcome
Failure counts and shares
OutcomeTestsShare of failuresShare of all tests
False alarmsA legitimate email triggered a warning.45772.3%6.4%
MissesA wrong or unrelated attachment went through without a warning.15624.7%2.2%
Wrong file flaggedThe warning identified the wrong attachment in the email.71.1%0.1%
Known Send gapsSome compressed, unsupported, unreadable or empty-file paths did not alert as expected.121.9%0.2%
All failures632100%8.9%

Stress test on 800 public legal documents

The November 10 run used 100 PDFs from each of eight sources. Mail Guard scored 5,132 of 5,646 (90.9%), correctly flagged 1,487 of 1,646 cases that should warn (90.3%), and left 3,645 of 4,000 valid emails alone (91.1%). It read 797 of 800 files and recorded zero crashes.

Stress test results by email scenario
ScenarioExpectedTestsCorrectFalse alarmsMissesWrong file
Forward to a colleaguePass800800000
Wrong clientFlag4646000
Right file, subject onlyPass8007376300
Right file, short emailPass8007366400
Unrelated file attachedFlag8007210790
One wrong file among severalFlag8007200737
Personal email addressPass80068811200
Right file, detailed emailPass80068411600
All scenarios—5,6465,1323551527
Accuracy by public document source
Stress test results by source
SourceTestsAccuracyMistakes caughtFalse alarms
UK legislation70198.1%199/20111
Indian gazettes70095.6%188/20019
India Code acts70292.5%174/20225
Indian High Court judgments70290.7%178/20241
UK Supreme Court documents73088.9%216/23067
Indian Supreme Court judgments70988.2%174/20949
US federal publications70187.7%181/20166
US Supreme Court opinions70185.6%177/20177

Each synthetic email drew its description from the source document, making some examples easier to match than a lawyer’s own wording. The stress test used public judgments and legislation. The 771-test deal-document suite measures a separate set of scenarios.

How Mail Guard checks an email

Mail Guard is an add-in for Outlook on the web and Outlook for Mac. Outlook supplies the draft to the add-in, which runs fast rules and small local models on the device. Conqr hosts the add-in and model downloads. The checks do not send email or attachment content to Conqr.

  1. 1

    Envelope

    Check addresses and domains for typos, lookalikes and risky destinations.

  2. 2

    Thread context

    Compare recipients with people and companies already on the thread.

  3. 3

    Message scan

    Look for privilege markers, identifiers, bank details and credentials.

  4. 4

    Attachment inspection

    Extract text locally, use OCR for scans and inspect files for hidden sheets, comments and tracked changes.

  5. 5

    Attachment matching

    Compare topic, parties, document type, confidentiality and matter references against the message and recipients.

  6. 6

    Review before Send

    Show the evidence and available fixes. The sender can correct the issue or choose Send anyway.

Attachment matching combines the signals into a risk score. At 70% or above it creates a finding. The score is hand-tuned and should be read as a risk indicator. Where Outlook supports it, senders can remove a file, change an address or move recipients to Bcc from the review card.

Check speed and Send safety

Measured attachment-check timings
CheckEnvironmentTypical95% within
First check of a new fileNovember 10 run2.1 s3.9 s
Repeat check of the same fileEarlier automated run0.02 s0.04 s
First check of a 100 KB Word documentEarlier Outlook web check4.3 s—
Alert after clicking SendEarlier Outlook web checkAbout 2 s—
Scanned 4 MB PDFEarlier Outlook web checkAbout 13 s—
30 attachments at onceEarlier automated run14.8 s—

The 23 failure conditions tracked here covered engine, Outlook, network and file failures. All 23 were handled safely. The broader Send safety suite passed 30 of 42 tests and recorded 12 known gaps.

Attachment matching supports English. Some unsupported attachments, including OneDrive links, attached emails, ZIP files and legacy .doc files, are marked as unchecked in the pane without a Send alert. Full live-Outlook and low-memory hardware verification for this version is pending.

Book a call

One Platform. Every AI. Built for Law Firms.

Draft faster, review smarter, search your firm's knowledge instantly - one secure platform, built for legal.

Book a demoSee it live
Conqr summarizing an agreement beside the clause it cites