Conqr Mail Guard Benchmark Report
Conqr Mail Guard checks outgoing email and attachments on the device before Send in Outlook. We tested it across five suites, including anonymised deal documents and 800 public legal documents.
- Tests passed
- 91.1%
- 6,491 of 7,123 tests
- Real-document accuracy
- 90.3%
- 5,797 of 6,417 tests
- External pipeline requests
- 0
- Observed in the tested browser runs
- Failure conditions handled safely
- 23 / 23
- 12 separate Send gaps documented
Test outcomes
6,491 of 7,123 tests passed. The two real-document suites account for 6,417 tests. Across the browser runs, the checking pipeline made zero external requests.
| Suite | Tests | Passed | Pass rate |
|---|---|---|---|
| Anonymised deal documents | 771 | 665 | 86.3% |
| Core legal scenarios | 300 | 300 | 100% |
| Odd and broken files | 364 | 364 | 100% |
| Send button safety | 42 | 30 | 71% |
| Public legal documents | 5,646 | 5,132 | 90.9% |
| All tests | 7,123 | 6,491 | 91.1% |
Where the 632 failures came from
False alarms accounted for 72.3% of failures. Of the 156 misses, 88 involved non-English files, 22 had too little readable text, and 46 involved English files that were close in topic.
| Outcome | Tests | Share of failures | Share of all tests |
|---|---|---|---|
| False alarmsA legitimate email triggered a warning. | 457 | 72.3% | 6.4% |
| MissesA wrong or unrelated attachment went through without a warning. | 156 | 24.7% | 2.2% |
| Wrong file flaggedThe warning identified the wrong attachment in the email. | 7 | 1.1% | 0.1% |
| Known Send gapsSome compressed, unsupported, unreadable or empty-file paths did not alert as expected. | 12 | 1.9% | 0.2% |
| All failures | 632 | 100% | 8.9% |
Stress test on 800 public legal documents
The November 10 run used 100 PDFs from each of eight sources. Mail Guard scored 5,132 of 5,646 (90.9%), correctly flagged 1,487 of 1,646 cases that should warn (90.3%), and left 3,645 of 4,000 valid emails alone (91.1%). It read 797 of 800 files and recorded zero crashes.
| Scenario | Expected | Tests | Correct | False alarms | Misses | Wrong file |
|---|---|---|---|---|---|---|
| Forward to a colleague | Pass | 800 | 800 | 0 | 0 | 0 |
| Wrong client | Flag | 46 | 46 | 0 | 0 | 0 |
| Right file, subject only | Pass | 800 | 737 | 63 | 0 | 0 |
| Right file, short email | Pass | 800 | 736 | 64 | 0 | 0 |
| Unrelated file attached | Flag | 800 | 721 | 0 | 79 | 0 |
| One wrong file among several | Flag | 800 | 720 | 0 | 73 | 7 |
| Personal email address | Pass | 800 | 688 | 112 | 0 | 0 |
| Right file, detailed email | Pass | 800 | 684 | 116 | 0 | 0 |
| All scenarios | — | 5,646 | 5,132 | 355 | 152 | 7 |
| Source | Tests | Accuracy | Mistakes caught | False alarms |
|---|---|---|---|---|
| UK legislation | 701 | 98.1% | 199/201 | 11 |
| Indian gazettes | 700 | 95.6% | 188/200 | 19 |
| India Code acts | 702 | 92.5% | 174/202 | 25 |
| Indian High Court judgments | 702 | 90.7% | 178/202 | 41 |
| UK Supreme Court documents | 730 | 88.9% | 216/230 | 67 |
| Indian Supreme Court judgments | 709 | 88.2% | 174/209 | 49 |
| US federal publications | 701 | 87.7% | 181/201 | 66 |
| US Supreme Court opinions | 701 | 85.6% | 177/201 | 77 |
Each synthetic email drew its description from the source document, making some examples easier to match than a lawyer’s own wording. The stress test used public judgments and legislation. The 771-test deal-document suite measures a separate set of scenarios.
How Mail Guard checks an email
Mail Guard is an add-in for Outlook on the web and Outlook for Mac. Outlook supplies the draft to the add-in, which runs fast rules and small local models on the device. Conqr hosts the add-in and model downloads. The checks do not send email or attachment content to Conqr.
- 1
Envelope
Check addresses and domains for typos, lookalikes and risky destinations.
- 2
Thread context
Compare recipients with people and companies already on the thread.
- 3
Message scan
Look for privilege markers, identifiers, bank details and credentials.
- 4
Attachment inspection
Extract text locally, use OCR for scans and inspect files for hidden sheets, comments and tracked changes.
- 5
Attachment matching
Compare topic, parties, document type, confidentiality and matter references against the message and recipients.
- 6
Review before Send
Show the evidence and available fixes. The sender can correct the issue or choose Send anyway.
Attachment matching combines the signals into a risk score. At 70% or above it creates a finding. The score is hand-tuned and should be read as a risk indicator. Where Outlook supports it, senders can remove a file, change an address or move recipients to Bcc from the review card.
Check speed and Send safety
| Check | Environment | Typical | 95% within |
|---|---|---|---|
| First check of a new file | November 10 run | 2.1 s | 3.9 s |
| Repeat check of the same file | Earlier automated run | 0.02 s | 0.04 s |
| First check of a 100 KB Word document | Earlier Outlook web check | 4.3 s | — |
| Alert after clicking Send | Earlier Outlook web check | About 2 s | — |
| Scanned 4 MB PDF | Earlier Outlook web check | About 13 s | — |
| 30 attachments at once | Earlier automated run | 14.8 s | — |
The 23 failure conditions tracked here covered engine, Outlook, network and file failures. All 23 were handled safely. The broader Send safety suite passed 30 of 42 tests and recorded 12 known gaps.
Attachment matching supports English. Some unsupported attachments, including OneDrive links, attached emails, ZIP files and legacy .doc files, are marked as unchecked in the pane without a Send alert. Full live-Outlook and low-memory hardware verification for this version is pending.
