ConqrLLM 1.2 Benchmark Results
Three models on 10,000 EDGAR contract-QA items: the ConqrLLM 1.2, GPT-5.5, and Claude Opus 4.8. The two findings that carry the report: ConqrLLM 1.2 has the lowest fabrication rate on out-of-scope questions, and every model’s useful-answer rate is low in closed-book settings but rises under retrieval.
1. Calibration - Closed-Book
On unanswerable items, a well-calibrated model should refuse rather than invent an answer. This measures how often each model fabricates instead of refusing.
2. Useful answers - closed-book vs retrieval
Closed-book performance is uniformly poor across all three models. Giving each model the answer’s source document (oracle RAG) tells a different story.
3. Pre-registered claims - outcome
ConqrLLM 1.2 has a lower fabrication rate than GPT-5.5 on unanswerable items (0.4% vs 90.3%, CIs do not overlap.)
ConqrLLM 1.2 has a lower fabrication rate than Opus (CIs overlap: 0.0–11.0 vs 0.6–16.2.)
RAG improves useful-answer rate vs closed-book for all three models (Every model rises under oracle RAG vs closed-book alone.)
ConqrLLM 1.2's refusal rate on unanswerable items is not worse than frontier models (Appropriate refusal sits at 100%, ahead of both frontier models.)
ConqrLLM 1.2's over-refusal rate is not higher than frontier models (21.7% vs GPT-5.5's 1.4%, a real trade-off.)
4. What the numbers say
ConqrLLM 1.2’s standout result is calibration: on silent documents it almost never fabricated an answer (0.4%, 12/3100), whereas GPT-5.5 fabricated on nearly every one (2800/3100). Against Opus the calibration edge is not statistically established at this sample size; both are near-zero in fabrication.
Closed-book useful-answer rates are low across the board (≤1.4%). No model reliably extracts contract facts without the source document. Retrieval is required: under oracle RAG, useful answers rise for all three, with Claude Opus 4.8 highest (24.6%), GPT-5.5 lowest (4.3%).
The cost of ConqrLLM 1.2’s caution shows in over-refusal: it declines 21.7% of answerable items, more than GPT-5.5 (1.4%) though less than Opus (52.2%). The low fabrication and the over-refusal are two sides of the same disposition.
