Evaluation Results

gpt-5-mini handles basic questions well — but it's too slow. 99% recall and 86% calc accuracy on basic, single-rule questions is strong, but the ~20k token RAG context injected per query pushes response times to an unacceptable range for a chat interface.

gpt-4.1-mini is faster and solid on recall, but falls apart on calc. Recall accuracy is competitive, and latency is much more acceptable. The problem is calculation: at roughly 50% calc accuracy, it frequently gets the arithmetic wrong even when it retrieves the right rules — making it unreliable for questions that require any computation.

Both models are mini-class, chosen for cost. These results cover basic questions only — harder scenarios with edge-case interactions or chained reasoning are not yet in this eval set. All results confirmed by human review.