Jebra AI Experiments #5
In our previous blog, we described how we successfully fine-tuned Qwen2.5-1.5B-Instruct using QLoRA to extract structured JSON from unstructured text for a domain-specific use case. The results were promising, with the model achieving 62% exact match accuracy and 97% field match accuracy.
However, a natural question arose: How does our fine-tuned small model compare against state-of-the-art frontier models like Claude Opus 5? Would a massive general-purpose model outperform our domain-specific fine-tuned model?
We decided to run the same evaluation pipeline on Claude Opus 5 to find out.
Evaluation Setup
We used the identical test set and the same evaluation metrics to ensure a fair comparison.
Exact Match: A binary score per sample, 1 if every field matches exactly and 0 otherwise, aggregated across the test dataset.
Field Match: Partial credit per sample based on the number of correct fields divided by total fields, aggregated across the test dataset.
Results
Despite being roughly 1,000 times smaller than Claude Opus 5, our fine-tuned Qwen2.5-1.5B achieved approximately 15 percentage points higher field-match accuracy: 96.72% versus 81.81%.
It also achieved 62% exact-match accuracy, compared with Claude Opus 5 at 0%. Beyond accuracy, the smaller fine-tuned model is much less expensive and provides much more control over your data.
Conclusion
This comparison demonstrates a powerful lesson in modern AI development: a well-tuned small model can outperform even the largest frontier models on domain-specific use cases.
