The first experiment gave us a clean result: when payroll rules are simple, spelling things out helps an off-the-shelf model perform well.
Then we made the agreements more complicated.
Not to be annoying. To reflect the reality of payroll agreements for shift work.
The clean result did not survive.
What changed in the second experiment
We ran Qwen2.5-1.5B-Instruct in GGUF format across a wide range of quantization levels, from highly compressed Q2 to near-full-precision FP16, on 2,500 payroll clauses with varying levels of complexity.
In this experiment, complexity means the number of different class types worked at different rates. The agreements varied structurally from:
- abbreviated text (S1)
- semi-structured language (S2)
- full, grammatically correct sentences (S3)
Complexity drove extraction accuracy
The F1 scores in the heatmap show a clear pattern: pay agreement complexity is the primary driver of extraction accuracy.
The more class types and rate structures a worker has, the harder the model has to work.
For simple agreements, such as a salaried worker or a single flat rate, semi-structured text is enough. Full prose adds no meaningful value and can even marginally reduce accuracy.
For complex agreements, especially the shift-based arrangements common in hospitality, fitness, and care work, full grammatical sentences perform much better than structured shorthand.
In the most complex cases, full sentences outperform abbreviated or semi-structured formats by nearly 21 percentage points. That is where off-the-shelf models struggle most, and where a fine-tuned model has the clearest commercial advantage.
Does a bigger model solve the problem?
We are building toward a fine-tuned model for payroll rule extraction.
Before we get there, we need to understand two things:
- the performance ceiling of an off-the-shelf model
- the best accuracy-to-compute tradeoff for a production environment
So we ran the same experiment across four quantization levels of the same base model: 2-bit, 4-bit, 8-bit, and 16-bit.
The 8-bit model is the current production tradeoff
The results were clear:
- The 2-bit model is not viable.
- The 4-bit model underperforms on complex structures, which makes it inadequate for shift work.
- The 8-bit and 16-bit models are statistically indistinguishable at every complexity level.
That makes 8-bit the best tradeoff we have measured so far. It gives us the practical accuracy of the 16-bit model without carrying the same compute cost.
But this experiment only tested one model family.
What we test next
The next step is running the same experiment across a selection of model architectures to see whether different base models yield different results.
They likely do.
That comparison will inform which model we fine-tune for payroll rule extraction, and where the biggest lift is likely to come from: cleaner input language, model architecture, quantization strategy, or domain-specific training.
For payroll automation, the important lesson is already visible: real-world complexity changes the problem. Simple examples make language models look clean. Shift-work payroll shows where they need help.
If you missed Payroll AI Experiment #1, you can start with Write for the Machine. It lays the groundwork for this second experiment by showing how the way payroll agreements are written can directly affect model performance.
