The first experiment gave us a clean result: when payroll rules are simple, spelling things out helps an off-the-shelf model perform well.
Then we made the agreements more complicated.
Not to be annoying. To reflect the reality of payroll agreements for shift work.
The clean result did not survive.
We ran Qwen2.5-1.5B-Instruct in GGUF format across a wide range of quantization levels, from highly compressed Q2 to near-full-precision FP16, on 2,500 payroll clauses with varying levels of complexity.
In this experiment, complexity means the number of different class types worked at different rates. The agreements varied structurally from:
The F1 scores in the heatmap show a clear pattern: pay agreement complexity is the primary driver of extraction accuracy.
The more class types and rate structures a worker has, the harder the model has to work.
For simple agreements, such as a salaried worker or a single flat rate, semi-structured text is enough. Full prose adds no meaningful value and can even marginally reduce accuracy.
For complex agreements, especially the shift-based arrangements common in hospitality, fitness, and care work, full grammatical sentences perform much better than structured shorthand.
In the most complex cases, full sentences outperform abbreviated or semi-structured formats by nearly 21 percentage points. That is where off-the-shelf models struggle most, and where a fine-tuned model has the clearest commercial advantage.
We are building toward a fine-tuned model for payroll rule extraction.
Before we get there, we need to understand two things:
So we ran the same experiment across four quantization levels of the same base model: 2-bit, 4-bit, 8-bit, and 16-bit.
The results were clear:
That makes 8-bit the best tradeoff we have measured so far. It gives us the practical accuracy of the 16-bit model without carrying the same compute cost.
But this experiment only tested one model family.
The next step is running the same experiment across a selection of model architectures to see whether different base models yield different results.
They likely do.
That comparison will inform which model we fine-tune for payroll rule extraction, and where the biggest lift is likely to come from: cleaner input language, model architecture, quantization strategy, or domain-specific training.
For payroll automation, the important lesson is already visible: real-world complexity changes the problem. Simple examples make language models look clean. Shift-work payroll shows where they need help.
If you missed Payroll AI Experiment #1, you can start with Write for the Machine. It lays the groundwork for this second experiment by showing how the way payroll agreements are written can directly affect model performance.