Dicts and Docs: The Value of Grammar and Documentation for LLM-Based Automation

When an LLM's output is underwhelming, the instinct is often to pay up for a bigger, newer model.

But before training, fine-tuning, or model shopping, there are two cheaper levers worth testing:

• Documentation: definitions that help the model understand what each field means.
• Grammar: output constraints that prevent the model from producing invalid structures or values.

In this experiment, we tested what those two levers buy you before any training.

Documentation changes how the model understands the task

Documentation operates on the model's understanding of the task.

At inference time, it injects domain semantics that pre-training is unlikely to include because they belong to a niche payroll context.

In this experiment, documentation means API-style field definitions. We built those definitions by unifying documentation patterns from leading HR and payroll companies, then using them to explain the fields the model needed to extract.

Instead of asking the model to infer what a payroll field means from its name alone, we gave it a definition.

Grammar changes what the model is allowed to output

Grammar operates on the model's output.

We compiled our 20-field schema into a formal grammar that constrains decoding. At each generation step, the sampler can only choose tokens the grammar allows.

That means the model cannot emit malformed JSON or choose a value outside what the schema permits.

Grammar does not make the model smarter. It makes the output obey the contract.

The experiment:

We ran the experiment on 264 payroll agreements spanning formal contracts through informal email exchanges and meeting transcripts, with offer letters, job postings, and staff handbooks in between.

Each document had an associated hand-verified gold answer.

The setup:

• Models: Qwen2.5-Instruct, run on llama.cpp
• Model sizes: 3B and 14B
• Quantization: Q8_0, quantized locally using llama.cpp's convert_hf_to_gguf from the published FP16 weights

We ran each model four times, once for each combination of the two factors we were testing:

• field names alone vs. full definitions
• free decoding vs. grammar-constrained decoding

That produced 2,112 extractions in total.

Here´s what we found:

Documentation substitutes for grammar.

With definitions in place, the 14B model is accurate across 92% of fields and 31% of records.

Grammar does not substitute for documentation.

Grammar improves output validity, but it cannot teach a convention. It can enforce the shape of an answer, but it cannot explain what the answer should mean.

Grammar without documentation can hurt.

When a contract says nothing about a policy, the right answer may be null. But grammar can force the model to choose a valid value anyway.

In this experiment, grammar without definitions forced answers on questions the document never addressed. The model picked false 50 times where the correct answer was null.

Definitions removed that error entirely.

Field accuracy is not the same as record accuracy

A 92% field accuracy score can make the system look nearly solved.

But a record is only correct when all twenty fields are correct.

On that stricter standard, less than a third of records survived.

The culprits have a name: bottlenecks.

We will discuss those in the next post.

The practical lesson

Before you fine-tune a model, test whether the model has enough context to understand the fields and enough structure to produce valid output.

Documentation tells the model what the field means.

Grammar tells the model what the answer is allowed to look like.

Both matter. But if the model does not understand the convention, constraining its output will not fix the underlying problem.

For payroll automation, better definitions may buy you more than a bigger model.