Qwen3.5-9B on document benchmarks: where it beats frontier models and where it doesn't.

Posted by shhdwi@reddit | LocalLLaMA | View on Reddit | 35 comments

We run an open document AI benchmark. 20 models, 9,000+ real documents. Just added all four Qwen3.5 sizes (0.8B to 9B). Now we have per-task breakdowns for every model.

You can see the results here : idp-leaderboard.org

Where all Qwen wins or matches:

OlmOCR (text extraction from messy scans, dense PDFs, multi-column layouts):

Qwen3.5-9B: 78.1
Qwen3.5-4B: 77.2
Gemini 3.1 Pro: 74.6
Claude Sonnet 4.6: 74.4
Qwen3.5-2B: 73.7
GPT-5.4: 73.4

9B and 4B are ahead of every frontier model on raw text extraction. The 2B matches GPT-5.4.

VQA (answering questions about document content, charts, tables):

Gemini 3.1 Pro: 85.0
Qwen3.5-9B: 79.5
GPT-5.4: 78.2
Qwen3.5-4B: 72.4
Claude Sonnet 4.6: 65.2
GPT-5.2: 63.5
Gemini 3 Flash: 63.5

This one surprised us the most. The 9B is second only to Gemini 3.1 Pro on VQA. It edges past GPT-5.4. It is 14 points ahead of Claude Sonnet and 16 points ahead of Gemini Flash. For a 9B open model, that VQA score is hard to explain.

KIE (extracting invoice numbers, dates, amounts):

Gemini 3 Flash: 91.1
Claude Opus 4.6: 89.8
Claude Sonnet 4.6: 89.5
GPT-5.2: 87.5
Gemini 3.1 Pro: 86.8
Qwen3.5-9B: 86.5
Qwen3.5-4B: 86.0
GPT-5.4: 85.7

Qwen-9B matches Gemini 3.1 Pro. Qwen-4B matches GPT-5.4. Both ahead of GPT-5-Mini (85.7), Claude Haiku (85.6), and Ministral-8B (85.7). A 4B model doing production-grade field extraction.

Where frontier models are clearly better.

Table extraction (GrITS):

Gemini 3.1 Pro: 96.4
Claude Sonnet: 96.3
GPT-5.4: 94.8
Gemini 3 Pro: 95.8
GPT-5.2: 86.0
Gemini 3 Flash: 85.6
Qwen3.5-4B: 76.7
Qwen3.5-9B: 76.6

Frontier models are 85 to 96 on tables. Qwen is stuck at 76 to 77 regardless of size. The 4B and 9B are essentially identical. This looks like an architecture limit, not a scale limit.

Handwriting OCR:

Gemini 3.1 Pro: 82.8
Gemini 3 Flash: 81.7
GPT-4.1: 75.6
Claude Opus: 74.0
Claude Sonnet: 73.7
GPT-5.4: 69.1
Ministral-8B: 67.8
Qwen3.5-9B: 65.5
Qwen3.5-4B: 64.7

Gemini dominates handwriting. Qwen is behind but not drastically behind GPT-5.4 (69.1 vs 65.5).

Scaling within the Qwen family:

Overall: 0.8B 58.0, 2B 63.2, 4B 73.1, 9B 77.0

Summary:

OCR extraction: Qwen 4B/9B ahead of all frontier models
VQA reasoning: Qwen-9B is #2 behind only Gemini 3.1 Pro. Beats GPT-5.4.
KIE field extraction: Qwen 4B/9B match frontier models
Table extraction: Frontier models lead by 10 to 20 points

Every prediction is visible. Compare Qwen outputs against any model on the same documents.

idp-leaderboard.org/explore

[-]

Blackhawk1282@reddit

I see all these benchmark values and people talking about how great these models are but still have yet to see them perform well in real world use cases. I have about 4000 pages of D&D 5e manuals, I have tried all the OCR tools, the new qwen3.5 models, and still have yet to get usable output by asking basic questions. It seems to me these benchmarks are intentionally built to get the models to score as close to 100 as possible.

stereo16@reddit

Maybe try separating the steps? Have the ocr model give you a markdown version of a manual, then query that with a regular model.

shhdwi@reddit (OP)

Yes, I am also figuring out ways to test on more real world docs, do you mind sharing what type of documents you are referring to here?

These are open benchmarks that I have used but your problem is correct, more better datasets in the benchmark can solve this

Kahvana@reddit

Try this one, the free public basic rules from dnd 5e:
https://media.wizards.com/2018/dnd/downloads/DnD_BasicRules_2018.pdf

Valuable-Map6573@reddit

alibaba be like: "great lets fire our tech lead who brought us this far"

MelonGx@reddit

https://imgur.com/a/EyFsNuL

My 3.9L toy PC is running Qwen3.5-9B(Q4_K_M)!

You can host it too!

qubridInc@reddit

Wins: OCR, VQA, KIE
Loses: tables, handwriting

Best for document extraction, not complex layouts

witek_smitek@reddit

Maybe it's a stupid question, but why there are no qwen3.5 27B dense and 35B MoE variants in that benchmark?

Intelligent-Form6624@reddit

I’d like to see these too

rebelSun25@reddit

Interesting. I was testing structured output on openrouter, with qwen 9b, 27b, gemini 3 flash and 3.1 flash preview.

Things were even, until i provided a slightly larger json schema. Then qern models became suddenly dumb. Whatever they did wel until that point, was suffering, including the additional json schema properties.

I wonder if I hit some limit

Long_comment_san@reddit

As I predicted a while ago, we're gonna hit a functional ceiling really quick. There're only so many tasks we have that would need AI help.

Monad_Maya@reddit

I think we need better benchmarks tbh.

Every other small model is supposedly beating the frontier ones at a super small subset of benchmarks.

rm-rf-rm@reddit

Rather than better benchmarks, we need tests - specifically e2e tests for agents.

Sure, we can have that too.

RRUser@reddit

Are there any other open source models that can compliment the Qwen series for Table extraction and Handwriting OCR?

There’s Nanonets-ocr-s , PaddleOCR VL, olmOCR 2 etc

do you plan on evaluating the bigger ones - 27B, 122B and 397B?

Zulfiqaar@reddit

Looks like it has some great uses! I'm also very surprised at GPT4.1 - why is it doing so well, as a non-reasoning model, compared to everything else there?

Interesting_lama@reddit

Lightonocr 2?

In my benchmark it was working better than glm ocr, dots ocr and paddle ocr.

Documents with heavy tables

seamonn@reddit

Light on OCR did better on technical documents while GLM OCR did better on Comics, Manga etc.

On my list, will add soon

It trades blows with GLM OCR

MokoshHydro@reddit

Comparison with GLM-OCR will be interesting.

https://idp-leaderboard.org/compare/?models=qwen3-5-9b,glm-ocr

Here’s the comparison

More equal comparison would be between 0.8B and 2B models

Miserable-Dare5090@reddit

nanonets ocr2 beats the 9B it seems

Yes, cause that’s a more customised model for this usecase

existingsapien_@reddit

lowkey insane that a 9B open model is hanging with frontier models 💀

Additional_Split_345@reddit

These results are interesting because document processing is one of the areas where smaller models can actually compete with frontier models.

OCR cleanup, layout understanding, and structured extraction are tasks where context length and pattern recognition matter more than deep reasoning.

Seeing a 9B model outperform some frontier APIs on text extraction isn’t that surprising if the training data contained a lot of document-style corpora.

For local setups this is huge because document pipelines (PDF parsing, invoices, forms) are one of the most common enterprise workloads.

Also you can hover to find out the exact results on capability radar

Septerium@reddit

That is great. Even with very long reasoning, it might be much more energy-efficient to use a small qwen model instead of Gemini or GPT if you can afford to wait