Handwriting OCR in mass
Posted by batty_1@reddit | LocalLLaMA | View on Reddit | 18 comments
I have about 50 million pages of handwritten/machine print mix documents. I want to convert all of these to markdown, preserving structure. I need as close to perfect accuracy as possible on the handwritten elements: these are boilerplate forms with handwritten elements, so those handwritten elements are really the critical "piece".
I've been trying some variation of this for about six months and could never quite get it right: decimal points would be removed, leading negative signs, sloppy handwriting completely misunderstood, etc.
recently, I revisited the problem and tried Qwen3.5:9b loaded up on my 4070 super and I was astonished by the results. Damn near 100% accuracy for even very complicated scenarios (faded handwriting, "one-line" markout corrections, etc.). I am still able to achieve 30-40 tokens per second and a page takes about 10-15 seconds - this is spun up and being called using Ollama's GGUF, thinking disabled.
The issue I'm having is that, in about 20% of the pages, Qwen hits a repetition loop and starts flood filling the markdown with empty rows ("| | | ...") until it exceeds the token allowance. This is a double whammy: it both truncates the page results and runs for 3-5x as long (average page is 400-600 tokens vs. filling 2048 tokens with nonsense).
Repetition penalties don't seem to work, nor does any amount of prompt manipulation. I've tried various other versions of the same model in vLLM and llama.cpp, but I can't achieve the same accuracy. The quantization they have on the Ollama side is magic.
I tried Gemma4 last night and had about 95% the accuracy and no repetition loops and about a 30% speed increase - which was great, but not good enough for this use case.
Has anyone else encountered this, or had a similar use case they worked through, and can provide some guidance? I appreciate it.
Fine tuning isn't off the table, and that might be what it takes, but I wanted to ask you guys, first.
(the elephant in the room: I don't intend on running all 50 million pages through my one 4070 ultra. just trying to get the pipeline solid first)
codeprimate@reddit
The first two things I would PoC for this task would be PaddleOCR and/or AWS Textract.
A combination of tesseract for a first pass and validation/correction with Gemma4 or Qwen would likely tighten your accuracy by providing better grounding context, and help mitigate thinking loops.
batty_1@reddit (OP)
Haven't tried paddleOCR just yet. I'll give that a shot and toss it in the mix.
I have tried Textract at length. I've found that Qwen outperforms even the pricier Textract here (AnalyzeDocument) - and a lot of the motivation with mobiliIng Qwen and having it do the heavy lifting is the crazy cost savings over Textract (1.5 pennies per page vs 0.1 pennies per page).
I tried Docling/Tesseract as a first pass purely and fed it to Qwen (telling things like "This page has approximately 280 words") and it cut the loops in half, but still not perfect.
codeprimate@reddit
That is some good feedback on Textract, I might just switch from it myself.
Old_Hospital_934@reddit
try using llama-server, also include repeat penalty and presence penalty (use from qwen's official webpage) This should help.
batty_1@reddit (OP)
So, llama-server's mmproj seems far less optimized than Ollama. It adds significant overhead per page. That was another hurdle I saw (other than the accuracy loss, too). I will give it another shot though.
Old_Hospital_934@reddit
Ollama uses llama.cpp in the backend, oh and if you have an nvidia gpu, why not try vLLM using wsl? it's far better and more optimized for concurrency (handling multiple requests at once)
ML-Future@reddit
Try using a less quantized model.
In llamacpp i always use for OCR: --reasoning-budget 0
batty_1@reddit (OP)
Ollama pulls q4 natively, I think. I pulled and tested the q8 variant and the loops went from ~20% of the time to 10% of the time. Speed took a penalty (from ~12s/page to ~25s/page). I jumped to qwen3.5:27b and had no more loops, but per page time skyrocketed to 1 minute.
codeprimate@reddit
One of the great things about ollama is modelfiles. You can use a gguf of whatever model or quant you can find.
tgreenhaw@reddit
Your context is getting overloaded. Try doing something to start with a fresh context on every page. Maybe keep alive=0
batty_1@reddit (OP)
I should have been clear, sorry. Each PDF page is resized to 300 dpi and sent individually in a fresh context window with a very simplistic prompt (" create a markdown equivalent of the attached image"). I don't think the looping is context-limited.
TheWaywardOne@reddit
You might want to try to integrate something like this into your workflow:
https://www.llamaindex.ai/blog/liteparse-local-document-parsing-for-ai-agents
-dev4pgh-@reddit
I can only tell you what I have observed anecdotally, but I have gotten the impression that quantization can really interfere with vision, and as you have observed causes more repetition loops. I would go with the largest quant you can run at reasonable speed.
I also have had Qwen models give the best transcription, but there are many new models I have not tested thoroughly. If Gemma 4 can get close without the looping problem, it may be worth looking further into. Also, if you can figure out a simple way to detect the repetition loops, perhaps you could rerun the ones that got stuck in those loops through Gemma 4, but keep the Qwen3.5 results for all others.
batty_1@reddit (OP)
I have found that asking for markdown is what throws it for a loop. Simple text extraction -> no loops. I thought about a two phase approach of having Qwen pull the text and have Gemma take the output + the image and tell it to reconcile to true text with the images structure - it's just a complicated pipeline.
Though that's actually a really slick fall back I hadn't considered- I can stream the output and if I get 20+ repeated rows, kill it and flag to rerun with Gemma. That way I can get 90% of the pages with Qwen with 99%+ accuracy and push the last 10% to Gemma and get 97%+ accuracy.
-dev4pgh-@reddit
Thanks for the response -- now I am thinking about adding that to my own project!
I guess you could do the same thing with the higher-parameter Qwen3.5 that you mentioned in your other comment got rid of loops, and might be even more accurate than Gemma 4? This would of course increase the time, but might be worth it if there are not TOO many loops.
If you ever find better ways to get around these repetition loops, please do share -- they are one of the most inconvenient issues with handwriting, and overcoming them would be a huge improvement!
Silver_Answer7134@reddit
have you tried adjusting the temperature setting to something super low like 0.1
HIba_LDN@reddit
Did you try GLM OCR? It’s only a 0.9b model, and it worked well for me although my documents didn’t have many handwritten pages.
batty_1@reddit (OP)
I tried - crashed and burned on the handwriting.