Multi-Language OCR for Mixed-Script Enterprise Documents

Arabic, Hebrew, Urdu, and Farsi don't just run right to left. Most people think that’s the only issue, but it’s really the simple bit. The tricky part is that these scripts combine three different rules, and a system must handle all three correctly at once or the output breaks down before extraction begins.

Character ordering must adhere to the Unicode bidirectional algorithm, not merely reverse left-to-right logic. The system must recognize scripts properly, treating joined Arabic letters as single units rather than separate code points. Line direction must stay consistent even when numbers or English text break into RTL text in the middle, which happens all the time in real documents. An Arabic purchase order containing an English SKU mid-sentence isn't a stress test. It's Monday.

Arabic has a complication that Hebrew lacks: a letter's shape changes with its position in a word. Initial, medial, final, isolated: four distinct glyph shapes for the same letter, linguistically speaking. A model, trained mainly on Latin text and fixed with a lookup-table swap, fails to recognize these forms as the same character in different guises. It only sees strange shapes and makes guesses, like a tourist trying to match letters in a phrasebook to a menu and hoping it works out.

Teams often make a mistake: they see this as a cleanup issue that another round can solve after extracting the text. It’s impossible. A standard OCR engine misorders characters inside words, flips word order across lines, and makes mixed bidirectional content unreadable. An Arabic invoice mixing Latin product codes ends up with fields that don’t line up and missing numbers. The output isn't just noisy, it's structurally wrong, and that matters more than it seems. Fuzzy matching or another model pass can fix noisy output later. You can't fix it later, since the right word order doesn't exist in the output anymore. The data isn't jumbled. It's gone, like a shredded document that's destroyed, not just disorganized.

The fix needs to be in the OCR layer itself, not a script added later. Miss one of those three needs, bidi support, script-smart splitting, or line-direction guessing, and mixed Latin-Arabic files break when you try to pull text out. Vendors tend to cite a single accuracy figure, typically derived from a sample with mostly Latin text. If you test the same system on Arabic or Hebrew documents from that batch, accuracy can fall from the high 90s to the high 80s, or even less. Cleanup can’t fix this, since the issue doesn’t originate there.

Why CJK tokenization errors cascade across every field after the first mistake

Chinese and Japanese cause unique issues with tens of thousands of characters and no word breaks. Korean, as it happens, uses spaces and avoids this tokenization problem. Chinese and Japanese words just flow together, so the system must figure out breaks from the characters alone.

Segmenters trained on Latin languages don't face this challenge. English tokens are mostly split by whitespace and punctuation; it's a task so easy it barely counts. CJK tokenization is a real guessing challenge, and a mistake there won’t just cause a minor, isolated error. The error spreads. A single word-split error early on throws off all later field extractions, since parsing needs to know where tokens begin and end. Get that boundary wrong once, and a downstream regex extractor could mark a correct kanji vendor name as invalid, just because segmentation passed it a mangled piece.

Japanese has a second, separate problem: vertical text. Traditional layouts are arranged in vertical columns that run right to left. A system that ignores layout treats those columns like lines of text, so the result is gibberish even when every character is read right. The recognizer might get every character right, but the document still looks like gibberish since the reading order was off before recognition started.

Add Latin text, an English brand, or a number code to Japanese text, and the system must change how it breaks words mid-line, not just mid-document. In international trade, it's common to see cross-border logistics documents with Chinese consignee details alongside English shipping terms. Getting it right requires detecting scripts by region, not just setting one language when the document enters the pipeline. This approach, of humans picking the language at ingestion, is wrong because the engine may not reliably auto-identify CJK script. Works for a single file. It fails on a daily queue of thousands of mixed documents, the only volume that matters in production.

How Devanagari's stacked characters expose the limits of bounding-box segmentation

Devanagari, Tamil, and Ethiopic scripts are part of the abugida family, where vowel markers attach to consonants in various positions, forming a single meaningful unit rather than separate characters. Unlike simpler scripts that just need a new character set, this can't be fixed with a lookup table. The syllable's identity comes from how its parts are arranged in space, so the system must interpret layout before it can grasp meaning.

A common OCR method, bounding-box segmentation, treats each character as if it's in its own separate box. Give it a Devanagari conjunct cluster and it performs as designed, isolating the base consonant and vowel diacritic in separate boxes, so treating them as two distinct characters rather than a single syllable. Downstream, the recognizer can't make sense of the cluster, segmentation ripped it up first. The extracted text loses whole syllables because segmentation breaks them into pieces that nothing was built to reassemble.

The key point to consider is that this isn't actually a recognition failure. It's a design assumption, borrowed wholesale from Latin script, applied to a writing system it was never built for. Hindi, Marathi, Nepali, and Devanagari documents are often used in South Asian medical, legal, and government records. Dropped syllables in those documents are hard to spot: the text seems fine, just incomplete, and finding them needs checks tuned to the script, not a standard review. Starting with a separate layer just for conjunct clusters solves this. Adding a correction after bounding-box segmentation has already erred is addressing an avoidable issue.

What per-region script detection actually means in a production pipeline, and why document-level language flags fail

Documents with a single language work fine with a single language flag. Most real enterprise documents don't cooperate. Invoices mixing Arabic addresses, English headers, and Chinese consignee fields are common in cross-border trade. It's Tuesday.

Per-region script detection means the system analyzes each part of the page separately, identifies the script in that area, and sends it to the matching recognition engine. The line between scripts stays sharp where one ends and the next starts. This allows one pipeline to process a bilingual English-French contract or Japanese-English invoice without manual document type identification beforehand.

Three things must work together for this to work. Script identification must happen automatically by inference, not be manually set up by a team for every expected document type. Detection must occur at the region or element level, as a page-wide flag would force a single model onto sections requiring distinct ones. Confidence scores must be given for each field, not averaged into one number for the whole document. A system could score 99% on English fields and 91% on Arabic ones in the same document, but if that difference is hidden in a single overall score, the failing part stays invisible to downstream users.

Field-level confidence makes that variance useful, not merely visible. They can pick a cutoff for each script and send anything below it to a person, before the mistake breaks something later. Documents like shipping manifests and customs declarations fail in ways that an aggregate accuracy metric doesn't capture, if detection isn't per-region. The document looks handled. It wasn't.

How the shift from character-recognition engines to vision-language models changes what is architecturally possible

Traditional OCR just turns pixels into characters, nothing more. Each problem discussed (directional inference, segmentation, layout logic) occurs after that point, added later as a standalone set of rules someone must write, test, and keep updating indefinitely for every new script.

It's what holds everything back. Adding a language to a rule-based pipeline creates another rule set, and that creates another potential failure point. It’s like whack-a-mole, with each mole as a different script family having its own rules, and it can’t handle the mixed-script volume that enterprise documents face daily.

VLMs work differently. A VLM processes the entire document image in one go, reasoning about layout, typography, and content together, rather than reading pixels, converting them to characters, and layering layout and direction rules afterward. Field edges, table formats, reading order, and label-value pairs are all figured out at once, not in layers.

The result: a Hebrew form field and the same one in Thai both produce identical structured data, with no need for custom extraction rules for either. How the model was trained now handles script diversity, not an operator configuring each document, which shifts the labor division greatly from the rule-based approach.

Still, the gap isn't fully bridged. A system that supports over 100 languages can still be under 80% accurate for the languages at the bottom of the list, even if its overall score is a comfortable 95 to 99% on standard documents. That range is genuine. It's also an average, and averages have a way of hiding exactly the script-level variance an enterprise needs to know before it commits to a vendor. Query vendors about their least accurate language, rather than the top performer. The answer reveals more about the system's design than the headline figure ever could.

How the leading systems handle mixed-script documents, what each does well and where each still fails

What matters isn't the label's language count. The problems we've covered, RTL flipping, CJK token breaks, Indic character pile-ups, and if confidence scores appear per field, are what really matter. Headline accuracy figures don't reveal the problems, and comparing vendors with just one number is the wrong approach.

ABBYY (FineReader Engine, FlexiCapture, Vantage) can process 190 to 201 languages on local servers and outputs structured XML or JSON. It excels with degraded sources, faxes, old scans, noisy historical documents, where the main challenge is reliably reading poor-quality paper. It achieves high character accuracy in multilingual, low-quality scans, making it popular in compliance industries valuing layout and raw data equally.

Google Cloud Document AI (and Cloud Vision) consistently earns high scores for layout detection and wide multilingual support, achieving 98% character accuracy on multilingual low-quality scans in tests. Its pay-per-use pricing suits enterprises with monthly document volumes that fluctuate instead of staying flat.

Microsoft Azure AI Document Intelligence combines OCR, layout analysis, and prebuilt and custom models. Its layout model extracts text, tables, and selection marks formatted for downstream LLM use. Its layout model extracts text, tables, and selection marks formatted for downstream LLM use, achieving over 99% accuracy on English and other European languages. Its non-English accuracy on certain document types may lag behind other systems. The gap is crucial for companies mainly using non-Latin scripts but irrelevant for Latin-script users.

PaddleOCR 3.0 is open-source and Apache-licensed, using PP-OCRv5 for multilingual text detection, PP-StructureV3 for layout analysis and table recovery, and PP-ChatOCRv4 for extracting key data. It handles over 100 languages, attracting teams keen to self-host, fine-tune for their own scripts, and avoid all per-page billing. This architecture takes less than 10MB, a plus for offline or low-power setups. Procurement decks often gloss over the true cost: an enterprise staffer must handle the work of running and maintaining the model themselves. That requires a real internal ML team, not just a budget you approve and then ignore, and many teams find that out the hard way after signing the contract.

Extend Parse 2.0 uses OCR, a vision-language model, and object detection, handling Arabic RTL, CJK tokenization, and Devanagari stacking via one API without needing separate language models or queues. It scores 0.847 Adjusted F1 and high Q&A accuracy across 100-plus languages on RealDoc-Bench, a benchmark that uses real production documents instead of clean test sets. It includes built-in per-region script detection and outputs structured JSON for direct use by downstream LLMs. This single API directly targets the operational problem noted earlier: teams must manually set a unique language queue for each document type, a system that fails when volume grows.

These firms, like Upstage, Naver Clova, and other Korea-specialized AI teams, handle Korean and CJK scripts better than English-first tools. They're worth considering for enterprises with mostly Korean, Japanese, or traditional Chinese documents, not just a few non-Latin texts mixed into a largely English pile.

If you learn one thing from comparing these, it’s that general-purpose accuracy and mixed-script accuracy are separate metrics, and most vendors share the one that makes them look better. Choosing the tool based on the highest number on the landing page is a mistake. Pick the one that matches the failure mode actually sitting in the document pile.

What enterprise teams consistently get wrong when they configure multilingual OCR for production

Relying on the listed language count is the most frequent error, and the one you should fix first. A vendor supporting 100 languages could get 99% in English yet fall below 80% for another language on their list. You must test against the enterprise's own documents before signing to know where a deployment actually lands, not after the contract's locked and the invoices are already flowing wrong.

Using just one language flag at ingestion for documents that actually mix scripts is the second most common mistake, and it directly causes the field-boundary bleed that per-region detection is meant to stop. When teams assume a document is monolingual just because it mostly appears so, Arabic invoices with embedded Latin product codes end up with misaligned fields that go unnoticed until reconciliation, three weeks later, when the numbers don't add up and someone has to figure out why.

If you test a system with perfect printed pages but real work uses messy faxes and handwritten forms, you're just doing a demo test, not a real one. How well a model reads a perfect PDF tells you almost nothing about its ability to process a crumpled faxed purchase order from a supplier three time zones away that arrived at 11pm. The difference between test cases and real-world use is the main point here: these errors are predictable, tied to specific scripts, and can be identified before going live. Finding them before deployment saves much more than discovering them afterward, in that one document field nobody checked until it was too late to fix.