--- license: mit base_model: baidu/Unlimited-OCR tags: - gguf - llama.cpp - ocr - vision-language - quantized pipeline_tag: image-text-to-text --- # Unlimited-OCR — GGUF (llama.cpp) GGUF conversions of [baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) (MIT) with a **measured per-quant quality ladder**: every number below traces to a reproducible evaluation run against exact rendered ground truth, so the cost of each quantization is interpretable rather than assumed. Part of a measured-ladder series for this model; MLX conversions evaluated with the same harness are in the collection. ## ⚠️ Runtime requirement — read first These files load on **stock llama.cpp built at commit `4fc4ec5` (build 168, 2026-07-01) or newer** — the first mainline llama.cpp with Unlimited-OCR (`deepseek2-ocr`) support (merged via PR #24969). You need **two files**: a language GGUF (e.g. `unlimited-ocr-Q5_K_M.gguf`) **and** the multimodal projector `mmproj-unlimited-ocr-F16.gguf`. One-click loaders that bundle an older llama.cpp (e.g. Ollama, LM Studio) will **not** load these until they update to a build that includes this architecture. If your tool reports an unknown architecture `deepseek2-ocr`, its llama.cpp is too old. Image inference is verified with **`llama-mtmd-cli`** (see Usage); `llama-server`'s multimodal endpoint does not yet accept this model's image prompt in this build (details in Usage). *(This runtime line is a property of the files. Note it is separate from how they were produced — see "How this was made" for the one converter-side change, which does not affect loading.)* ## Which file should I use? **Recommended default: `Q5_K_M` (2.07 GiB)** — the smallest variant with **no measured degradation** on this corpus (CER within noise of the BF16 baseline). `Q8_0` and `Q6_K` reproduce the BF16 baseline's *recognized text exactly* here (identical CER on every page) but are larger for no measured quality gain — though both decode ~25% faster than Q5_K_M on this hardware (see tok/s); prefer `Q8_0` if RAM allows and throughput matters. Below Q5_K_M, quality falls off a cliff. | File | Size (GiB) | Overall CER | Δ vs BF16 | CER excl. loop pages | Loop rate | Notes | |---|---|---|---|---|---|---| | `unlimited-ocr-BF16.gguf` | 5.473 | 0.78% | — (baseline) | 0.78% | 0/24 | Public anchor / baseline | | `unlimited-ocr-Q8_0.gguf` | 2.911 | 0.78% | +0.00 pp | 0.78% | 0/24 | Recognized text identical to BF16 here | | `unlimited-ocr-Q6_K.gguf` | 2.434 | 0.78% | +0.00 pp | 0.78% | 0/24 | Recognized text identical to BF16 here | | **`unlimited-ocr-Q5_K_M.gguf`** | **2.067** | **0.74%** | **−0.04 pp (within noise)** | 0.74% | 0/24 | **Recommended default** | | `unlimited-ocr-Q4_K_M.gguf` | 1.816 | 15.64% | +14.86 pp | 3.88% | 1/24 | Usable for clean/numeric pages; dense-tier CER 45%, one loop page — not the default here | | `unlimited-ocr-Q4_0.gguf` | 1.585 | 44.02% | +43.24 pp | 9.12% | 2/24 | Published as measured 4-bit cliff evidence | `mmproj-unlimited-ocr-F16.gguf` (0.769 GiB) is required by **every** variant. ## Measured quality (vs GGUF-BF16 baseline) Baseline is the **BF16 GGUF running under llama.cpp** — never a different runtime. Deltas are computed on unrounded values. | Variant | Overall CER | Overall WER | CER clean | CER dense | CER numeric | CER excl. loop | Loop rate | max len-ratio | gen tok/s | |---|---|---|---|---|---|---|---|---|---| | BF16 | 0.7792% | 0.7206% | 0.00% | 1.60% | 0.74% | 0.7792% | 0/24 | 1.13 | 172.5 | | Q8_0 | 0.7792% | 0.7206% | 0.00% | 1.60% | 0.74% | 0.7792% | 0/24 | 1.13 | 243.7 | | Q6_K | 0.7792% | 0.7206% | 0.00% | 1.60% | 0.74% | 0.7792% | 0/24 | 1.13 | 237.3 | | Q5_K_M | 0.7399% | 0.6881% | 0.00% | 1.60% | 0.62% | 0.7399% | 0/24 | 1.13 | 194.3 | | Q4_K_M | 15.6406% | 15.6419% | 0.95% | 45.23% | 0.74% | 3.8810% | 1/24 | 3.85 | 256.9 | | Q4_0 | 44.0241% | 50.1559% | 1.85% | 121.44% | 8.79% | 9.1207% | 2/24 | 5.53 | 299.2 | - **CER = character error rate** (Levenshtein / reference length); **WER** analogous on whitespace tokens. Lower is better. - **CER excl. loop pages** is the mean over pages that did *not* run away, so a single collapsed page shows up as a loop rather than silently inflating the headline. `Q8_0` and `Q6_K` match the BF16 baseline's per-page CER exactly on all 24 pages; `Q5_K_M` differs on one page (net −0.04 pp, within noise). - **Loop rate** counts pages whose output exceeded 1.5× the reference length (the runaway-repetition failure mode). - `gen tok/s` is **isolated generation throughput** (generated tokens ÷ time after model-load and vision-encode), single-image, M3 Max 36 GB, Metal. - **Peak memory is not reported** — the CLI does not expose it and it is not estimated. Corpus: 24 synthetic pages (3 difficulty tiers × 8 — clean prose, dense small-font, digit-heavy invoice), deterministic seed, exact rendered ground truth. Prompt `document parsing.`, temperature 0. **Decoding runs with repetition suppression OFF** (`--repeat-penalty 1.0`, no DRY), deliberately, so quantization instability shows up as honest high CER and visible loop pages rather than being masked. Normalization (grounding tokens `<|det|>…<|/det|>`, bbox coordinates, markdown, whitespace) is applied identically to reference and hypothesis, so CER reflects recognition, not layout markup. Detokenization is handled natively by llama.cpp (no byte-marker post-processing needed). Eval harness (corpus generator + CER/WER scorer) is open-source: https://github.com/vimalnakrani08/unlimited-ocr-eval-harness (the scoring core is backend-agnostic; the runner is MLX-specific). ## The headline: the 4-bit cliff, and mixed vs single-recipe 4-bit 8-, 6-, and 5-bit carry no measured recognition cost on this corpus; then 4-bit drops sharply, and the damage concentrates in the **dense small-font tier** (Q4_K_M 45% / Q4_0 121% dense CER, while clean pages stay ≤1.85%). The two 4-bit files use different quantization recipes (verified from the GGUF tensor types — described exactly, not assumed): - **`Q4_0`** — every transformer weight matrix at `Q4_0` (118 tensors), the output projection kept at `Q6_K` (1), and all normalization tensors in `F32` (36). A single low-bit recipe across the body. - **`Q4_K_M`** — a **mixed-precision K-quant**: `Q4_K` (95 tensors) with selected tensors promoted — `Q6_K` (12, incl. output and some attention-value projections), `Q8_0` (6, some feed-forward down projections), `Q5_0` (6, some expert down projections), and `F32` norms (36). Mixed precision helps a lot at 4-bit — `Q4_K_M` (15.64% CER, 1 loop) is **2.8× lower CER** than `Q4_0` (44.02% CER, 2 loops) — but **neither reaches BF16-level recognition**. The interpretable takeaway: for this architecture, the usable floor on this corpus is Q5_K_M; 4-bit is a genuine quality cliff that mixed precision softens without removing. ## Loop taxonomy (the failure modes behind the loop rate) The high-CER 4-bit pages fail in two distinct ways; `len-ratio` catches both: - **Verbatim cycle** — the model repeats a fixed phrase indefinitely. *Example (Q4_0, one dense page):* "The results of the data management system were recorded by the last year of the last year." repeated to the token cap. - **Non-terminating drift** — the model stays on the document's topic but confabulates fresh, wrong sentences and never emits end-of-sequence. *Example (Q4_0, one dense page):* the template "The quarterly report …" continued with varying invented predicates. The BF16/8/6/5-bit variants terminate cleanly on every page. ## Secondary: DRY sampler ("deployed behavior") — a measured trade, not a free win The primary numbers above use **no** repetition suppression. The base model's official pipeline applies `no_repeat_ngram_size`; llama.cpp's analogue is the DRY sampler. As a clearly-labeled secondary measurement, the two 4-bit files were re-run over all 24 pages with DRY (`--dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2 --dry-penalty-last-n -1 --dry-sequence-breaker none`, else identical): | Variant | Overall CER OFF → DRY | Previously-looping pages | Previously-clean pages | New loop induced by DRY | |---|---|---|---|---| | Q4_K_M | 15.64% → 10.47% | 286.1% → 1.3% | **3.88% → 10.86%** | one dense page (14.5% → 136.7%) | | Q4_0 | 44.02% → 28.46% | 428.0% → 146.9% | **9.12% → 17.69%** | one dense page (0% → 218%) | DRY lowers the aggregate by rescuing the worst pre-existing loops, **but it damages previously-clean pages** — most on the numeric/dense tiers whose correct output is legitimately repetitive (template invoices; Q4_K_M numeric rises 0.74% → 7.27%) — and can **induce new catastrophic loops** on pages that were fine without it. So repetition suppression is a trade specific to your document mix, not a universal fix, which is why the honest primary ladder is scored with it off. (The legitimately-repetitive-content explanation was hypothesized before the measurement rather than fitted to it, is consistent with the observations, and is not a proven mechanism.) ## Secondary: R-SWA (Reference Sliding Window Attention) Unlike current MLX ports, llama.cpp **does** implement Unlimited-OCR's R-SWA (constant-KV sliding window with an always-visible reference/visual prefix) — in open PR **[#24975](https://github.com/ggml-org/llama.cpp/pull/24975)** (`LLAMA_SWA_TYPE_REFERENCE`), **implemented but not yet merged** at publish time. - **The published GGUFs run under stock mainline (build 168 / `4fc4ec5`), which uses full multi-head attention (MHA)** for this architecture — the ladder above is all full-MHA. - To measure R-SWA itself, the **same BF16 GGUF** was run on a local build of master + #24975 (build 173 / `4d242d6`, isolated worktree). Activation is proven, not assumed: the identical file loads as `n_swa = 128` on that build vs `n_swa = 0` on stock mainline. That build also promotes the V-cache to F32 (a requirement noted in the PR). BF16, single-page fidelity, same 24 pages, same flags: | | BF16 · MHA (mainline) | BF16 · R-SWA (#24975) | |---|---|---| | Overall CER | 0.7792% | 0.3344% | | CER dense | 1.60% | 0.00% | | CER numeric | 0.74% | 1.00% | | gen tok/s | 172.5 | 168.2 | R-SWA and MHA produce **identical output on 21 of 24 pages**; of the three that differ, one dense page drives the overall improvement (MHA 12.8% → R-SWA 0%), numeric moves the other way (+0.27 pp), and no loops are introduced. So at BF16 on single pages, **R-SWA ≈ MHA fidelity** — which also validates that the full-MHA ladder above is a fair proxy for this model. **Important scope note:** this corpus is single-page documents, so it measures *fidelity parity only*, not R-SWA's headline **multi-page constant-memory** regime — that regime is not exercised here and is a logged follow-up. ## Cross-runtime note (secondary — never part of any degradation column) On the identical frozen corpus, this **GGUF BF16 baseline scores 0.78% overall CER vs 1.62% for the MLX BF16** conversion of the same weights. The per-tier picture is mixed and does not point one way: GGUF is much better on the dense tier (1.60% vs 4.86%) but the **numeric tier goes the other way** (GGUF 0.74% vs MLX 0.00%). This reflects unspecified differences between the two runtimes (detokenization, image preprocessing, kernels); isolating the cause is a logged follow-up. It is reported here only as an observation and never enters a quantization-degradation figure. ## Usage ### llama-mtmd-cli (verified — this is what produced every number above) ```bash llama-mtmd-cli \ -m unlimited-ocr-Q5_K_M.gguf \ --mmproj mmproj-unlimited-ocr-F16.gguf \ --image page.png \ -p "document parsing." \ --chat-template deepseek-ocr \ --temp 0 \ --repeat-penalty 1.0 \ --flash-attn off \ -n 2600 -c 16384 --no-warmup ``` Every flag matters: `--chat-template deepseek-ocr` (required), `--temp 0` (greedy), `--repeat-penalty 1.0` (suppression off — the measured setting; `--flash-attn off` matches the reference), `-c 16384` (large enough that the 2600-token cap, not the context, bounds any loop). Prompts follow DeepSeek-OCR vocabulary: `document parsing.` (layout + grounding markup, used for all measurements here), `Free OCR.` (plain text). Prompt viability is runtime-specific: on this build `Free OCR.` emits end-of-sequence immediately, while on mlx-vlm 0.6.3 the same prompt runs away into a repetition loop. Prefer `document parsing.` and strip the `<|det|>…<|/det|>` grounding markup downstream if you only want text. ### llama-server The server hosts the model (`llama-server -m unlimited-ocr-Q5_K_M.gguf --mmproj mmproj-unlimited-ocr-F16.gguf --chat-template deepseek-ocr -c 16384 --flash-attn off`), but in this build its multimodal request path returns `Failed to tokenize prompt` for this model (the chat template's image placeholder is not injected on the server endpoint). Use `llama-mtmd-cli` for image inference until upstream server support lands. ## How this was made 1. Converted from `baidu/Unlimited-OCR` (original config, unmodified hyperparameters) with llama.cpp's `convert_hf_to_gguf.py` → BF16 language GGUF + F16 mmproj. 2. **One converter-side change, disclosed for reproducibility:** mainline's converter did not yet have this checkpoint's tokenizer pre-tokenizer hash registered, so conversion aborted. The tokenizer is DeepSeek-V3 family — its pre-tokenizer regexes match llama.cpp's `deepseek-v3` type byte-for-byte — so the hash was registered to `deepseek-v3` (a one-line addition to `get_vocab_base_pre`). This is a **converter** change only; it does **not** affect the runtime or how these files load (they load on stock mainline — see the runtime line at the top). It is a candidate for upstreaming via `convert_hf_to_gguf_update.py`. 3. Quantized with `llama-quantize` (Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q4_0). Q4_K_M measured at 1.816 GiB. 4. Smoke-tested (BF16, CER 0.0000% on a held-out page) before quantizing, then the full ladder was evaluated (tables above). ## Limitations — read this - **These are single-page OCR measurements.** R-SWA's long-horizon, constant-memory multi-page regime is not exercised by this corpus (see the R-SWA section). Very-long single-pass multi-page behavior is unmeasured here. - **4-bit is a measured quality cliff** on this architecture — do not deploy `Q4_0`/`Q4_K_M` for dense documents without running the harness on your own distribution first. - Synthetic corpus: clean renders, three fonts, English. Real-world scans (skew, noise, handwriting, other scripts) are not covered by these numbers. - Figures are specific to this corpus and to llama.cpp build 168 (`4fc4ec5`). ## Attribution Base model © Baidu, released under the MIT License — please cite their [technical report](https://arxiv.org/abs/2606.23050). This repository contains converted and quantized weights and adds no training. GGUF conversion and quantization by the uploader; R-SWA is the work of llama.cpp PR #24975.