From Pixels to Structure: Lightweight Vision-Language Models for Document OCR and Structured JSON Extraction

Basu Bir U, Christlein V, Maier A, Zinnen M (2026)

Publication Type: Conference contribution

Publication year: 2026

Journal

Lecture Notes in Computer Science Springer Verlag

Publisher: Springer Science and Business Media Deutschland GmbH

Book Volume: 16974 LNCS

Pages Range: 502-519

Conference Proceedings Title: Lecture Notes in Computer Science

Event location: Wien AT

ISBN: 9783032360380

DOI: 10.1007/978-3-032-36039-7_30

Abstract

While massive, closed-source Vision-Language Models (VLMs) set strong benchmarks for document understanding, their dependence on commercial Application Programming Interfaces (APIs) limits adoption in institutional archives due to data autonomy concerns, recurring costs, and the environmental footprint of hyperscale computing. This is especially acute in heritage digitization, where documents include historical handwriting, domain-specific terminology (e.g., jewelry, prehistory, architecture), and non-standard layouts requiring high-dimensional structured extraction. We present a comparative study of eight open-source lightweight VLMs (≤7B parameters) for Optical Character Recognition (OCR)-to-structure across three university heritage collections. Given a document image, models must extract text and generate schema-compliant JavaScript Object Notation (JSON), enabling automatic validation and downstream use. We evaluate models under a constraint-aware protocol across zero-shot, few-shot, and fine-tuning settings, measuring extraction fidelity and structured-output quality using Character Error Rate (CER), Approximate Normalized Levenshtein Similarity (ANLS)∗, and mean Average Precision F1 (mAP-F1). To assess practical improvements, we run controlled additional studies on a baseline fine-tuning pipeline, testing the independent impact of (i) hyperparameter optimization, (ii) classical image preprocessing (illumination flattening, denoising, and Contrast Limited Adaptive Histogram Equalization (CLAHE)), and (iii) multi-stage training, each compared directly to the finetune baseline. Finally, we analyze the trade-off between dataset-specific fine-tuning and a single multi-dataset checkpoint, where joint training enables one model to operate across collections but can shift performance between datasets. Overall, we show that carefully adapted ≤7B VLMs can provide a sustainable, private, high-performing alternative to manual transcription or commercial black-box systems, and we offer actionable guidance for heritage institutions seeking institution-controlled OCR-to-JSON extraction. Code for this paper is available at our project repository (https://github.com/uddipan77/Analysis-of-Lightweight-Vision-Language-Models-for-Document-OCR-and-Structured-Output-Generation).

Authors with CRIS profile

Vincent Christlein Lehrstuhl für Informatik 5 (Mustererkennung) Andreas Maier Lehrstuhl für Informatik 5 (Mustererkennung) Mathias Zinnen Lehrstuhl für Informatik 5 (Mustererkennung)

How to cite

APA:

Basu Bir, U., Christlein, V., Maier, A., & Zinnen, M. (2026). From Pixels to Structure: Lightweight Vision-Language Models for Document OCR and Structured JSON Extraction. In Gernot A. Fink, Alicia Fornés, Koichi Kise, Daniel Lopresti (Eds.), Lecture Notes in Computer Science (pp. 502-519). Wien, AT: Springer Science and Business Media Deutschland GmbH.

MLA:

Basu Bir, Uddipan, et al. "From Pixels to Structure: Lightweight Vision-Language Models for Document OCR and Structured JSON Extraction." Proceedings of the 20th International Conference on Document Analysis and Recognition, ICDAR 2026, Wien Ed. Gernot A. Fink, Alicia Fornés, Koichi Kise, Daniel Lopresti, Springer Science and Business Media Deutschland GmbH, 2026. 502-519.

BibTeX: Download