Low-Rank Adaptation vs. Fine-Tuning for Handwritten Text Recognition - Hüttner, Lukas; Mayr, Martin; Gorges, Thomas; Wu, Fei; Seuret, Mathias; Maier, Andreas; Christlein, Vincent (IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW), 2025)
The continuous expansion of neural network sizes is a notable trend in machine learning, with transformer models exceeding 20 billion parameters in computer vision. This growth comes with rising demands for computational resources and large-scale datasets. Efficient techniques for transfer learning thus become an attractive option in setups with limited data, as in handwriting recognition. RecentlThe continuous expansion of neural network sizes is a notable trend in machine learning, with transformer models exceeding 20 billion parameters in computer vision. This growth comes with rising demands for computational resources and large-scale datasets. Efficient techniques for transfer learning thus become an attractive option in setups with limited data, as in handwriting recognition. Recently, parameter-efficient fine-tuning (PEFT) methods, such as low-rank adaptation (LoRA) and weight-decomposed low-rank adaptation (DoRA), have gained wide-spread interest. In this paper, we explore tradeoffs in parameter-efficient transfer learning using the synthetically pretrained Transformer-Based Optical Character Recognition (TrOCR) model for handwritten text recognition with LoRA and DoRA. Additionally, we analyze the performance of full fine-tuning with a limited number of samples, scaling from a few-shot learning scenario up to using the whole dataset. We conduct experiments on the popular IAM Handwriting database as well as the historical READ 2016 dataset. We find that (a) LoRA/DoRA does not outperform full fine-tuning as opposed to a recent paper and (b) LoRA/DoRA is not substantially faster than full fine-tuning of TrOC

PASAL: Progress- and sparsity-aware loss balancing for heterogeneous dataset fusion - Gorges, Thomas; Scholz, Teresa; Saloman, Stefan; Zinnen, Mathias; Hoffmann, Juliane; Gourmelon, Nora; Maier, Andreas; Hettenkofer, Sebastian; Christlein, Vincent (Information Fusion, 2025)
Machine learning has seen widespread application in many areas. Despite theoretical advancements, the demand for qualitative and extensive data foundations is increasing. Real-world datasets are often small and combining them is challenging due to the resulting sparsity and heterogeneity. Existing combination techniques merge datasets into a common space before training, causing drawbacks such as data loss and distortion of annotations. To address this, we fuse heterogeneous datasets by jointly training dataset-specific weighted sub-networks. Balancing losses from heterogeneous data sources is challenging, as current techniques are inadequate. We propose a novel progress- and sparsity-aware loss balancing method (PASAL), which adaptively balances sub-network losses based on individual learning progress and sparsity. As an example, we present the application of PASAL to the olfaction domain, where predicting smell properties based on molecular structure is difficult due to subjective impressions, typically limited data, and a lack of unified datasets. By evaluating PASAL on the DREAM Olfaction Prediction Challenge, we improve the current state-of-the-art method from a Z-Score of 9.92 to 10.10. Furthermore, by treating our AI as an annotator, we surpass human performance in the odor and pleasantness categories with statistical significance. Our findings are supported by a feature analysis, indicating that our heterogeneous combination methodology enhances odor predictio
Keywords: Dataset fusion; Heterogeneous data; Deep learning; Loss balancing; Scent prediction; Olfaction