AMD-HookNet for Glacier Front Segmentation - Wu, Fei; Gourmelon, Nora; Seehaus, Thorsten; Zhang, Jianlin; Braun, Matthias; Maier, Andreas; Christlein, Vincent (IEEE Transactions on Geoscience and Remote Sensing, 2023)
Knowledge on changes in glacier calving front positions is important for assessing the status of glaciers. Remote sensing imagery provides the ideal database for monitoring calving front positions; however, it is not feasible to perform this task manually for all calving glaciers globally due to time constraints. Deep-learning-based methods have shown great potential for glacier calving front delineation from optical and radar satellite imagery. The calving front is represented as a single thin line between the ocean and the glacier, which makes the task vulnerable to inaccurate predictions. The limited availability of annotated glacier imagery leads to a lack of data diversity (not all possible combinations of different weather conditions, terminus shapes, sensors, etc. are present in the data), which exacerbates the difficulty of accurate segmentation. In this article, we propose attention-multihooking-deep-supervision HookNet (AMD-HookNet), a novel glacier calving front segmentation framework for synthetic aperture radar (SAR) images. The proposed method aims to enhance the feature representation capability through multiple information interactions between low-resolution and high-resolution inputs based on a two-branch U-Net. The attention mechanism, integrated into the two branch U-Net, aims to interact between the corresponding coarse and fine-grained feature maps. This allows the network to automatically adjust feature relationships, resulting in accurate pixel classification predictions. Extensive experiments and comparisons on the challenging glacier segmentation benchmark dataset CaFFe show that our AMD-HookNet achieves a mean distance error (MDE) of 438 m to the ground truth outperforming the current state of the art by 42%, which validates its effectivenes

Contextual HookFormer for Glacier Calving Front Segmentation - Wu, Fei; Gourmelon, Nora; Seehaus, Thorsten; Zhang, Jianlin; Braun, Matthias; Maier, Andreas; Christlein, Vincent (IEEE Transactions on Geoscience and Remote Sensing, 2024)
Position changes in glacier calving fronts are important indicators for evaluating the health of ice sheet outlet glaciers and changes in ice dynamics. However, manual delineation of calving fronts in remote sensing imagery is a time-consuming task, resulting in potential large costs. Deep-learning-based methods have made remarkable progress in automatically segmenting and delineating glacier calving fronts from remote sensing imagery. The relatively few remote sensing images and the limited geometric changes for glacier observations both reduce the diversity of the data and exacerbate the difficulty in accurate segmentation. Here we describe a novel automatic method for detecting glacier calving fronts in synthetic aperture radar (SAR) images, termed HookFormer. Our approach processes high-resolution (target) and low-resolution (context) inputs with a unified transformer architecture. The global–local tokens from the context and the target branches are integrated purely by the proposed cross-attention mechanism and cross-interaction module to complement and enhance each other. Moreover, we redesign the HookFormer architecture based on the convolutional neural network (CNN) model AMD-HookNet aiming to improve computational efficiency while achieving significant performance gains with only half of the model parameters/FLOPs. We conduct an in-depth analysis and make extensive comparisons based on the challenging glacier segmentation benchmark dataset CaFFe. As the first pure transformer approach, HookFormer sets a new state of the art with a mean distance error (MDE) of 353m to the ground truth, outperforming the baseline, Swin-Unet, and AMD-HookNet by 53%, 39%, and 19%, respectively.<br

Data-efficient handwritten text recognition of diplomatic historical text - Mayr, Martin; Neumeier, Katharina; Krenz, Julian; Bürcky, Simon; Kordon, Florian; Seuret, Mathias; Zöllner, Jochen; Wu, Fei; Maier, Andreas; Christlein, Vincent (Multimedia Tools and Applications, 2025)
Traditional methods in handwritten text recognition primarily focus on generating basic transcriptions, which often fall short for in-depth humanities research. Our study enhances this by providing diplomatic transcriptions for German studies, meticulously reproducing the original manuscripts, including layout and expanded abbreviations. State-of-the-art sequence-to-sequence approaches for handwritten text recognition predominantly use Connectionist Temporal Classification (CTC) as an auxiliary loss of the encoder output to improve robustness and accuracy. This is not possible in this task due to the great differences in the length of diplomatic transcriptions. We propose using the basic transcription instead of the diplomatic one as an additional target for the CTC feedback. Additionally, we introduce positional encoding at the intersection between the encoder and decoder to resolve the conflict of competing encoder objectives, balancing CTC loss reduction with the maintenance of implicit positional encoding for the decoder. Our empirical tests on the newly created dataset “Nuremberg Letterbooks” demonstrate significant data efficiency improvements. With only 4000 training lines (about 130 transcribed pages), we achieve a Character Error Rate (CER) of 9.39% without expanded abbreviations and 12.07% with expanded abbreviations, outperforming the baseline errors of 14.26% and 68.21%, respectivel
Keywords: Handwritten text recognition; Historical document analysis; Transformers; Diplomatic transcriptions; Sample efficiency

Deep learning-based debris flow hazard detection and recognition system: a case study - Wu, Fei; Zhang, Jianlin; Liu, Dunlong; Maier, Andreas; Christlein, Vincent (Scientific Reports, 2025)
Debris flows are characterized by their suddenness, rapidity, large scale and destructive power, causing serious threat to the population in mountainous areas. Surveillance cameras are widely used in geological hazard monitoring and early warning projects. So far, video cameras are used as a passive tool for post inspection and not as an active role for debris flow monitoring and early warning. Inspired by recent developments of anomaly detection in the field of computer vision, in this paper, we propose a novel automatic debris flow detection and recognition system based on deep learning. It consists of a video feature extraction network using a 3D convolutional neural network (CNN), a debris flow hazard detection network using a multi-layer perceptron (MLP), and a debris flow hazard recognition network for verification employing another CNN. The proposed system takes the video sequences captured by the cameras as inputs and enables the detection and recognition of debris flow hazards. All the networks are optimized and evaluated on a newly annotated image dataset called Debrisflow23. Extensive experimental evaluations with a detection accuracy of 86.3 % AUC, a recognition accuracy of 83.7 % AUC, and an overall identification accuracy of 88.1 % AUC on the test dataset demonstrate that the proposed method possesses accurate and reliable debris flow warning capability. Thus, further precautions can be taken in advance to reduce the damage to human settlements and infrastructure caused by debris flows.<br
Keywords: Transfer learning; Convolutional neural network; Debris flow; Hazard detection and recognition

Evaluating learned feature aggregators for writer retrieval - Mattick, Alexander; Mayr, Martin; Seuret, Mathias; Kordon, Florian; Wu, Fei; Christlein, Vincent (International Journal on Document Analysis and Recognition, 2024)
Transformers have emerged as the leading methods in natural language processing, computer vision, and multi-modal applications due to their ability to capture complex relationships and dependencies in data. In this study, we explore the potential of transformers as feature aggregators in the context of patch-based writer retrieval, with the objective of improving the quality of writer retrieval by effectively summarizing the relevant features from image patches. Our investigation underscores the complexity of leveraging transformers as feature aggregators in patch-based writer retrieval. While we have experimented with various model configurations, augmentations, and learning objectives, the performance of transformers in this task has room for improvement. This observation highlights the challenges in this domain and emphasizes the need for further research to enhance their effectiveness. By shedding light on the limitations of transformers in this context, our study contributes to the growing body of knowledge in the field of writer retrieval and provides valuable insights for future research and development in this area.
CRIS-Team Scopus Importer:2024-06-28
Keywords: Computer vision; Document analysis; Document retrieval; Feature aggregation

fang: Fast Annotation of Glyphs in Historical Printed Documents - Kordon, Florian; Weichselbaumer, Nikolaus; Herz, Randall; van der Loop, Janne; Mossman, Stephen; Potten, Edward; Seuret, Mathias; Mayr, Martin; Wu, Fei; Christlein, Vincent (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2024)
The extraction and analysis of large numbers of glyphs, and the associated opportunities for constructing a corpus of glyphs from types of the fifteenth century, offer significant research potential for scholars in book science. Such a corpus could be used in many ways, not least in assisting in the identification of fragments, charting the movements of type, and examining the impact of wear on type. Recognising this potential, we have developed fang (Code is available at https://github.com/Werck-der-buecher/FAnG.), a software that efficiently extracts and categorises glyphs from historical printed documents. Our approach involves several stages: (1) using Optical Character Recognition to extract glyphs in bulk, (2) employing a joint energy-based model for character classification and out-of-distribution pruning, and (3) providing a comprehensive toolset for manual review and editing, including deletions/reassignments and sorting by similarity. A significant strength of this design is the utilisation of existing text transcriptions and the context-awareness of trained language models, eliminating the need for explicit glyph location ground truth or glyph templates. By parallelising the extraction, we can quickly process entire digitised books with hundreds of pages, setting our system apart from existing glyph annotation tools. In experiments on digital reproductions of the Catholicon and 36-line Bible, the method demonstrates good spatial coverage of the detected glyphs, high character classification accuracy, and yields a low number of outliers. Our system represents a significant advancement in historical document analysis, providing researchers with an efficient tool for glyph extraction and categorisation.
Created from Fastlane, Scopus look-up
Keywords: Glyph extraction; Historical document analysis; Optical character recognition

G2EMOT: Guided Embedding Enhancement for Multiple Object Tracking in Complex Scenes - Ma, Jianbo; Wu, Fei; Li, Chunjiang; Tang, Chuanming; Zhang, Jianlin; Xu, Zhiyong; Li, Meihui; Liu, Dongxu (IEEE Transactions on Instrumentation and Measurement, 2024)
The one-shot multiple object tracking framework simultaneously generates detected targets and re-identification (ReID) embeddings, and employs them to associate previous tracks. However, the continuity between the learning and extraction within the ReID task is unconsciously neglected because of treating them as two isolated stages. This leads to unreliable ReID embeddings and poses a challenge to target matching especially in complex and crowded scenarios. To this end, we propose a guided embedding enhancement multiple object tracker, named G2EMOT. It consists of three innovative designs aiming at improving the embedding representation capability for individual targets. First, based on the original feature map, a global instance-specific context decoupling (GICD) module is devised to facilitate the respective feature learning for detection and ReID branches. Then, a heatmap guided embedding enhancement (HGEE) module is introduced to connect the processes of embedding learning and extraction, ensuring that the detected target coordinates are accurately aligned with the discriminative ReID embeddings. Finally, to associate and stabilize matched targets, we present a novel two-gate guided embedding update (TG-GEU) strategy that dynamically updates ReID embeddings. With the proposed three components, G2EMOT achieves outstanding performance on popular MOT benchmarks, while outperforming the existing one-shot tracking methods by a large margin. In particular, it realizes IDF1 of 76.1% on MOT17 and IDF1 of 74.2% on MOT20. The source codes are released at https://github.com/ydhcg-BoBo/G2EMOT.<br
Keywords: Multiple object tracking; One-shot tracker; Embedding learning; Track association

ICDAR 2024 Competition on Multi Font Group Recognition and OCR - van der Loop, Janne; Kordon, Florian; Mayr, Martin; Christlein, Vincent; Wu, Fei; Rodriguez Salas, Dalia; Weichselbaumer, Nikolaus; Seuret, Mathias (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2024)
This competition investigates the performance of several methods for two types of analyses of early modern prints: (1) optical character recognition, and (2) font recognition at the character level. We have created and published a novel dataset that contains the ground truth for both tasks. The dataset has been carefully curated and annotated by an expert with several years of expertise in transcribing early modern prints. Both tasks involved two distinct tracks, differing in ground truth management: one that only allows the participants to use the provided data for model training and a second that removes this restriction. Out of the five participating teams, four participated in the first track, and three in the second one. The best team reached a text Character Error Rate (CER) of 0.82 % and a font CER of 2.96 % for the first track. In the second track, these numbers could be slightly improved to 0.81 % text CER and 2.78 % font CER.
Created from Fastlane, Scopus look-up
Keywords: Document analysis; Font recognition; Historical documents; Optical character recognition

Lightweight cross-attention-based HookNet for historical handwritten document layout analysis - Wu, Fei; Seuret, Mathias; Mayr, Martin; Kordon, Florian; Zöllner, Jochen; Wind, Sebastian; Maier, Andreas; Christlein, Vincent (International Journal on Document Analysis and Recognition, 2025)
Handwritten document layout analysis is a fundamental step in digitizing scanned ancient documents for further processing (e.g., optical character recognition). So far, single branch-based fully convolutional networks (FCN) dominate this field. However, we contend that this task faces significant challenges, particularly in layouts with only semantic differences rather than differences in character appearance. For example, in the U-DIADS-Bib dataset, distinguishing between the main text and chapter headings can confuse existing FCNs due to the presence of similar distractors. It is, thus, critical to integrate layout structural information into the network learning processes. Moreover, the single branch-based networks have an upper limit of constructing document contextual relationships. Therefore, we propose a novel two-branch framework, called lightweight cross-attention-based HookNet (Light-HookNet), for handwritten document layout segmentation. The layout contextual information is connected and interacted with the cross-attention mechanism between a global context branch and a local target branch. This allows to achieve information enhancement inside the target branch and information exchange across both branches. Additionally, the reduced network parameters and computational costs make the proposed method both lightweight and efficient. Extensive experimental results and performance comparisons with state-of-the-art approaches on the newly proposed U-DIADS-Bib dataset and the popular DIVA-HisDB dataset demonstrate the superiority and effectiveness of the proposed method.<br
Keywords: Attention; Two-branch neural network; Historical document images; Handwriting layout analysis

Low-Rank Adaptation vs. Fine-Tuning for Handwritten Text Recognition - Hüttner, Lukas; Mayr, Martin; Gorges, Thomas; Wu, Fei; Seuret, Mathias; Maier, Andreas; Christlein, Vincent (IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW), 2025)
The continuous expansion of neural network sizes is a notable trend in machine learning, with transformer models exceeding 20 billion parameters in computer vision. This growth comes with rising demands for computational resources and large-scale datasets. Efficient techniques for transfer learning thus become an attractive option in setups with limited data, as in handwriting recognition. RecentlThe continuous expansion of neural network sizes is a notable trend in machine learning, with transformer models exceeding 20 billion parameters in computer vision. This growth comes with rising demands for computational resources and large-scale datasets. Efficient techniques for transfer learning thus become an attractive option in setups with limited data, as in handwriting recognition. Recently, parameter-efficient fine-tuning (PEFT) methods, such as low-rank adaptation (LoRA) and weight-decomposed low-rank adaptation (DoRA), have gained wide-spread interest. In this paper, we explore tradeoffs in parameter-efficient transfer learning using the synthetically pretrained Transformer-Based Optical Character Recognition (TrOCR) model for handwritten text recognition with LoRA and DoRA. Additionally, we analyze the performance of full fine-tuning with a limited number of samples, scaling from a few-shot learning scenario up to using the whole dataset. We conduct experiments on the popular IAM Handwriting database as well as the historical READ 2016 dataset. We find that (a) LoRA/DoRA does not outperform full fine-tuning as opposed to a recent paper and (b) LoRA/DoRA is not substantially faster than full fine-tuning of TrOC

Stably Adaptive Anti-Occlusion Siamese Region Proposal Network for Real-Time Object Tracking - Wu, Fei; Zhang, Jianlin; Xu, Zhiyong (IEEE Access, 2020)
Siamese region proposal network has made remarkable achievements in visual object tracking because of its balanced accuracy and speed. However, it regards tracking as a local one-shot detection task, which lose the power of updating the appearance model online thereby cannot handle the object-occlusion, fast motion and out-of-view situations. To tackle this problem, we propose a method that combines adaptive Kalman filter with Siamese region proposal network (Anti-occlusion-SiamRPN) to make full use of the object spatial-temporal information. Specifically we first extract target features through deep network and then uses adaptive Kalman filter to predict target trajectory in these difficult scenarios. Further this trajectory is used to select the candidate area of the next frame for Siamese region proposal network, which improve the searching mechanism. In this way, the introduction of adaptive Kalman filter makes the tracking process online learning which makes up for the disadvantage that Siamese region proposal network can only track offline. In addition, a hard example discrimination method (HEDM) is proposed to estimate whether the occlusion occurs and how seriously it is, which also improve Kalman filtering mechanism to make it update adaptively. Our method being evaluated with the speed of 80 FPS on five widely-applied challenging benchmarks including OTB2013, OTB2015, OTB50, VOT2016 and VOT2018. The extensive experimental results demonstrate our method achieves state-of-the-art effects and great improvement in comparison to other trackers.
Created from Fastlane, Scopus look-up
Keywords: Kalman filter; occlusion judgment; real-time tracker; Siamese region proposal network; Visual tracking

STCMOT: Spatio-Temporal Cohesion Learning for UAV-Based Multiple Object Tracking - Ma, Jianbo; Tang, Chuanming; Wu, Fei; Zhao, Can; Zhang, Jianlin; Xu, Zhiyong (IEEE International Conference on Multimedia and Expo (ICME), 2024)
Multiple object tracking (MOT) in Unmanned Aerial Vehicle (UAV) videos is important for diverse applications in computer vision. Current MOT trackers rely on accurate object detection results and precise matching of target reidentification (ReID). These methods focus on optimizing target spatial attributes while overlooking temporal cues in modelling object relationships, especially for challenging tracking conditions such as object deformation and blurring, etc. To address the above-mentioned issues, we propose a novel Spatio-Temporal Cohesion Multiple Object Tracking framework (STCMOT), which utilizes historical embedding features to model the representation of ReID and detection features in a sequential order. Concretely, a temporal embedding boosting module is introduced to enhance the discriminability of individual embedding based on adjacent frame cooperation. While the trajectory embedding is then propagated by a temporal detection refinement module to mine salient target locations in the temporal field. Extensive experiments on the VisDrone2019 and UAVDT datasets demonstrate our STCMOT sets a new state-of-the-art performance in MOTA and IDF1 metrics. The source codes are released at https://github.com/ydhcg-BoBo/STCMOT.<br
Keywords: Multiple Object Tracking; Spatio-Temporal Model; Unmanned Aerial Vehicle

Updating Siamese trackers using peculiar mixup - Wu, Fei; Zhang, Jianlin; Xu, Zhiyong; Maier, Andreas; Christlein, Vincent (Applied Intelligence, 2023)
Siamese network-based trackers are commonly used for tracking due to their balanced accuracy and speed. However, the fixed template from the initial frame is used to localize the target in the current frame, and never being updated in Siamese trackers. This loses the power of updating the appearance model online and easily induces tracking failures for the intrinsic properties of tracking processes such as constantly changing scenes and endless distractors. To address this problem, we propose a simple yet effective visual tracking framework for updating the appearance model in a novel formulation that introduces a peculiar mixup method both for training and inference phase of Siamese trackers (named PMUM-Siam). It consists of a template matching network and a mixup module. First, instead of center-cropping the inexact predictions for tracking, we use the template matching network which trained with predefined anchor boxes to learn to select the best candidate from similar distractors. Second, the mixup module is used to fuse and update the trade-off appearance model between the best candidate and the groundtruth. Our method greatly enhances the capability of target identification and target localization in Siamese trackers. To further demonstrate the generality of the proposed method, we integrate our PMUM-Siam into two representative Siamese trackers (SiamFC and SiamRPN+ +). Extensive experimental results and comparisons on five challenging object tracking benchmarks including OTB-2013, OTB-2015, OTB-50, VOT-2016, and VOT-2018 show that PMUM-Siam achieves leading performance with an average speed of 300 FPS.
Created from Fastlane, Scopus look-up
Keywords: Object tracking; Siamese network; Template update