Interactive Surgical Training in Neuroendoscopy: Real-Time Anatomical Feature Localization using Natural Language Expressions - Matasyoh, Nevin Musula; Schmidt, Rudiger; Zeineldin, Ramy; Spetzger, Uwe; Mathis-Ullrich, Franziska (IEEE Transactions on Biomedical Engineering, 2024)
Objective: This study addresses challenges in surgical education, particularly in neuroendoscopy, where the demand for optimized workflow conflicts with the need for trainees' active participation in surgeries. To overcome these challenges, we propose a framework that accurately identifies anatomical structures within images guided by language descriptions, facilitating authentic and interactive learning experiences in neuroendoscopy. Methods: Utilizing the encoder-decoder architecture of a conventional transformer, our framework processes multimodal inputs (images and language descriptions) to identify and localize features in neuroendoscopic images. We curate a dataset from recorded endoscopic third ventriculostomy (ETV) procedures for training and evaluation. Utilizing evaluation metrics, including “R@n,” “IoU= θ ,” “mIoU,” and top-1 accuracy, we systematically benchmark our framework against state-of-the-art methodologies. Results: The framework demonstrates excellent generalization, surpassing the compared methods with 93.67 % accuracy and 76.08 % mIoU on unseen data. It also exhibits better computational speed compared with other methods. Qualitative results affirms the framework's effectiveness in precise localization of referred anatomical features within neuroendoscopic images. Conclusion: The framework's adeptness at localizing anatomical features using language descriptions positions it as a valuable tool for integration into future interactive clinical learning systems, enhancing surgical training in neuroendoscopy. Significance: The exemplary performance reinforces the framework's potential in enhancing surgical education, leading to improved skills and outcomes for trainees in neuroendoscopy.
CRIS-Team Scopus Importer:2024-09-27
Keywords: Anatomical feature localization; Biomedical imaging; Endoscopic Third Ventriculostomy; Feature fusion; Multimodal deep learning; Neuroendoscopy; Neurosurgery; Surgery; Surgical training; Task analysis; Training; Transformer; Transformers; Visualization

Optimising speech recognition using LLMs: an application in the surgical domain - Matasyoh, Nevin Musula; Zeineldin, Ramy; Mathis-Ullrich, Franziska (Current Directions in Biomedical Engineering, 2024)
Automatic speech recognition (ASR), powered by deep learning techniques, is crucial for enhancing humancomputer interaction. However, its full potential remains unrealized in diverse real-world environments, with challenges such as dialects, accents, and domain-specific jargon, particularly in fields like surgery, persisting. Here, we investigate the potential of large language models (LLMs) as error correction modules for ASR. We leverage Whisper-medium or ASRLibriSpeech for speech recognition, and GPT-3.5 or GPT-4 for error correction. We employ various prompting methods, from zero-shot to few-shot with leading questions and sample medical terms to correct wrong transcriptions. Results, measured by word error rate (WER), reveal Whisper's superior transcription accuracy over ASR-LibriSpeech, with a WER of 11.93% compared to 32.09%. GPT-3.5, with the few-shot with medical terms prompting method, further enhances performance, achieving a 64.29% and 37.83% WER-reduction for Whisper and ASR-LibriSpeech, respectively. Additionally, Whisper exhibits faster execution speed. Substituting GPT-3.5 with GPT- 4 further enhances transcription accuracy. Despite some few challenges, our approach demonstrates the potential of leveraging domain-specific knowledge through LLM prompting for accurate transcription, particularly in sophisticated domains like surgery.
CRIS-Team Scopus Importer:2024-10-25
Keywords: Automatic speech recognition; Error correction; Few-shotprompting; Large language model; Zero-shot prompting

SAMSurg: Surgical Instrument Segmentation in Robotic Surgeries Using Vision Foundation Model - Matasyoh, Nevin Musula; Mathis-Ullrich, Franziska; Zeineldin, Ramy (IEEE Access, 2024)
Integrating Minimally Invasive Surgery (MIS) with robotic systems has revolutionized surgical procedures by enhancing precision, reducing patient discomfort, and shortening recovery times. However, the dynamic and visually complex environment of robotic surgeries poses significant challenges in the accurate semantic segmentation of surgical instruments, a critical task for navigating surgical robots and ensuring surgical safety. This necessitates the development of advanced computational techniques capable of overcoming these challenges. To address these limitations in robotic surgeries, we introduce SAMSurg, an adaptation of the Segment Anything Model (SAM). This vision foundation model accurately segments diverse objects in images for various computer vision applications. SAMSurg leverages the pre-trained representations of SAM's image and prompt encoders, focusing on fine-tuning the mask decoder to adapt to the specific demands of surgical instrument segmentation in MIS. The model is evaluated using over 77K image-mask pairs from various surgical datasets, demonstrating its adaptability across different surgical interventions, disciplines, and instrument types. This extensive evaluation underscores SAMSurg's superior performance against state-of-the-art models like SAM and MedSAM, with Dice Similarity Coefficient (DSC) scores as high as 96.90%. Furthermore, it demonstrates strong generalization capabilities across various surgical contexts, achieving higher DSC scores than state-of-the-art models, such as SAM and MedSAM, on unseen datasets. SAMSurg's performance shows a significant improvement in the segmentation of surgical instruments for robotic surgeries, promising to enhance surgical techniques and patient care outcomes.
CRIS-Team Scopus Importer:2025-02-04
Keywords: Box prompting; robotic surgery; surgical instrument segmentation; vision foundation model