Label extraction from PET/CT reports using Large Language Models - Bracci, Jacopo; Capobianco, Nicolo; Shah, Vijay; Spottiswoode, Bruce; Giobergia, Flavio (EANM’24 Abstract Book Congress, 2024)
Aim/Introduction: PET/CT clinical reports are typically written in free-text format. Consequently, while they contain a wealth of valuable and reliable information, it is challenging to store and structure this information in a way that it can be searched, modeled and analyzed in a consistent and efcient manner. This study aims to assess the feasibility of extracting structured data from textual reports by employing the latest Large Language Models (LLMs). Materials and Methods: We reviewed PET/CT reports from 31 patients with confrmed lung cancer. A human reader synthesized these reports into a structured format, categorizing each anatomical location (AL) as either “Presence” or “Absence” of a lesion. We then prompted publicly available LLMs to structure given reports, using Few-Shot (FS) learning [1] with up to 4 shots (reports and expected output fed to the model as examples), to assess the impact on performance. The evaluation was conducted on 27 held-out reports, using the F1-score metric. The dataset’s AL distribution included an average of 23.4 ALs for “Absence” (80 unique ALs) with the 5 most frequent being: [bladder: 31, spleen: 31, liver: 29, kidneys: 24, aorta: 22] and 6.9 ALs for “Presence” (143 unique ALs) with the 5 most frequent being: [right-upper-lobe: 4, right-lung-parenchyma: 4, right-lower-lobe: 3, right-middle-lobe: 3, left-upper-lobe: 3]. We evaluated six LLMs: GPT-4-Turbo, Llama3-70B, Llama3-8B, Mistral-7B, Mixtral-8x22B, Mixtral-8x7B. Results: The results for the “Absence” class showed that Llama3-70B performed best, with an average F1-score of 93.02% when prompted with 4-shots. All models demonstrated a signifcant increase in performance from 0-shots (i.e., no examples provided) to 1-shot, with an average F1-score increase of 30.12% (from 35.20% to 65.32%). The performance plateaued with 2 or more shots, averaging 80.24% across all models. For the “Presence” class, Mistral-7B had the highest average F1-score of 69.00% with 3 shots. A similar trend was observed with an increase from 0 to 1-shot, with the average F1-score increase of 20.53% (from 35.77% to 56.30%). The performance was also stable with 2 or more shots, with an average F1-score of 57.25% across all models. Conclusion: The study indicates that LLMs are capable of structuring PET/ CT clinical report data efectively, with few-shot learning being a critical factor for achieving high accuracy. Future research should explore additional LLMs for optimal performance and the refnement of the shots used to enhance model performance. References: [1] Brown T B et al 2020 Language models are fewshot learners (arXiv:2005.14165