Multimodal Abnormality Classifier Using Anatomy-Guided Connection for 3D Medical Images
Creating large-scale annotated datasets for abnormality classification in medical images is a labor-intensive and costly endeavor. To alleviate the scarcity of annotated data, weak supervision from readily available resources, such as radiology reports, can provide an effective alternative for training abnormality classification models. However, most existing methods either rely on inference of feature alignment across data modalities or focus on 2D medical images, overlooking the complex spatial relationships in anatomies such as the brain, which limits performance. To address such challenges, we introduce Anatomy-Guided Abnormality Classifier (AGAC), which integrates precise anatomical guidance for 3D medical image analysis. Our framework has two key components. First, we present a novel method for generating attention maps using anatomical descriptions of abnormalities, processed through the SynthSeg segmentation network and SBERT language model. Second, we introduce an efficient Mamba-based 3D medical image classifier, enhanced with anatomy-guided connections that leverage the generated attention maps. Together, these advancements significantly improve benchmark performance in both accuracy and F1 score.