Recent advances in vision-language models (VLMs) such as CLIP and BLIP have demonstrated strong generalization in visual reasoning tasks. However, their potential for medical image analysis, especially breast cancer prediction from mammograms, remains underexplored. This study investigates how a pretrained VLM can be adapted for full mammographic classification. Unlike prior approaches that rely on costly region-of-interest (ROI) annotations, we process entire mammograms and adapt a general-purpose VLM (EVACLIP) using soft prompts, selective fine-tuning, and bidirectional fusion strategies. We compare different fusion methods, including Concatenation, Gated-Residual, Cross-Modal, Co-Weighted and Bi-Attention. Experiments on the CBISDDSM dataset show that bidirectional fusion methods consistently outperform other fusion approaches, while providing enhanced explainability through improved attention localization. Results also demonstrate that our adapted generalpurpose VLM significantly outperforms a mammographyspecific model (Mammo-CLIP), under domain-shift, in both zero-shot and linear-probe settings. This suggests that largescale general-purpose VLMs, when properly adapted, can outperform domain-specific models, reducing the need for extensive annotation and paired image-text training.

This work was supported by LARSyS FCT funding (DOI: 10.54499/LA/P/0083/2020, 10.54499/UIDP/ 672 50009/2020, 10.54499/UIDB/50009/2020). F. Sarwar gratefully acknowledges the invaluable support of ISCTE-IUL and Instituto de Telecomunicações

Vision-Language Models,Fusion Techniques,Breast Cancer Prediction,Multimodal Learning

• Computer and Information Sciences - Natural Sciences
• Electrical Engineering, Electronic Engineering, Information Engineering - Engineering and Technology