The field of medical artificial intelligence (AI) is advancing rapidly, heralding a new era of diagnostic accuracy and patient care. Researchers have been focusing on developing AI solutions for specific tasks, but current medical AI systems are often limited to narrow applications, hindering their broader adoption in clinical practice.
In face of this limitation, in a new paper A Generalist Learner for Multifaceted Medical Image Interpretation, a research team from Harvard Medical School, Jawaharlal Institute of Postgraduate Medical Education and Research, and Scripps Research Translational Institute proposes MedVersa, a generalist AI model designed to enable flexible learning and tasking for medical image interpretation.
The core innovation of MedVersa lies in its use of a large language model as a learnable orchestrator. This orchestrator integrates multimodal inputs and executes tasks using language and vision modules. This architectural design allows MedVersa to overcome the limitations of traditional approaches by combining visual and linguistic supervision in its learning processes and supporting on-the-fly task specification through language.
MedVersa is a versatile model capable of excelling in both vision-language tasks, such as generating radiology reports and answering visual questions, and vision-centric challenges, including detecting anatomical structures and segmenting medical images. This dual capability enables MedVersa to train on diverse medical data across multiple modalities and tasks, resulting in general, shared representations.
To support the development of MedVersa, the researchers curated a diverse, multimodal dataset called MedInterp, specifically designed for multifaceted medical image interpretation. Training and assessing MedVersa on the MedInterp dataset demonstrated that it surpasses state-of-the-art specialist counterparts in nine tasks.
In radiology report generation, MedVersa outperformed MAIRA-1 21, a specialist multimodal model from Microsoft, and Med-PaLM M 13, a generalist biomedical foundation model from Google that is ten times larger than MedVersa. Additionally, MedVersa excelled in visual localization tasks, surpassing a well-established object detector in localization tasks. Furthermore, MedVersa demonstrated superior performance compared to state-of-the-art specialist methods in various other tasks, including longitudinal study comparisons, region-of-interest captioning, open-ended visual question answering, and chest pathology classification.
To the best of the research team’s knowledge, MedVersa is the first generalist medical AI (GMAI) model to support multimodal inputs, outputs, and on-the-fly task specification. The development of MedVersa potentially unlocks new opportunities for building more versatile GMAI models.
The paper A Generalist Learner for Multifaceted Medical Image Interpretation is on arXiv.
Author: Hecate He | Editor: Chain Zhang
We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.
In addition to our world-class entertainment and luxurious ambiance, our sapphire new york offers a gourmet dining experience that rivals the finest restaurants in the city. Our menu features a variety of delectable dishes, expertly prepared by our talented chefs. From sumptuous appetizers to exquisite main courses and decadent desserts, our culinary offerings are designed to delight your taste buds.
I have a great gaming site. There are a lot of great games out there: compatibility testing games, kiss simulation games, office dating, dating with handsome guys… But I’ll keep it for solo play. I won’t tell you it’s a love tester game.