Decision making in radiology often requires unifying multimodal data sources, including radiology concept knowledge base, structured patient electronic medical records (EMR) with clinical notes, and structured EMR metadata. Existing artificial intelligence (AI) systems are ineffective at integrating and reasoning across these modalities. Hence, many complex diagnostic decisions remain reliant on clinicians supported primarily by fragmented search tools and manual heuristics. While recent advances in retrieval-augmented generation (RAG) and Large Language Models (LLMs) offer encouraging developments, limitations persist in their adaptability, multimodal reasoning capabilities, and decision reasoning transparency. Despite the success of preference optimization, reinforcement learning (RL), and agentic frameworks in broader AI domains, their application in image-guided clinical workflows remains relatively unexplored. We introduce a multi-agent AI framework which synthesizes radiology domain knowledge and real time patient data in order to aid radiologists in uncovering clinical insights. Agents query from a centralized multimodal vector store and a structured data store, and an RL fine-tuning process improves system performance. Evaluating on 100 clinical questions, our system achieves Recall@5 of 0.88, Context Recall of 1.00, and Semantic Similarity of 0.94.
Watch the team present this project at 01:19:12 in the session recording here.
Keywords: agentic AI, radiology, clinical decision support, reinforcement learning, retrieval-augmented generation, centralized vector store, multimodal, queryable structured data
Faculty Advisor
Dr. Pamuksuz, a professor of AI, specializes in applied mathematics, machine/deep learning, responsible and generative AI. He has contributed to various analytics journals, including IEEE Transactions on Artificial Intelligence, and has shared his insights at several national and international conferences.
Since joining the University of Chicago as a faculty member in 2018, Dr. Pamuksuz has taught a range of subjects, including data mining, machine learning, and linear & non-linear models, along with more specialized areas like AI-data science for leaders and Generative AI Research. His supervision of capstone theses has often centered on computer vision and natural language processing.
In the professional realm, Dr. Pamuksuz has an impressive record. He has led data science teams at several Fortune 500 companies, provided expert consultancy in architecting cloud-based machine learning solutions, and co-founded Inference Analytics in 2018. Under his leadership, Inference Analytics was recognized in 2023 as one of the top Machine Learning Companies in Illinois, marking a significant milestone in healthcare analytics in Chicago.
Dr. Pamuksuz’s academic path has taken him through some of Illinois’ most prestigious universities. He completed his MS in Computer Engineering/Science at Northwestern University and went on to earn his Ph.D. from UIUC. Today, he contributes his expertise as a faculty member at the University of Chicago. This journey, connecting three significant academic institutions, reflects a strong foundation and dedication to his field and a deep engagement with the state’s rich educational landscape. Outside of his professional endeavors, Dr. Pamuksuz is enthusiastic about hackathons, not only participating in several but also organizing them twice annually since 2020. His involvement has yielded notable success in various data challenges. As a sports fan, he follows Champions League Soccer, supporting Galatasaray, and enjoys engaging in basketball and volleyball during his leisure time. He also enjoys smooth/gypsy jazz where you can find him on Wednesdays in his favorite place Green Mill.