This study investigates four parameter-efficient adaptation strategies for developing a healthcare-specialized large language model (LLM) based on LLaMA 3.1 8B, designed to enhance accuracy, factual reliability, and interpretability in clinical question answering. Using a unified experimental pipeline, the team trained and evaluated models across three benchmark biomedical QA datasets, PubMedQA (1,000 samples), MedQA (USMLE-style), and MedMCQA (182,000 questions), and validated them on MIMIC-CXR (377,000 radiology reports) for cross-domain generalization. The four evaluated methods include (1) a Baseline model, (2) In-Context Learning (ICL), (3) Direct Preference Optimization (DPO), and (4) Low-Rank Adaptation (LoRA) Fine-Tuning, complemented by a Memento continual-learning extension for memory-augmented reasoning.
Quantitative analysis shows that LoRA Fine-Tuning achieved the best performance with a macro-average accuracy of 0.75 (+0.17 vs. baseline) and faithfulness improvement of +19%, answer relevance +20%, and context precision +21%. DPO reduced speculative outputs by 15% and improved stylistic alignment with clinician-preferred reasoning, while ICL enhanced interpretability (+12% human-rated clarity) but exhibited prompt sensitivity (±3.7%). Results demonstrate that parameter-efficient adaptation can achieve comparable or superior factual grounding at 60% lower training cost than full fine-tuning.
Overall, these findings confirm that scalable, domain-specialized adaptation methods, particularly LoRA and DPO, can make medical LLMs both cost-effective and clinically reliable, paving the way for trustworthy AI-assisted diagnostic reasoning within healthcare systems.
Watch the team present this project at 07:58 in the session recording here.
Keywords: Healthcare LLM, clinical decision support, parameter-efficient fine-tuning, LoRA, Direct Preference Optimization, In-Context Learning, RAGAS evaluation, biomedical NLP, diagnostic accuracy, factual faithfulness
Faculty Advisor
Dr. Pamuksuz, a professor of AI, specializes in applied mathematics, machine/deep learning, responsible and generative AI. He has contributed to various analytics journals, including IEEE Transactions on Artificial Intelligence, and has shared his insights at several national and international conferences.
Since joining the University of Chicago as a faculty member in 2018, Dr. Pamuksuz has taught a range of subjects, including data mining, machine learning, and linear & non-linear models, along with more specialized areas like AI-data science for leaders and Generative AI Research. His supervision of capstone theses has often centered on computer vision and natural language processing.
In the professional realm, Dr. Pamuksuz has an impressive record. He has led data science teams at several Fortune 500 companies, provided expert consultancy in architecting cloud-based machine learning solutions, and co-founded Inference Analytics in 2018. Under his leadership, Inference Analytics was recognized in 2023 as one of the top Machine Learning Companies in Illinois, marking a significant milestone in healthcare analytics in Chicago.
Dr. Pamuksuz’s academic path has taken him through some of Illinois’ most prestigious universities. He completed his MS in Computer Engineering/Science at Northwestern University and went on to earn his Ph.D. from UIUC. Today, he contributes his expertise as a faculty member at the University of Chicago. This journey, connecting three significant academic institutions, reflects a strong foundation and dedication to his field and a deep engagement with the state’s rich educational landscape. Outside of his professional endeavors, Dr. Pamuksuz is enthusiastic about hackathons, not only participating in several but also organizing them twice annually since 2020. His involvement has yielded notable success in various data challenges. As a sports fan, he follows Champions League Soccer, supporting Galatasaray, and enjoys engaging in basketball and volleyball during his leisure time. He also enjoys smooth/gypsy jazz where you can find him on Wednesdays in his favorite place Green Mill.