Legal decisions carry heavy consequences for a person’s liberty and livelihood,
yet language models are increasingly trusted to reason over legal text. The danger is not
the obvious mistake but the silent one: a fluent, confident answer that is wrong, slips
past review, and surfaces as a missed argument or inadequate representation. That cost
is borne by the client, not the model. We ask not how accurately small, open-weight
models identify legal holdings, but where their competence breaks down. Using
CaseHOLD, we introduce a model-free measure of question difficulty from the semantic
geometry of the answer choices, and use it to locate a collapse point, the threshold
where accuracy falls to random guessing. We assess retrieval and lightweight fine-
tuning against this measure, yielding a difficulty-rated framework that shows where a
legal model stops reasoning and begins guessing.
Watch the team present this project at 01:38:43 in the session recording here.
Keywords: Legal reasoning, CaseHOLD, Semantic difficulty estimation, Language model
failure thresholds
Faculty Advisor
Batu Gundogdu holds a Ph.D. in Electrical Engineering and is an Associate Clinical Professor at the University of Chicago, bringing over 15 years of experience developing deep-learning models across the fields of speech, language, and medical imaging. Most recently, he served as Head of AI Engineering at Eva, an AI compute startup, building algorithms for a novel next-generation AI hardware.
Previously, Dr. Gundogdu served as Research Faculty in the Department of Radiology at the University of Chicago for four years, where he led AI efforts at the MRI Research Center.
In the UChicago MS in Applied Data Science program, he shares his practical insights and experiences with students on being an AI engineer and scientist across diverse venues, including research institutions (UChicago MRI Research Center), corporate companies (Analog Devices – ADI), government (Navy and NATO), and startups (Eva). He teaches courses on Bayesian Machine Learning for Generative AI and Deep Learning.
Dr. Gundogdu also teaches in UChicago’s MS in Applied Artificial Intelligence program, where he is a founding faculty member. Drawing on his experience across research, industry, and startups, he helps students connect advanced AI concepts with the realities of building and deploying AI systems in practice. He is also a Co-Founder of VicVox, a boutique AI startup analyzing social discourse.