Helpful AI Models: You can't always get what you want, but you might get what you need by Prof Jordan Boyd-Graber

Abstract

AIs are trained in many ways, depending on the application. For example: on specific tasks, the goal is to maximize accuracy; with "general purpose" LLMs, the goal is to give users answers they want.

This talk argues that the focus should be slightly different: we should specifically measure human-computer workflows and optimize the ability of the combined team at that task. I'll discuss three different examples of human-computer teams that our group has explored: learning new vocabulary, strategic negotiation, and identifying false claims. For learning new vocabulary, we adapt alignment tuning to combine what looks helpful and is truly helpful into a flashcard scheduler that can improve the overall quality of study aids from our QA system. For strategic negotiation, we have computer agents help humans play a board game called Diplomacy to assist human players think strategically and detect lies, which we capture using an analysis of grounded statements with abstract meaning representation and value functions. Finally, we show that computers can help humans identify false statements — but only when the computer is not confidently incorrect. I'll then close with a discussion of how these questions lead into a broader discussion of human skill vs. computer skill, how to measure that, and on what datasets.

Biography

Jordan Boyd-Graber is a full professor at the University of Maryland. He has worked on model evaluations for human-centered topic models, psychologically inspired leaderboards, human-computer machine translation, and question answering. Of his twenty former PhD students, five have gone on to tenure track positions. He and his students have been recognized with paper awards at EMNLP (2023), IUI (2018), NAACL (2016), and NeurIPS (2009, 2015), and he won the 2015 Karen Spärk Jones Award and a 2017 NSF CAREER Award.

He served as PC for ACL 2023, SAC for EMNLP and NAACL, AC for ACL, NAACL, EMNLP, and NeurIPS, Poster Chair for EMNLP 2022, Tutorial Chair for ACL 2017, and Advisor for the ACL 2014 SRW.

He previously was an assistant professor at the University of Colorado, Visiting Research Scientist at Google Zürich, and Praktikant at the Berlin-Brandenburg Akademie der Wissenschaften. His undergraduate degrees are in Computer Science and History at the California Institute of Technology, and he received his PhD from Princeton University.