CMU Artificial Intelligence Seminar Series sponsored by Fortive
Tuesday, May 11, 2021
Time: 12:00 - 01:00 PM ET
Jon Kleinberg -- Aligning Superhuman AI with Human Behavior: Chess as a Model System
Relevant Paper(s):
- Assessing Human Error Against a Benchmark of Perfection
- Aligning Superhuman AI and Human Behavior: Chess as a Model System
- Learning Personalized Models of Human Behavior in Chess
Abstract: In domains where AI systems have achieved superhuman performance, there is an opportunity to study the similarities and contrasts between human and AI behaviors at the level of granular actions, not just aggregate performance. Such an analysis can yield several potential sources of insight. First, by studying expert-level human performance through the lens of systems that far surpass this performance, we can try to characterize the settings in which human errors are most likely to occur. Second, we can try to adapt high-performing AI systems to match human behavior as closely as possible at an action-by-action level,
We pursue these goals in a domain with a long history in AI: chess. For our purposes, chess provides a domain with many different levels of human expertise, together with data from hundreds of millions of online games that each record a specific decision together with its context. However, applying existing chess engines to this data, including an open-source implementation of AlphaZero, we find that they do not predict human moves well.
We develop new methods for predicting human decisions at a move-by-move level much more accurately than existing engines, and in a way that is tunable to fine-grained differences in human skill. From this, we discover aspects of chess positions that serve as predictors of human error, as well as algorithms that are able to operate in this domain in a more "human-like" way. One of our algorithms, the Maia chess engine, can be tried at lichess.org, where it has played over 300,000 games to date with users of the platform.
This is joint work with Ashton Anderson, Reid McIlroy-Young, Sendhil Mullainathan, Siddhartha Sen, and Russell Wang.
Bio: Jon Kleinberg is the Tisch University Professor in the Departments of Computer Science and Information Science at Cornell University. His research focuses on the interaction of algorithms and networks, the roles they play in large-scale social and information systems, and their broader societal implications. He is a member of the National Academy of Sciences and the National Academy of Engineering, and the recipient of MacArthur, Packard, Simons, Sloan, and Vannevar Bush research fellowships, as well awards including the Harvey Prize, the Nevanlinna Prize, and the ACM Prize in Computing.