CMU Artificial Intelligence Seminar Series sponsored by


Back to Seminar Schedule

Tuesday, April 5, 2022

Time: 12:00 - 01:00 PM ET
Recording of this Online Seminar on Youtube

Uri Alon -- Natural Language Reasoning with Language Models of Code

Relevant Paper(s):

  • N/A

Abstract: In this talk, I will show that LMs that were pretrained on *code* can be better natural language reasoners than LMs that were trained (mostly) on natural language, even when the task does not involve source code at all. In a class of structured NL reasoning tasks, I will show how we can frame the task as code generation; this makes LMs of code such as Codex better reasoners than LMs of natural language such as T5 and GPT-3. Another class of mathematical reasoning tasks was recently unlocked by methods that require LLMs to generate their explicit reasoning steps, such as "chain-of-thought" (Wei et al., 2022). Such methods employ LMs for both understanding the problem description by decomposing it into steps, as well as solving each step of the problem. While LMs seem adept at the step-by-step decomposition part, they often make logical and arithmetic mistakes in the solution part. I will show how LMs of *code* can decompose the natural language problem into runnable steps, which allows us to offload the solution to a programmatic runtime such as a Python interpreter. That is, instead of learning to solve the problem directly, we teach the model to generate a program that solves the problem. Across a variety of benchmarks, this approach leads to more accurate results than much larger models such as PALM-540B using chain-of-thought.

Bio: Uri Alon is a Postdoctoral Researcher at LTI, working with Prof. Graham Neubig on NLP and learning from source code. Previously, he obtained his PhD at the Technion (Israel), where he worked on modeling programming languages and graphs. Currently, he is also interested in the synergy of neural models with symbolic components such as retrieval, programs, and automata. His personal website is at https://urialon.ml. Feel free to reach out with any questions or comments about the talk.