CMU Artificial Intelligence Seminar Series sponsored by


Back to Seminar Schedule

Tuesday, April 5, 2022

Time: 12:00 - 01:00 PM ET
Recording of this Online Seminar on Youtube

Uri Alon -- Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval

Relevant Paper(s):

Abstract: Retrieval-based language models (R-LM) model the probability of natural language text by combining a standard language model (LM) with examples retrieved from an external datastore at test time. While effective, a major bottleneck of using these models in practice is the computationally costly datastore search, which can be performed as frequently as every time step. In this talk, I will present RetoMaton - retrieval automaton - which approximates the datastore search, based on (1) clustering of entries into "states", and (2) state transitions from previous entries. This effectively results in a weighted finite automaton built on top of the datastore, instead of representing the datastore as a flat list. The creation of the automaton is unsupervised, and a RetoMaton can be constructed from any text collection: either the original training corpus or from another domain. Traversing this automaton at inference time, in parallel to the LM inference, reduces its perplexity, or alternatively saves up to 83% of the nearest neighbor searches over kNN-LM (Khandelwal et al., 2020), without hurting perplexity.

Bio: Uri Alon is a Postdoctoral Researcher at LTI, working with Prof. Graham Neubig on NLP and learning from source code. Previously, he obtained his PhD at the Technion (Israel), where he worked on modeling programming languages and graphs. Currently, he is also interested in the synergy of neural models with symbolic components such as retrieval, programs, and automata. His personal website is at https://urialon.ml. Feel free to reach out with any questions or comments about the talk.