LLMs are advanced AI for tasks like text and code generation. As LLMs scale towards being distributed, efficiently serving LLM requires multiple hardware units to work together. To manage how LLMs process user requests across these resources, developers use different strategies —…
Modern machine learning (ML) kernels evolve rapidly with new optimizations and hardware features. ML compilers must accommodate richer memory hierarchies, more forms of thread cooperation, and backends with different execution models, while retaining the control needed for peak p…
The allocation of scarce donor organs constitutes one of the most consequential algorithmic problems in healthcare. Organ transplantation remains in many cases the only viable option for critically ill patients. Unfortunately, demand far outweighs existing supply: in the United …
Long-context understanding is a critical frontier for large language models (LLMs). Many of the most valuable applications, such as code generation, processing entire books, and modeling DNA sequences, either require long-context understanding to work well or become dramatically …
This post is based on our write-up Model self-identification could be subliminally transferred. Our implementation is provided in the accompanying repository.
Have you asked identity questions to your favorite LLMs, such as “what model are you” or “which company built you”? The a…
It is 2040, and you’re a hungry, busy PhD student in your office who has to attend a talk in thirty minutes. You place a DoorDash order from your favorite restaurant. An autonomous robot picks up your order from a restaurant in a mall. It then drives itself from the mall to your …
Randomness is a fundamental resource used in computer science for a variety of applications, including stochastic simulation, statistical modeling, AI, machine learning, and cryptography.
A prominent application of random sampling is text generation with large language models, wh…
Ask a modern image generator for a picture of an arctic fox, and behind the scenes it will start from pure random noise and repeatedly nudge that noise toward an image — often running a large neural network 50, 100, or even 250 times in sequence before a single picture emerges. T…
Suppose a hospital runs a statistical analysis over millions of patient records to determine whether a drug is effective. The computation takes days of cluster time, and the raw data cannot be shared due to privacy regulations. A regulatory agency needs to trust the result—but …
We often think about improving application performance by speeding up some part of the application itself. This can be done in many ways, such as optimizing the application code, using a more efficient algorithm, or even offloading parts of the application to specialized hardware. But a lot of the overheads that applications experience today are instead imposed by the underlying hardware interface. In this blog post, we will look at how the interface exposed by existing network interface cards (NICs) imposes significant overheads on modern applications and how a new NIC interface called Ensō can help eliminate these overheads.
