LLMs are advanced AI for tasks like text and code generation. As LLMs scale towards being distributed, efficiently serving LLM requires multiple hardware units to work together. To manage how LLMs process user requests across these resources, developers use different strategies —called orchestration patterns— to handle user requests, such as how to split tasks, manage data, and route requests.

However, Most current systems use “coarse-grained” APIs. This means they offer general, high-level commands (like “process this entire request”) and use a “pre-configured” orchestration pattern that’s set in advance and hard to change without system restarts. This “one-size-fits-all” approach is inflexible, often leading to slower responses, reduced capacity, and wasted resources because different situations benefit from different orchestration patterns. Developers can’t easily customize or adapt these hidden, built-in strategies.

LLM MicroServing introduces “fine-grained” APIs. These allow for very specific, detailed commands, giving developers precise control over how requests are processed. Inspired by RISC computer design (simple instructions combining for complex tasks) versus older CISC design (one complex command), Microserving uses a programmable router to break down requests into smaller, manageable steps. This allows dynamic adaptation and customization of orchestration patterns for better speed and efficiency.

In this blog post, we will discuss:

• Current distributed serving systems and their limitations.

• MicroServing APIs: their meaning and how to build orchestration patterns with them.

• How to enable real-time adjustments to orchestration patterns with Microserving API.

• The performance advantages of using MicroServing’s adaptive approach.

Figure 1. Microserving Overview. By translating coarse-grained API calls to fine-grained microserving API with a programmable router, we can compose various orchestration patterns and support dynamic reconfiguration.

Background: LLM Serving Orchestration

Making Large-Language Models (LLMs) available efficiently is challenging, often involving multiple hardware units. Managing how user requests are processed across these disparate resources is key to performance.

At the core of this process is the LLM engine—the software system responsible for running the LLM, executing its complex calculations, and generating responses. Understanding how these engines are utilized in different orchestration patterns helps clarify the challenges MicroServing addresses:

1. Data Parallel: This is a straightforward pattern where incoming user requests are distributed across different LLM engines, often in a round-robin fashion (cycling through the available engines). While simple to implement, it may not always be the most efficient, especially if different requests have varying computational needs or if engines have different current loads or specialized data.

2. Prefill-Decode Disaggregation: This more advanced strategy separates the LLM’s work into two main phases, often assigning them to different, specialized engines:

• Prefill: This is the initial, computationally intensive processing of the user’s input prompt. During this stage, the model works to understand the input and prepare for generating a response.

• Decode: This is the subsequent, step-by-step generation of the output text, token by token.

To make this separation effective, crucial information must be transferred between the stages. This involves:

• KV Cache: LLMs use an attention mechanism that computes “Key” and “Value” states for each token (piece of text). These states are stored in a KV cache to avoid redundant calculations for already processed parts of the input during the decode phase, significantly speeding it up.

• KV Transfer: To enable a separate decode engine to take over, the KV cache generated by the prefill engine for the initial prompt is transferred to the decode engine. This allows the decode engine to start generating the response without reprocessing the original input.

Figure 2. Existing approaches configure orchestration patterns in startup time, which involves restarting the system to adjust orchestration strategies, while microserving allows dynamic reconfiguration

Each orchestration pattern has its strengths and weaknesses depending on the workload. For instance, prefill-decode disaggregation might perform well for requests with moderate input lengths but could be less efficient for very short or very long inputs where one set of specialized engines (prefill or decode) might be overwhelmed while the other is underutilized.

Ideally, an LLM serving system could dynamically switch between different orchestration patterns as the characteristics of incoming requests change. However, as highlighted in the overview, existing LLM serving systems often treat the processing of a request as a single, indivisible operation and bake the scheduling logic deep into the system. This leads to two significant problems:

1. Development Rigidity: Developers building services on top of these systems find it hard to customize or experiment with different orchestration patterns without making complex changes to the underlying framework code.

2. Production Inflexibility: Adjusting orchestration strategies in a live production environment typically requires restarting the system. This causes service interruptions and complicates the management of deployments.

Our approach: MicroServing APIs

MicroServing addresses this limitation by exposing three simple fine-grained APIs that allow precise control over system operations, such as transferring key-value (KV) data between engines and initiating token generation with existing context KV. In addition, the APIs are fully context-cache aware. With these fine-grained APIs, dynamic orchestration patterns can be implemented easily in just a few lines of code.

Core MicroServing APIs

class LLMEngine:

    def prep_recv(prompt, end, seq_id) -> Tuple[KVAddr, int]:
        """
        One-line summary: Prepare KV entry allocation for receiving from remote.

        Detailed explanation: Match prompt[:end] in the context cache of this
        engine to get the matched prefix length matched_len.
        Allocate KV entry for prompt[:end] to prepare receiving the KV data of
        prompt[matched_len:end] from another engine
        Return KV address and matched_len.
        """
        pass

    def remote_send(prompt, kv_addr_info, recv_rank, begin, end, seq_id):
        """
        One-line summary: Send KV to remote

        Detailed explanation: Generate KV of prompt[begin:end] on this engine
        by matching in context cache and prefilling.
        Transfer the KV of prompt[begin:end] to the address encoded in
        kv_addr_info on engine recv_rank.
        Return when all KV transfers are finished.
        """
        pass

    def start_generate(prompt, begin, seq_id) -> AsyncGenerator[int]:
        """
        One-line summary: Start token generation

        Detailed explanation: Prefill prompt[begin:] on this engine and start
        decode on this engine.
        Return generated tokens.
        """
        pass

Note that the APIs are independent of the underlying engine implementation. For example, an engine may decide to pipeline the LLM inference and KV cache transfer layer-by-layer.

Also, the microserving endpoints can be used to run on all RPC style APIs, including REST.

How to compose orchestration patterns with MicroServing APIs?

With MicroServing APIs, users can implement different orchestration patterns on a programmable router. Here is how patterns can be implemented:

Data parallel (round-robin)

Since no KV transfer is required, we only need to call the start_generate API.

Figure 3. Data parallel via microserving API

Prefill-decode disaggregation

The sequence of API calls is as follows:

1. prep_recv (decode engine): Prepares the decode engine to receive KV entries.

2. remote_send (prefill engine): Sends KV entries from the prefill engine.

3. start_generate (decode engine): Initiates token generation on the decode engine.

Figure 4. Prefill-decode disaggregation via microserving API

With MicroServing’s context-cache-aware design, this pattern can further utilize context cache. For example, prep_recv returns the matched prefix length on the decode engine. The matched prefix length is then passed to the prefill engine to avoid redundant KV transfer and computation.

Figure 5. Context-aware prefill-decode disaggregation via microserving API

Balanced prefill-decode disaggregation

One issue of prefill-decode disaggregation is that the prefill and decode workloads can get unbalanced. When processing long prompts, the prefill engine can get over-utilized while the decode engine runs with low utilization or even stays idle.

Thanks to the fine-grained MicroServing API, we can explore a different strategy that dynamically offloads a part of the prefill computation into the decode engine. To achieve this, the router needs to decide decode_start (the position that the decode engine starts to prefill) and pass it into all the APIs like below.

Figure 6. Balanced prefill-decode disaggregation via microserving API

Context Cache Migration

When serving QA workloads, developers tend to place the context cache of different categories into different engines and dispatch incoming traffic based on which context category it matches. Consider there are several engines, with some specialized for the history context and others for the science context. If there are more science requests than history requests, we may want to switch some history engines to science engines through context migration, and vice versa.

MicroServing supports this via prep_recv and remote_send , enabling efficient KV transfers between engines without interrupting service.

Figure 7. Context migration via microserving API

Dynamic Reconfiguration: Adapting to Workload Changes

Another advantage of MicroServing’s flexible fine-grained API is its ability to dynamically reconfigure orchestration patterns based on workload characteristics, without changes to the underlying engines. The programmable router allows seamless switching between different orchestration strategies. For example:

• When the Prefill:Decode ratio (time to prefill the input tokens / total time to decode all the output tokens) increases, more prefill computation can be shifted to the decode engine by balanced prefill-decode disaggregation.

• If most of the prompt data is found in the decode engine’s context cache, the system can bypass the prefill engine entirely, directly calling start_generate to process the non-cached part of the prompt.

This adaptability ensures that MicroServing can optimize performance for different traffic patterns.

The router-side code to reconfigure orchestration pattern can be like below:

Figure 8. Supporting various orchestration patterns and dynamic reconfiguration via microserving API

In comparison, other systems often implement different orchestration patterns in separate codepaths, which makes it hard to customize new orchestration patterns, and often requires system restart to reconfigure patterns.

Evaluation

We evaluate MicroServe’s programmability for various LLM engine orchestration patterns and the performance of each pattern. The benchmarks are performed with Llama-3.1-70B-Instruct FP8 with 8 H100 SXM GPUs. For each model, we start 2 LLM engines, each of which has tensor parallel degree 4. The engine orchestration patterns evaluated are:

• Data parallel

• Prefill-decode disaggregation

• Balanced prefill-decode disaggregation with offload ratio 0.1 (the percentage of prefill moved to decode engine)

Although MicroServing cannot dynamically reconfigure tensor parallel degree because of weight resharding issue, we benchmark one LLM engine with tensor parallel degree 8 to serve as a strong baseline in low request rate cases.

We use LLMPerf ’s synthetic sonnet dataset to construct requests with an average input length of 3000 and an average output length of 100. We evaluate fixed request rates ranging from 1.6 to 5.6 requests/s. The figures will be shown with the x-axis representing the request rate (requests/s) and the y-axis representing TTFT(time to first token)/TPOT(time per output token, the average time per output token after the first token is received)/JCT(job completion time).

Figure 9. Evaluation of different orchestration patterns on Llama 3.1 70B

In both figures above, we find that the optimal orchestration pattern under different disaggregation patterns shifts. Using one engine with a larger tensor parallel degree achieves up to 8% lower mean JCT than other strategies when the request rate is 1.6 requests/s, but it scales poorly as the request rate increases. Prefill-decode disaggregation achieves up to 16% lower mean JCT and 28% P99 JCT than data parallel. This significant speedup attributes to the substantial reduction of TPOT in disaggregation, achieved by eliminating the decode interference caused by long prefill in data parallelism. As the request rate increases, heavier traffic puts more pressure on the prefill engine, so moving part of the prefill computation to decode the engine makes the system balanced and reduces latency.

The evaluation shows that different orchestration patterns have different preferences on the traffic. Some have lower TTFT, which performs better on prefill-dominated workloads. Others have lower TPOT, more suitable on decode-dominated workloads. MicroServing provides microserving APIs and programmability, supporting the representation of various orchestration patterns and the dynamic reconfiguration among these patterns.

Conclusion: Unlocking New Possibilities with MicroServing

MicroServing provides an efficient and flexible framework for orchestrating LLM engines, enabling dynamic adaptation to varying workloads. Its fine-grained APIs allow easy composition of orchestration patterns, from simple round-robin dispatch to complex prefill-decode disaggregation, in only a few lines of router code. MicroServing’s flexibility ensures that developers can optimize their systems without needing to reconfigure engines or disrupt production environments.

By integrating MicroServing with MLC-LLM , our research group is opening up exciting opportunities for the community to experiment with and improve LLM orchestration patterns. We look forward to collaborating with others to refine dynamic adaptive reconfiguration algorithms and expand the library of orchestration patterns supported by MicroServing.