Every time you ask a chatbot a question, accept a code suggestion in your editor, or hear a voice assistant answer in full sentences, a large language model (LLM) is doing one deceptively simple thing: predicting the next token, then the next, then the next. Teaching a model to make good predictions is training: weeks of feeding it trillions of tokens on thousands of GPUs. It happens once per model version, it's measured in days and dollars, and nobody is waiting on the other side of the screen. Putting the trained model in front of users is inference, a different problem entirely: it happens billions of times a day, each request is measured in milliseconds, and someone is always waiting.

Serving a model sounds simple (load the weights, run the forward pass) until you try it at scale.

• Every request needs GPU memory to remember the conversation so far (the KV cache).
• Thousands of requests arrive at once, with prompts of wildly different lengths.
• The GPU sits mostly idle unless you batch them cleverly.

A whole ecosystem of inference engines has grown up to solve exactly these problems.

• vLLM made memory management efficient with PagedAttention.
• SGLang made it cheap to reuse shared prompt prefixes with RadixAttention.
• NVIDIA's TensorRT-LLM squeezes every cycle out of NVIDIA hardware.
• llama.cpp brought LLMs to laptops and commodity hardware.

On top of these engines sit orchestrators that scale them across clusters, and on top of those, a crowded market of managed services that run the whole thing for you. Whether you build your own stack or buy one from a vendor, you face the same question: which option is good enough for my users? And what does “good enough” even mean?

Many teams never run an engine themselves; they buy one, wrapped in a service. The market looks like a pyramid, and every step up trades control for convenience.

Teams with deep engineering bandwidth build near the base; teams without that bandwidth climb toward the top. Either way, each step up hands one more decision to someone else, including how fast your users' tokens arrive. So how do you pick a level? It comes back to what “good enough” means.

Ask a product team what they need and you'll hear “feelings”. The chatbot should feel responsive. The voice agent should feel smooth, never breaking mid-sentence. The code completion should feel crisp and fast. Those feelings are real, but you can't hold a vendor to a feeling, and you can't tell whether last week's release made things better or worse. Lord Kelvin said it best, in an 1883 lecture:

“When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.”

It's usually paraphrased as what can't be measured can't be improved. So the first job is translation.

• “Responsive” becomes the time until the first token appears.
• “Smooth” becomes the gap between one token and the next.
• “Crisp” becomes the total time to finish a short completion.

Once a feeling becomes a number, three questions follow, and they make up the rest of this post:

1. Measure (SLI): what number do we watch?
2. Promise (SLO): what value of it do we commit to, and for how many requests?
3. Contract (SLA): what happens when we break that promise?

The three answers nest inside one another, like rings.

Each ring has its own tribe.

• Academics live in the innermost one: a systems paper picks an SLI, builds a smarter scheduler, and shows the number moving the right way. When it states a promise at all, it's often borrowed from another paper and tested under traffic the authors chose.
• Serving teams live in the middle: they decide which SLIs matter and what to promise. Too often, that means whatever the dashboard already tracks, an average or a p50, rather than what a user actually feels.
• Management lives on the outside, where account teams, lawyers and finance turn promises into contracts. The remedy is usually a capped service credit, and the fine print defining the SLI is rarely what anyone negotiated over.

Each tribe tends to treat the other rings as someone else's problem. They aren't. An optimization that improves an SLI nobody promised is a paper, not a product. A contract written on an SLI nobody measures is a promise waiting to be broken. And as we'll see, LLM SLAs cover whether the service is up, not whether it's fast, and for a better reason than you might think.

A service level indicator (SLI) is simply the number you watch. In LLM serving, that number has been redefined twice in about two years, and each time the old one turned out to hide something.

• Latency vs. Throughput (2022–2023). Orca (OSDI 2022) and vLLM (SOSP 2023) judged a serving system by throughput and by normalized latency: each request's end-to-end time divided by the number of tokens it generated, averaged over all requests. It's a fine number for comparing schedulers, but a request that waits five seconds for its first word and then streams quickly can average out to look healthy.
• Towards Disaggregated Inference (2024). Splitwise (ISCA 2024) showed that a request has two phases with different bottlenecks. The prompt is processed in one compute-heavy burst (prefill), then the answer is generated one token at a time (decode), limited by memory bandwidth. So it measured them separately: time to first token (TTFT) for prefill, and time between tokens (TBT) for decode. DistServe (OSDI 2024) used the same split, with the average time per output token (TPOT) for decode.
• Watch the tail, not the mean (2024). Sarathi-Serve (OSDI 2024) showed why the average still hides the pain: a request can have a healthy TPOT and still freeze for seconds mid-answer when a long new prompt gets scheduled alongside it. Those generation stalls only show up in the distribution of token gaps, so it targeted P99 TBT. Today's engines and benchmark tools mostly call this inter-token latency (ITL).

Which of these numbers matters depends on how the output is consumed: streamed to someone who is watching or listening, or used only once it's complete. Each one is a distribution, not a single number:

Notice what's missing from both rows: a target. “TTFT” is a measurement; “TTFT under 500 ms for 95% of requests” is a promise. That's the next ring.

A service level objective (SLO) turns a measurement into a promise. A complete one answers five questions, and you can read all five off a single sentence:

“95% of chat requests see their first token within 500 ms, measured over each rolling 28-day window.”

• What is measured? Time to first token: the SLI.
• How fast? 500 ms: the threshold.
• Out of how many? 95% of requests: the attainment target. That's the same statement as “p95 TTFT ≤ 500 ms”.We usually represent an attainment target using percentiles.
• Which requests? Chat requests
• Over what period? A rolling 28 day window.

Drop any one of these and the promise quietly turns into a wish. In papers, product pages and design docs alike, the same shortcuts keep showing up:

• No percentile. “TTFT under 400 ms” could mean the average, the median or every single request, and those describe very different systems.
• An average. A healthy mean TPOT can sit on top of multi-second stalls, as the previous section showed.
• One number for every workload. A threshold that suits chat is far too strict for summarization and far too loose for code completion.
• No window, no traffic. A promise kept during one quiet benchmark run says little about Monday morning.

Goodput, introduced by DistServe (OSDI 2024), is what you get when you take the promise seriously. Throughput counts every token a system produces; goodput counts only the work that kept its promise. DistServe defines it as “the maximum request rate that can be served adhering to the SLO attainment goal (say, 90%) for each GPU provisioned”, and it sets a different promise for each workload:

That one change flips what gets optimized. A system can push throughput up by packing ever-larger batches while every user waits longer for each token; goodput punishes exactly that. It's also the number a serving team actually needs for capacity planning: how many requests per GPU can we serve while keeping our promise?

A promise you can measure is an SLO. A promise that costs you something when you break it is an SLA. That's the outer ring.

A service level agreement (SLA) is an SLO with a price tag attached. It's the ring most engineers never read, because it lives on a legal page rather than a dashboard. Almost every SLA has four parts:

• The SLI. Errors, not milliseconds: the share of requests that fail with a server error or never complete.
• The target. A monthly uptime percentage, usually 99-point-something.
• The remedy. A service credit: a slice of next month's bill, paid out in tiers and capped. You get a discount, not your users' patience back.
• The exclusions. Traffic beyond the capacity you reserved, throttling and quota errors, preview features, and anything caused by “customer actions”. Bursts above your reservation are often carved out, which is exactly when you'd need the promise most.

Notice what's missing: speed. That's not an oversight. A vendor sees your requests, not your product, so it can't know what “fast enough” means for your users. The latency promise is yours to write, on top of the vendor's availability promise.

My hope is that you now have a working picture of how a system in production is measured, and how users hold service providers accountable, using a framework of three nested rings: the measurement (SLI), the promise (SLO) and the contract (SLA). The framework is widely used in practice, but the devil is in the details. Every SLO is conditioned on a variable that rarely reaches its audience: papers do state it, usually in one line of the evaluation section, but it seldom travels with the headline number in the abstract. What isn't revealed becomes an assumption, and we all know the dangers of partial knowledge. The variable in question is the request arrival pattern (queueing theory calls it the arrival process): when requests show up, not just how many arrive per second on average. vLLM, Splitwise and DistServe all benchmark with Poisson arrivals, requests arriving independently at a steady rate, which ServeGen calls the "de facto approach". Yet when Alibaba characterized its production LLM traffic in ServeGen (2025), it found arrivals "often bursty (CV > 1)", and that "Poisson processes (CV=1) often poorly model bursty workloads".

In the next article, I'll motivate why this variable matters: how the literature models incoming traffic, how that differs from real-world traffic, and which models (Gamma and Weibull processes, for example) capture it better. We'll also see how changing this single variable, with the same model, hardware and average rate, can turn a feasible SLO into an infeasible one, and with it the SLA built on top.

• Thomson, W. (Lord Kelvin). Electrical Units of Measurement. Lecture at the Institution of Civil Engineers, 3 May 1883; in Popular Lectures and Addresses, Vol. 1, 1889.
• Yu et al. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022.
• Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.
• Patel et al. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. ISCA 2024.
• Zhong et al. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024.
• Agrawal et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024.
• Xiang et al. ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production. arXiv:2505.09999, 2025.
• Beyer et al. (eds.). Site Reliability Engineering, Chapter 4: Service Level Objectives. O'Reilly, 2016.
• Beyer et al. (eds.). The Site Reliability Workbook, Chapter 2: Implementing SLOs. O'Reilly, 2018.
• Amazon Web Services. Amazon Compute Service Level Agreement.
• Google Cloud. Cloud Storage Service Level Agreement.