LLM inference glossary
18 terms buyers meet when evaluating LLM inference providers.
- Serverless inference
- Running a model on capacity shared with other customers and paying per token or per request, with no servers to manage.
- Dedicated endpoint
- A model deployment on hardware reserved for one customer, usually billed per GPU-minute or GPU-hour whether or not it is busy.
- Provisioned throughput
- Reserved model capacity bought in throughput units for a term, giving predictable capacity and cost.
- Bring your own model (BYOM)
- Uploading your own or fine-tuned weights, or a serving container, for the provider to run.
- OpenAI-compatible API
- An API that accepts the same request and response format as OpenAI’s, so existing OpenAI client code works by changing the base URL, key, and model name. Coverage of parameters varies.
- Open-weight model
- A model whose trained weights are published for download and self-hosting, under the model’s own license.
- Zero data retention (ZDR)
- A policy or setting under which prompts and outputs are not written to durable storage after the request completes. Exceptions for abuse monitoring or specific features are common.
- Data residency
- Where data is stored and processed geographically. Processing location can differ from storage location under global routing.
- Per-token pricing
- Billing by the number of input and output tokens, usually quoted per 1 million tokens, often with different prices for input, cached input, and output.
- Cached input tokens
- Input tokens that repeat a recent prompt prefix and are billed at a lower rate by providers that support prompt caching.
- Batch inference
- Submitting many requests to be processed asynchronously within a time window, often at a discount to on-demand prices.
- Aggregator (model router)
- A service that exposes one API and forwards each request to one of many third-party providers.
- BYOK (bring your own key)
- Using your own provider API key or encryption key through another platform, so requests run under your account or keys.
- LoRA adapter
- A small set of fine-tuned weights applied on top of a base model, which some providers can serve without a full custom deployment.
- Scale to zero
- Automatically shutting down a deployment when it has no traffic, so you stop paying for idle GPUs at the cost of a cold start later.
- BAA (business associate agreement)
- A HIPAA contract required before a vendor handles protected health information for a covered entity.
- SOC 2 Type II
- An independent auditor’s report on how a company’s security controls operated over a period. Type I covers design at a point in time.
- ISO/IEC 27001
- An international certification for an information security management system.