Define the unit you are optimizing
Cost per million tokens is useful for infrastructure, but product economics should track cost per successful task. A cheaper response that fails and must be retried can cost more. Pair cost with a task-quality score, error rate and latency objective.
Separate prefill from decode, cached from uncached input, and steady traffic from bursts. This reveals whether the main problem is excessive context, an oversized model, poor batching, repeated prefixes or idle capacity.
Apply reversible changes before structural ones
Remove duplicated instructions and unused retrieved documents, cap output where the task permits, and cache deterministic work. Then route well-understood tasks to the smallest model that passes the same evaluation. These changes are easier to reverse than a new serving fleet or distillation program.
At the engine level, continuous batching increases utilization while prefix caching can reuse KV-cache work for shared prefixes. vLLM documents automatic prefix caching; TensorRT-LLM documents in-flight batching and KV-cache reuse. Gains depend on the workload, so observe queue delay and cache hit rate.
Quantize and buy capacity only with evidence
Quantization can reduce weight memory and increase effective throughput, but output quality can change by model and task. Run a fixed evaluation set at each candidate precision before moving production traffic. Preserve the baseline so savings are not confused with a quality regression.
Dedicated GPUs can lower unit cost when demand keeps them productively occupied. Compare against current token-priced spend with measured throughput and utilization, then add engineering, orchestration, monitoring and failure capacity. If the decision only works under perfect utilization, wait.