Understanding inference cost per token
Research6 mins read
Introduction
Usage-based pricing makes costs visible, which makes them worth understanding. Three levers dominate the bill.
The three levers
Smaller fine-tuned models cut cost per token; shorter prompts cut tokens per request; and batching raises throughput on the same hardware.
