| Scenario | Requests/Day | In Tokens | Out Tokens | Growth | Cache Hit | Retry |
|---|---|---|---|---|---|---|
| Customer support assistant | 25,000 | 650 | 220 | 8% | 12% | 1.5% |
| Internal analytics copilot | 6,000 | 1,250 | 300 | 4% | 18% | 2% |
| Developer coding helper | 12,000 | 900 | 450 | 10% | 9% | 3% |
RequestsMonth = RequestsDay × DaysPerMonthWithRetry = RequestsMonth × (1 + RetryRate)Effective = WithRetry × (1 − CacheHitRate)
InputTokens = Effective × AvgInputTokensOutputTokens = Effective × AvgOutputTokens
TokenCost = (InputTokens/1,000,000)×PriceIn + (OutputTokens/1,000,000)×PriceOutAPISubtotal = TokenCost + (Effective×PerRequestFee)APIAfterDiscount = APISubtotal×(1−Discount) + PlatformFee
TotalTokens = InputTokens + OutputTokensNeededTPS = (TotalTokens / SecondsPerMonth) / UtilizationDevices = ceil(NeededTPS / DeviceTokensPerSecond)
EgressGB = (Effective × AvgResponseKB) / (1024×1024)Total = (CoreCost + EgressCost) × (1 + SafetyMargin)
RequestsDay(m) = Base × (1 + GrowthRate)^(m−1).
- Enter your requests/day and realistic token averages from logs.
- Set retry and cache hit rates from observed behavior.
- Choose a projection horizon and monthly growth rate aligned to plans.
- Fill API pricing (per‑million token rates) and any fees or discounts.
- If self-hosting, enter measured tokens/sec per device, utilization target, and hourly costs.
- Press Calculate, then export CSV for spreadsheets or PDF for sharing.
Model tokens as the primary cost driver
Input and output tokens usually explain most spend. Start with log-based averages, then separate prompt growth from response verbosity. For example, 25,000 requests/day at 650 input and 220 output tokens produces about 661 million tokens/month before retries and cache. At $2.50 and $7.50 per 1M tokens, output pricing dominates when answers are long. Track median and p95 tokens to size your safety margin. In practice, teams review token histograms weekly and flag any 10% drift as a regression—an early signal.
Quantify retries and regeneration overhead
Retries convert small reliability issues into predictable cost multipliers. A 1.5% retry rate increases paid traffic by roughly 1.015×, but regeneration features can push this above 5% during rollouts. Treat timeouts, tool failures, and user “try again” clicks as the same class of overhead. If you reduce retry rate from 3% to 1%, a 12‑month budget drops by about 2% even with flat pricing.
Leverage caching and routing to cut spend
Caching lowers effective paid requests without changing user volume. A 20% cache hit rate reduces token billing and egress by 20%, which often beats micro-optimizing prompts. Route trivial queries to smaller models and reserve premium models for complex tasks; even a 30% mix shift can halve output-token costs. Measure cache hit rate per intent, not globally, because support and search flows behave differently.
Forecast growth and peak-month exposure
Growth compounds quickly, so plan for peak months, not only averages. An 8% monthly growth rate means month 12 is about 2.33× month 1, raising both spend and capacity needs. Add platform fees and egress early; fixed items look small in month 1 but matter at low volume. Use the table to identify the first month where peak cost exceeds your approval threshold.
Translate throughput into self-hosted capacity
Self-hosting depends on sustained tokens/second and utilization targets. Convert monthly tokens to average TPS: divide total tokens/month by seconds/month, then divide by utilization. If workloads average 252 tokens/s and you target 55% utilization, you must provision about 458 tokens/s. With 3,500 tokens/s per device, that is one device, but peaks may require several. Overhead and fixed ops costs often decide break-even.
Which token metric should I use: average, median, or p95?
Use medians for typical cost, p95 for worst‑case planning, and averages for finance rollups. If your traffic is bursty, p95 plus a safety margin better reflects real peak invoices and capacity needs.
How do I estimate cache hit rate before launching a cache?
Start with intent sampling. Identify repeatable queries, templates, and retrieval hits, then assume 5–15% initially. After launch, measure hits by endpoint and intent weekly, because a single feature can double the rate.
Why can output pricing dominate the bill?
Outputs often carry more tokens than prompts, and many providers price them higher. Longer answers, tool traces, and citations increase output tokens. Tight response formats and summarization can lower spend without reducing request volume.
What does utilization target mean for self-hosted planning?
It is the fraction of sustained throughput you are comfortable using. Lower targets create headroom for spikes and maintenance. A 55% target means you provision almost 1.82× the average tokens/second to protect latency.
How should I choose the safety margin percentage?
Set it to cover unknowns: token drift, growth forecast error, and performance variance. Teams often start at 10% and adjust after two billing cycles. If logs are noisy or product scope is changing, use 15–25%.
When is self-hosting typically cheaper than an API?
It tends to win when volume is high, traffic is steady, and you can run devices at healthy utilization. If fixed ops costs are large or workloads are spiky, APIs can be cheaper even with higher token rates.