6
AI
How do you estimate what an LLM feature will cost before launch?
Context
Finance wants a monthly cost estimate for our new AI assistant before we launch, and my honest answer is "somewhere between $2K and $40K".
What I know: expected daily active users, rough messages per session, our prompt size. What I don't know: how long conversations really get, how often tools are called, and how much prompt caching will save.
How do you produce a number you can defend? Shadow traffic, a pilot cohort, a spreadsheet model? What surprised you after launch?
Community discussion
4 comments
Have you made this decision in production? Share your reasoning.
Sign in to commentPut a hard budget cap and per-user rate limit in before launch. Our worst surprise was one customer's integration looping on the API overnight.
Model it per conversation, not per message: system prompt + retrieved context + history grows with every turn. Our turn-10 messages cost 6x our turn-1 messages until we started summarising history.
Then run a pilot with 2% of users for two weeks and fit the model to reality. Our spreadsheet was off by 2.3x; the pilot was off by 15%.
The per-conversation framing is the missing piece. Our estimate assumed every message was a turn-1 message.
Track cost per successful outcome (resolved ticket, completed task), not cost per token. It is the only number finance and product both care about, and it tells you when a cheaper model is a false economy.