I know this community is focused on local models, but hear me out — this tool might actually help make the case for local inference better than any benchmark.

LLM Cost Profiler tracks every API call your code makes to OpenAI/Anthropic and shows you exactly what you're spending, where, and why. The interesting part for this community: it exposes which tasks are ludicrously overpriced relative to their complexity.

For example, in my own codebase it found:

• A classifier using GPT-4o that outputs one of 5 labels — a task any decent 7B local model handles easily. Cost: \~$89/week on API calls.
• Thousands of duplicate calls to the same prompt — zero caching. Local inference with caching would make this effectively free.
• A summarizer where 34% of calls were retries from format errors. A well-tuned local model with constrained generation eliminates this entire class of waste.

If you're trying to convince your team to invest in local inference infrastructure, this tool gives you the ammunition. "Here's the exact dollar amount we'd save by moving X task to a local model."

It's Python, MIT licensed, stores everything in local SQLite.

GitHub: https://github.com/BuildWithAbid/llm-cost-profiler

Planning to add support for tracking local model inference costs too (compute time based costing) — would that be useful to anyone here?