Can you pick the perfect LLM without breaking the bank?

Adaptive LLM Routing under Budget Constraints

Large language models have transformed natural language processing, but their varying capabilities and costs create significant challenges for practical deployment. Consider a customer service chatbot handling diverse queries. Simple questions like "What are your business hours?" can be handled effectively by smaller, cost-effective models. However, complex inquiries such as detailed product comparisons requiring excellent reasoning and planning capabilities demand more powerful—and expensive—models.

This scenario illustrates the fundamental challenge: balancing performance with cost-effectiveness across varying query complexities. Existing approaches treat LLM routing as a supervised learning problem, assuming complete knowledge of optimal query-LLM pairings. These methods face two critical limitations: gathering labeled datasets requires expensive responses from each model for every query, and they lack adaptability to evolving query distributions.

Traditional routing strategies for LLMs rely heavily on supervised learning with full knowledge of optimal pairings, making them impractical for real-world deployment scenarios where comprehensive mappings are unavailable and user queries constantly evolve.