The relevant cost is the total effort required to produce a usable result at the promised quality.
Classify work before routing it
Some tasks are narrow and easy to verify; others require more context or judgement. Identify those differences using representative customer work. A routing system should be based on observed performance, not the assumption that every short prompt is easy. A brief instruction can contain an important exception, while a long structured task may have a straightforward answer.
Count the whole path
Imagine a hypothetical lower-cost model succeeds on an initial attempt but frequently requires a retry or a stronger model to finish. Its nominal price does not describe the total cost of the workflow. Include all attempts, additional latency and any human review. Compare the cost per accepted result under the same quality criteria, rather than comparing provider prices in isolation.
Route around the completed task
Classify
Understand task difficulty and risk.
Attempt
Measure actual success and latency.
Escalate
Use explicit fallback conditions.
Compare
Calculate total cost per accepted result.
Make escalation deliberate
Define the conditions that trigger another model or a human decision. These might include missing evidence, a failed validation or a task outside the supported category. Avoid endlessly retrying the same failing approach. Give the user a useful explanation when the system cannot finish within its operating limits, and retain enough context that the next step does not need to reconstruct the task.
Watch for changing workloads
Review performance by task type as customer usage evolves. A route that worked for early adopters may be unsuitable for a new customer segment or document format. Keep a stable evaluation set and inspect real failures. Routing is a continuing product decision: the objective is to preserve the customer’s quality expectation while using resources sensibly, not to make a cost chart improve by quietly accepting worse results.
Sources
- Anthropic — source and reference material ↗ Anthropic · Primary source
Have additional information? Request a correction.
