Leaderboards Do Not Answer the Question. Your Evaluation Set Does.
Every model release arrives with a chart showing it beating the previous leader on a public benchmark. None of those benchmarks contain your documents, your customers' phrasing or your tolerance for a wrong answer. A model that tops a reasoning leaderboard can still misread the invoice format your finance team uses, and a mid table model can handle it perfectly at a fraction of the cost.
The only number that matters is how a model performs on a representative sample of your own workload, measured the same way every time. With that in hand, model selection stops being a debate and becomes a procurement decision.
Start With the Task, Not the Model
Production LLM workloads fall into a handful of task types, each stressing a model differently. Name yours before you look at a vendor page.
- Extraction. Pulling structured fields out of invoices, contracts or clinical notes. Per field precision matters more than fluency. Mid tier and small models often do this well when the schema is tight.
- Classification. Routing, tagging, intent. High volume, latency sensitive, easy to measure. Small models earn their keep here.
- Generation. Drafting replies, summaries, reports. Quality is partly subjective, so evaluation needs rubrics and a strong grader.
- Reasoning. Multi step analysis and planning under several constraints. Frontier models justify their cost here.
- Agentic tool use. The model chooses functions and arguments over several turns. Call format reliability and error recovery dominate.
- Code. Generating, reviewing or migrating code. Correctness is testable, which makes this the easiest category to evaluate.
Most real products mix several of these. A support flow classifies the ticket, extracts account details, reasons about policy and generates a reply. Treat each step as its own selection decision. Few systems need the strongest model at every step.
The Eight Dimensions That Drive the Decision
1. Latency and throughput
A voice agent needs its first token in well under a second. A nightly document pipeline does not care. Larger models are slower and costlier at high concurrency. Decide whether the workload is interactive or batch, and check whether the provider discounts batch processing.
2. Cost per task at volume
Do not compare per token prices. Compare cost per completed task, including retries, tool call round trips and the long system prompt you will inevitably grow, multiplied by monthly volume. A model that costs three times more per token but finishes in one attempt can beat a cheaper one that needs two retries.
3. Context window and long documents
Advertised context and usable context are different things. Many models degrade on retrieval from the middle of very long inputs. Test at your real document lengths. Often the better architecture is chunking plus retrieval rather than one enormous prompt.
4. Tool calling reliability
For agents, measure how often the model produces a schema valid call, picks the right tool and recovers sensibly from a tool error. A model that hallucinates a parameter one time in fifty will break a production workflow daily. Frontier tiers are meaningfully more reliable here, and that gap is worth paying for.
5. Multimodal needs
Scanned documents, screenshots and audio need either a natively multimodal model or a separate OCR and speech layer feeding a text model. Native is simpler; layered gives more cost control. Test both on your samples.
6. Data residency and privacy
There are three deployment shapes. Direct vendor APIs are fastest to start and get new models first. The same models hosted through a major cloud provider keep traffic inside your existing account, region and compliance boundary. Self hosted open weight models give full control over where data goes, at the cost of running inference yourself. Regulated clients often require the second or third option, so settle this first.
7. Fine tuning and prompt caching
Prompt caching cuts cost sharply when every call repeats the same long instructions, schemas or reference material. Confirm support and how long cached prefixes persist. Fine tuning is most useful for enforcing a consistent output format or house style, not for teaching the model new facts.
8. Vendor stability and deprecation policy
Models get retired. Read the policy: how much notice you get, whether versions stay pinned, whether a migration path is published. Then design so that swapping a model is a configuration change, not a rewrite.
Model Tiers: When Each Earns Its Place
Think in tiers, not brand names.
Frontier models (the top tiers of Anthropic's Claude family, OpenAI's GPT family and Google's Gemini family) are strongest at reasoning, long horizon agent work and reliable tool use. They cost the most and are slowest. Use them where a wrong answer is expensive, where the task genuinely requires multi step reasoning, or as the grader and final fallback in a cascade.
Mid tier models from the same vendors (Anthropic's Sonnet tier is a typical example, with Opus at the frontier) are the workhorse for most production traffic. They handle extraction, generation and moderate reasoning well, respond quickly and cost a fraction of the frontier tier. In the systems we have shipped this year, most calls land here.
Small and open weight models (Llama, Mistral, Qwen, Gemma and their fine tuned derivatives) win on classification, simple extraction, embeddings, routing and anything that must run on your own hardware or at very high volume. They are weaker at multi step reasoning and complex tool use, so keep them on narrow, well specified tasks.
Routing and Cascades: Cheap First, Escalate on Doubt
The most cost effective production systems do not pick one model. They route. A small model handles the request first; if confidence is low, the output fails validation or the request matches a hard pattern, the call escalates to a stronger tier. A frontier model grades a sample of the cheaper model's outputs, giving a continuous quality signal without frontier prices on every call.
A routing policy can be a configuration file that the evaluation harness also reads:
{
"task": "support_ticket_triage",
"default": "small-open-weight-v1",
"escalate_to": "mid-tier-v3",
"final_fallback": "frontier-v2",
"escalate_when": [
"confidence < 0.80",
"schema_validation_failed",
"ticket.priority == 'P1'",
"language not in ['en', 'es']"
],
"grader": { "model": "frontier-v2", "sample_rate": 0.05 },
"budget_cap_usd_per_day": 400
}Every model name is a string in config. A new version means changing the string, running the evaluation and merging if the numbers hold.
Choosing a model is not a one time decision. It is a pipeline you run every time a vendor ships something new, and the pipeline is only as good as the evaluation set feeding it.
Build a Golden Evaluation Set
This is the single most valuable asset in your AI program, and the one most teams skip.
- Collect 200 to 500 real examples from production logs or a pilot. Synthetic examples miss the messiness that breaks models.
- Label the correct output. Exact values for extraction and classification; a reference answer plus rubric for generation. Have a domain expert review the labels.
- Over represent edge cases. Ambiguous inputs, malformed documents, multiple languages, empty fields. Make about a quarter of the set deliberately hard.
- Version it in the repository. When a business rule changes, the evaluation set changes in the same pull request.
- Hold a slice back that nobody tunes prompts against, so you can detect overfitting.
Metrics per task type
- Extraction: per field precision and recall, plus exact match on the full record.
- Classification: accuracy, macro F1 and a confusion matrix.
- Generation: rubric score from a frontier grader, spot checked by humans weekly.
- Agentic tool use: task completion rate, schema valid call rate, steps to completion, rate of unsafe or unnecessary actions.
- Code: tests passing, lint clean, human review on a sample.
Run the Evaluation in CI
Treat a model change like any other code change. The suite runs on every pull request that touches a prompt, a routing rule or a model identifier, and fails the build if accuracy drops below threshold, latency exceeds budget or cost per task rises past a limit. Results post to the pull request as a table.
The payoff is that a model upgrade becomes routine. An engineer changes one string, the harness runs a few hundred examples, and the team merges or declines in an afternoon.
Decision Table
| Use case | Recommended tier | Approach |
|---|---|---|
| High volume classification or routing | Small or open weight | Fine tune on labeled data; escalate low confidence cases to mid tier |
| Structured extraction from standard documents | Mid tier | Strict output schema, validation, one retry, then escalate |
| Long document analysis or contract review | Mid tier with frontier fallback | Chunk and retrieve; test at real lengths; cache reference material |
| Multi step agent with tool calls | Frontier | Strict tool schemas, step limits, human approval for irreversible actions |
| Code generation and migration | Frontier | Tests as the acceptance gate; human review of every change |
| Regulated data that cannot leave your boundary | Self hosted open weight, or cloud hosted frontier in region | Deploy inside your VPC; retain nothing beyond policy |
How RG INSYS Approaches Model Selection
Every AI engagement at RG INSYS starts with the evaluation set, not the model. Our senior engineers work alongside AI coding agents to build the golden set, the harness and the routing layer in the first sprint, so every later decision has numbers behind it. That workflow delivers about 3x faster than a conventional team at roughly 60% lower cost than onshore rates, with 80% or more automated test coverage and human review of every change.
If you are deciding which model to put behind a feature, or you inherited an integration chosen from a leaderboard, see how we build LLM agents and copilots, how we handle AI and ML integration into existing products, and our QA automation practice that makes evaluation in CI routine. Start with the AI readiness assessment, or contact us with your use case and we will return a written scope within 48 hours.
Frequently asked questions
Should we standardize on a single LLM vendor?
No. Standardize on an abstraction layer and an evaluation harness, then use whichever model scores best per task. Single vendor dependence exposes you to price changes, deprecations and outages. Most production systems we build route across at least two tiers, often from different vendors, behind a common interface that makes swapping a configuration change.
How often should we re-run model evaluations?
On every pull request that touches prompts, routing or model identifiers, and on a monthly schedule to catch silent vendor side changes. Also re-run whenever a new candidate model could lower cost or improve quality. Because the harness is automated, each run costs an afternoon of compute rather than a project.
Are open weight models good enough for production?
For narrow, well specified tasks such as classification, simple extraction and embeddings, yes, and they are often the best choice when data cannot leave your infrastructure. For complex reasoning and multi step agent work, frontier tier hosted models remain more reliable. The right answer is usually both, arranged as a cascade.
How large does the evaluation set need to be?
Between 200 and 500 labeled, real examples is enough to distinguish models with confidence for most business tasks. Fewer than 100 and noise swamps the signal. Coverage matters more than raw size: include the ambiguous, malformed and rare inputs that actually cause production failures, and keep a held out slice nobody tunes against.
Pick the right model with numbers, not a leaderboard
Tell us what your product needs an LLM to do and we will help you build the evaluation set, routing layer and CI harness that make the decision defensible. Share your use case and you will have a written scope within 48 hours.
Book a Free Consultation