Start with model families, not a leaderboard
A useful shortlist usually includes a general-purpose hosted model, a lower-cost model and, where control matters, an open-weight alternative. OpenAI’s GPT family, Anthropic’s Claude family and Google’s Gemini family offer text generation, coding and multimodal capabilities across different products. Meta’s Llama family and models from Mistral add options for organisations that want more deployment flexibility. Availability and features vary by version and hosting provider.
Do not confuse a consumer assistant subscription with API access. A chat product includes an interface and product-specific features; an API exposes capabilities for your own application, with separate billing, rate limits and contractual terms. Enterprise agreements may add administration, security controls and negotiated retention settings that are absent from an individual subscription.
| Family | Useful evaluation focus | Main caution |
|---|---|---|
| OpenAI GPT | Tool use, structured responses, multimodal workflows | Features differ between model endpoints |
| Anthropic Claude | Document work, coding, instruction following | Long inputs still need retrieval testing |
| Google Gemini | Multimodal inputs and Google Cloud integration | Product and regional availability vary |
| Llama / Mistral | Deployment control and task-specific adaptation | Licensing and operations remain your responsibility |
Evaluate the task you actually have
Build an initial set of 50–100 representative cases. For invoice extraction, include poor scans, credit notes, multiple currencies and missing purchase-order numbers. For customer support, include ambiguous requests, outdated policies and cases that must be escalated. Keep a separate holdout set so prompt improvements do not merely memorise your examples.
Score observable outcomes: correct fields, valid schema, supported citations and appropriate refusals. Have reviewers assess answers without seeing the model name. Record median and tail latency, not just the fastest response. A model that is slightly more accurate but regularly exceeds a customer-facing timeout may be the wrong operational choice.
Context-window capacity is an input limit, not a guarantee of comprehension. Test whether the model finds relevant details in the middle of long documents and distinguishes old instructions from current policy. Retrieval can reduce cost and improve traceability, but it introduces its own failure point: a perfect answer is impossible when the right passage never reaches the model.
Price the successful outcome
Hosted APIs commonly charge separately for input and output tokens. Cached inputs, batch processing, reasoning modes and additional tools can change the bill. Compare current rates on the official OpenAI pricing page, Anthropic pricing page and Gemini API pricing page. A low headline token rate is not a reliable estimate of total cost.
Suppose a workflow processes 10,000 documents monthly, each using 3,000 input tokens and 500 output tokens. Before retries, that is 30 million input tokens and five million output tokens. Multiply each by its applicable rate, then add retrieval, tool calls, infrastructure and human review. Divide the total by accepted results—not attempted requests.
Smaller models often suit classification, routing and constrained extraction. Larger or reasoning-oriented models may justify their expense on ambiguous analysis. Routing between them can help, but the router also needs evaluation: confidently sending difficult cases to a weak model defeats the saving.
Choose control deliberately
Hosted services reduce infrastructure work and provide rapid access to capabilities. Their disadvantages include external dependencies, changing model availability and provider-specific interfaces. Open-weight deployment can offer tighter control over location and configuration, but requires capacity planning, security patching, monitoring and licence review. “Open weights” does not automatically mean unrestricted commercial use.
For sensitive information, inspect data-processing terms, subprocessors, retention, training-use policies and regional processing commitments. Avoid assuming that settings in a chat application apply to an API. Keep credentials outside prompts, restrict access to retrieved documents and log only the information needed for debugging.
Finish the pilot with a model version, a written evaluation report, a spending limit and a fallback plan. Re-run the test set before changing providers or versions. Your best model is the one that meets a measurable threshold inside a dependable system—not the one that wins every public comparison.