New language models appear every few months, each claiming to be faster, cheaper and smarter. Teams building products face a practical question: which one should we use? Leaderboards and marketing pages cannot answer it, because your task, your data and your users are specific. This article offers a framework we use when helping clients choose, with the emphasis on measuring instead of guessing. Specific model names and prices change quickly, so we focus on the questions that stay useful.
Step 1: define the task precisely
"Add AI to the product" is not a task. Write down what the model must do, for whom, and what a good result looks like. Typical tasks include:
- Answering questions from your documents, with sources.
- Drafting replies, descriptions or marketing text.
- Extracting fields from invoices, forms or emails.
- Classifying and routing messages or tickets.
- Calling tools, such as searching orders or booking a slot.
- Holding a voice conversation in Arabic.
Each task stresses a model differently. Extraction needs reliable structured output. Support answers need faithfulness to the source. Voice needs low delay. Content drafting needs style control. Naming the primary demand narrows the field before you test anything.
Step 2: list the constraints that decide the shortlist
Language and dialect
If your users write in Arabic, test Arabic seriously. Check Modern Standard Arabic, Egyptian dialect, Gulf dialect and Franco-Arabic, and check how well the model handles mixed Arabic and English. Quality can differ noticeably between models, so rely on your own examples rather than claims.
Data privacy and location
Decide what data the model will see and whether it may leave your environment. Read each provider's terms on retention and training use. For sensitive data, consider providers with stricter agreements, regional processing, or models you can host yourself. We explore these rules in more detail in our article on responsible AI and data privacy.
Latency
A chat that answers in a few seconds is acceptable. A phone conversation is not. Interactive and voice products need fast first responses, which often means smaller or faster models, streaming output and careful design of the rest of the pipeline.
Cost at your real volume
Estimate cost per task, not per token. Count the average input and output length, the number of calls per user action, and how many users you expect. A model that is slightly better but ten times more expensive may not make business sense for a task that runs thousands of times a day. In voice products such as Tele Agent, per-call cost tracking is a core feature for this reason.
Step 3: build your own evaluation set
This is the most valuable step and the one teams skip. Gather fifty to a hundred real examples of the task, with the answer you would consider correct. Include easy cases, hard cases, messy inputs and cases where the right behaviour is to decline. Then run each candidate model on the same set with the same instructions.
Score what matters
- Correctness: is the answer right, judged by someone who knows the subject?
- Faithfulness: does it stick to the supplied sources instead of inventing details?
- Format reliability: does it follow the required structure every time?
- Tone and language: does it sound right in Arabic and in English?
- Safe behaviour: does it refuse or escalate when it should?
Have two people score a sample so you can see how subjective the judgement is. Keep the results in a simple table so decisions are traceable.
Step 4: compare the total picture
| Factor | Question to ask |
|---|---|
| Quality | Does it pass our evaluation set at the level we need? |
| Cost | What is the cost per completed task at our expected volume? |
| Speed | Is the response fast enough for the interface? |
| Privacy | Do the terms and location match our data rules? |
| Reliability | What are the rate limits, uptime history and support terms? |
| Lock-in | How hard is it to switch if prices or terms change? |
Often the answer is not a single model. Many products route simple tasks to a smaller, cheaper model and reserve a stronger one for hard cases. That adds complexity, so introduce it only when the savings justify the effort.
Step 5: design for change
Models improve, prices move and providers change policies. Keep the model behind a thin interface in your code, store prompts and settings as configuration, and keep your evaluation set ready to rerun. Switching then means running a test and changing a setting rather than rewriting the application. Log model versions with results, so a change in behaviour can be traced.
Do not forget everything around the model
In practice, retrieval quality, clear instructions, good source content, business rules and human review influence the result as much as the model itself. Before paying for a bigger model, check whether better documents or a clearer prompt fix the problem.
If you are choosing a model for a new product or reconsidering your current one, the Narmar team can help you build an evaluation set and compare options on your real data. Read about our AI services, look at our projects, or contact us.
