Do You Need a Frontier Model to Route Support Tickets?

I benchmarked Jev, GPT‑6, Gemini and ten local open models on 4,200 tickets and queries. The model mattered less than I expected, until the task got hard.

I started this project to test a simple idea. Jev is a typed decision API: you send a ticket and a list of categories, and it returns a choice with a probability for every option. My hypothesis was that it would route customer‑support tickets more cheaply than general‑purpose language models, and maybe more accurately.

Five datasets, 18 models, nearly 100 benchmark runs and about $28 of API spend later, the answer is more useful than a yes or no:

• For a small, well‑defined routing policy, you don’t need a frontier model. Once the test data and the prompt were right, a $4‑per‑100k‑tickets model, and even a free local model, routed tickets as accurately as the most expensive models.
• For fine‑grained intents and out‑of‑scope detection, the bigger models earn their price, mainly because their answers are right more often and their confidence lets you safely automate more.
• Jev was not the cheapest option, but among the accurate models its probabilities were the most informative confidence scores.

TL;DR

• My first benchmark was wrong. 100,000 rows, but only 445 unique sentences and keyword labels. It ranked models on my labelling quirks. Rebuilding the dataset moved the same models from 64–85% to roughly 90–100% accuracy and reshuffled the ranking.
• Writing the routing policy into the prompt beat switching models. One rewritten set of category descriptions lifted every model, e.g. gpt‑5.4‑mini from 87.7% to 98.7% and gpt‑5.4‑nano from 80.7% to 93.6%.
• On clean and deliberately hard support tickets, small models are enough. gpt‑6‑luna scored 99.5–100% for $2–4 per 100k tickets. Locally, gemma4:31b scored 99.9% on the rebuilt set and 98.9% on the hard set for $0 per token.
• On public, human‑labelled data (BANKING77, CLINC150) the gap reopens. The best models reached 85% and 94.6%; gpt‑6‑luna 82.9% and 88.6%; Jev 80.5% and 87.2%; the best local models 74.8% and 81.4%.
• Confidence quality decides how much you can automate. gpt‑6‑luna without reasoning was 86% accurate on CLINC150 but could automate almost nothing at a 95% target, because it is just as confident when wrong. Among the accurate models, Jev’s probabilities separated right from wrong answers best.
• Jev packing matters. 20 tickets per call instead of one cut cost by 45% and raised accuracy. But Jev’s cost grows with the number of labels, from $12.60 per 100k tickets with 5 labels to $96 with 151.

The question

Accuracy on its own doesn’t tell you whether a model can route tickets without a person. What I care about is:

• How many tickets can it handle alone while staying above an accuracy target? I used 95% as the main target.
• What does that cost per 100,000 tickets, including the tickets that still go to a person?
• How fast is each decision?

A ticket counts as automated when the model’s confidence clears the lowest threshold that keeps accuracy on the automated tickets at or above the target. Everything below that threshold goes to a person. For total cost I assume $1.00 per human review. That’s a placeholder; use your own number.

What I tested

Every model got the same instructions and label descriptions. Hosted models ran through the providers’ Batch APIs where possible, which halves the price. I also ran gpt‑6‑luna in real time, because Jev always answers in real time and batch results can take minutes to hours.

Lesson 1: my first benchmark was lying to me

My first dataset had 100,000 rows but only 445 unique sentences. The “clear” template literally contained the answer (“please fix this billing issue”). The “ambiguous” tickets were labelled by a coin flip. The “edge” tickets were labelled by a keyword a reasonable person would ignore: “The system is behaving unexpectedly regarding the password logic since the last update” counted as account, not technical.

On that data every model looked mediocre, and the ranking mostly measured which model happened to match my quirks. granite4.2:30b looked like the best open model; on the rebuilt dataset it came near the bottom of the local models.

Takeaway: before you compare models, read 50 rows of your test set. If a person would argue with the labels, you’re measuring label noise.

Lesson 2: say your routing policy out loud

On the rebuilt dataset, the most disputed tickets were questions like “My Netflix password isn’t working, what do I do?” My labelling rules call that off‑topic: it is support, but for another company’s product. My prompt only said off‑topic means “unrelated to customer support”. gpt‑6‑sol and gemini‑3.8‑flash scored 0% on these tickets and gemini‑3.1‑pro 6%; the best result was 69%. The models called them account, and honestly they had a point.

So I rewrote the category descriptions to match the labelling rules exactly, and added one precedence sentence: “If the message mentions several areas, choose the one the customer is explicitly asking for; background mentions and areas the customer says it is not about do not count.” Nothing else changed.

Every model improved. The frontier models went from 0–6% to 100% on the “other company” tickets. gpt‑5.4‑mini rose from 87.7% to 98.7%, gemini‑3.5‑flash‑lite from 93.2% to 98.7%, and Jev (20 per call) from 95.3% to 98.6%.

Takeaway: the prompt is part of the model. A routing policy that lives only in someone’s head costs more accuracy than a bigger model wins back.

Result 1: on clean tickets, the task is nearly solved

With clean data and a clear policy, most models are at or near the ceiling:

Eight configurations scored 99.5% or higher, including the cheapest hosted model, gpt‑6‑luna (100% with default reasoning, 99.5% without), and three local gemma4 models (99.9–100%). Jev scored 98.6–99.2% when packing 20–50 tickets per call. The weaker models made most of their errors in the same place: the off‑topic look‑alikes. muse‑glimmer:30b, which reasons silently even with thinking turned off, scored 98.9% here and 98.7% on the hard set, but took 17.5–18.7 seconds per ticket, too slow for real‑time routing on a laptop.

Result 2: the hard set separates Jev and the small OpenAI models

The hard set was built to break models. It has long email threads where the newest message asks for something different from the quoted history, requests buried between unrelated remarks, two‑word messages, three languages, heavy typos, sarcasm, and off‑topic mail that looks like support: out‑of‑office replies, vendor pitches, and questions about Netflix, Spotify or Tesla.

The current hosted models barely noticed. gemini‑3.1‑pro, gemini‑3.8‑flash, gpt‑6‑sol, gpt‑6‑luna and gemini‑3.5‑flash‑lite all scored 99.7–100%. Jev (94.4%), gpt‑5.4‑mini (95.6%) and gpt‑5.4‑nano (85.7%) lost almost all their points in one place: off‑topic look‑alikes. Jev routes other companies’ support questions by topic (“Netflix password” becomes account, “Tesla app” becomes technical) and tends to treat vendor sales pitches as general.

The local models split into two groups. gemma4:31b (98.9%) and muse‑glimmer (98.7%) held up best, but both are slow. gemma4:31b takes 5.7 seconds per ticket on my Mac and muse‑glimmer 18.7. The faster gemma4:26b fell to 93.4%, and its errors were almost all in email threads (66%): in every one of its 48 wrong thread answers it routed the ticket by the quoted older message instead of the newest request. If you route email, strip quoted history before classifying, or test for this explicitly.

Result 3: public, human‑labelled data

My v2 and v3 datasets are synthetic, and I wrote the labels. To check the conclusions against labels I didn’t write, I ran the same models on two public benchmarks that researchers have used for years: BANKING77 (77 banking intents) and CLINC150 (150 intents plus out‑of‑scope). They use their original labels, and the label names are the only descriptions.

This is where model size shows up:

• BANKING77: gemini‑3.8‑flash 85.0%, gpt‑6‑sol 84.8%, gemini‑3.1‑pro 84.3%, gpt‑6‑luna 82.9%, Jev 80.5%, gemini‑3.5‑flash‑lite 78.6%. The intents are very fine‑grained (card_arrival vs card_delivery_estimate), and published studies have found label errors in this dataset, so nobody gets close to 100%.
• CLINC150: gemini‑3.8‑flash 94.6%, gemini‑3.1‑pro 93.6%, gpt‑6‑sol 92.1%, gpt‑6‑luna 88.6%, Jev 87.2%. The biggest gap is in recognising out‑of‑scope queries: gemini‑3.8‑flash caught 90% of them, gpt‑6‑sol 83%, Jev 76%, gpt‑6‑luna 71%, and gpt‑5.4‑mini just 6%.
• Local models fell furthest behind here: the best reached 74.8% on BANKING77 (gemma4:26b) and 81.4% on CLINC150 (qwen3.6), and they caught 0–62% of out‑of‑scope queries. Their confidence also separated right from wrong poorly, so at a 95% target they could automate only 1–36% of queries.

Can you trust the confidence score?

Accuracy isn’t enough to automate. You need a confidence score that is lower when the model is wrong, so you can send those tickets to a person.

On CLINC150, gpt‑6‑luna without reasoning was 86% accurate, but it could automate only 0.2% of queries at a 95% target: its self‑reported confidence was near 1.0 for right and wrong answers alike. Among the models above 85% accuracy, Jev’s probabilities were the most informative. Its average confidence was 0.93 on correct answers and 0.69 on wrong ones, a gap of 0.23. For gpt‑6‑sol, gpt‑6‑luna and the two larger Gemini models the gap was 0.02–0.18, and 15–23% of their wrong answers came with a confidence of 0.99 or higher. That’s why Jev could automate 77% of CLINC150 queries at 95%, more than gpt‑6‑luna with reasoning (72%), despite a lower raw accuracy.

Takeaway: check that confidence actually drops on wrong answers before you use it to automate. A model that says “1.0” for everything can’t be thresholded.

Jev: packing tickets into one call

Jev charges per call, and a large part of each call is fixed overhead. Sending 20 tickets in one call, as separate questions, is the biggest cost lever:

On the clean set, going from 1 to 20 tickets per call cut cost from $22.70 to $12.60 per 100k tickets and raised accuracy from 96.1% to 98.6%. Past 20, savings are small. One catch: every question repeats the full label list, so Jev’s per‑ticket cost grows with the number of labels. It was $12.60 per 100k with 5 labels, $61 with 77 (BANKING77) and $96 with 151 (CLINC150).

What it costs

For a five‑label support policy, the numbers are simple. gpt‑6‑luna matches the frontier models at $2–4 per 100k tickets, against $42–56 for gemini‑3.8‑flash and gpt‑6‑sol and about $250 for gemini‑3.1‑pro. Jev costs $12.60–13.30 per 100k with 20 tickets per call. Local models cost nothing per token.

With fine‑grained intents the picture flips, because every query a model can’t automate costs a human review:

At $1 per review, gemini‑3.8‑flash has the lowest total cost on CLINC150 even though it is ten times more expensive per query than gpt‑6‑luna. It automates 97% of queries at 95% accuracy, against 72% for gpt‑6‑luna.

So, do you need a frontier model?

Not for a small, clearly written routing policy. With five categories and rules written into the prompt, gpt‑6‑luna and gemini‑3.5‑flash‑lite matched gpt‑6‑sol and gemini‑3.1‑pro within a fraction of a percentage point, for roughly a tenth to a hundredth of the price, and a local gemma4:31b came within about one point. The extra spend buys nothing measurable.

Yes, or at least a “flash” tier, when the label space is large or out‑of‑scope detection matters. On 77 and 150 fine‑grained intents, the stronger models were 2–7 points more accurate than gpt‑6‑luna and Jev, much better at spotting out‑of‑scope queries, and able to automate far more at a 95% target. If a human review costs anything close to a dollar, that difference pays for the model many times over.

Jev was competitive on clean tickets (98.6–99.2%) and had the most usable confidence scores, which matters if you automate on a threshold. It was not the cheapest: gpt‑6‑luna in real time was about three times cheaper at similar latency, and Jev’s price rises with the number of categories. Its clearest weakness was out‑of‑scope look‑alikes.

What I’d do in production

1. Write the routing policy down and put it in the prompt, including what’s out of scope (other companies’ products, auto‑replies, vendor pitches). That alone was worth more than any model upgrade.
2. Build a test set from real tickets and have two people label a sample. Where they disagree, fix the policy, not the model.
3. Start with the cheapest model that meets your target on that test set: gpt‑6‑luna, gemini‑3.5‑flash‑lite, or a local gemma4 if data can’t leave your network (31B for accuracy, 26B for speed, but strip quoted email history first).
4. Check the confidence score before you automate on it. Plot accuracy against the share automated. If the line is flat, use a model with real probabilities (Jev, or token log‑probabilities) or add a second opinion.
5. Escalate the uncertain tail to a person or a stronger model. For large or fine‑grained label sets, test a flash or frontier tier: fewer escalations can pay for it.
6. Pack requests if your provider charges per call; with Jev, 20 tickets per call was the sweet spot.
7. Re‑run the benchmark whenever the prompt, policy or model changes. Each hosted run took minutes and cost between a few cents and a few dollars.

Limitations

• Synthetic tickets. v2 and v3 are generated from templates with written rules. They are far more varied than my first attempt, but real inboxes are messier. The public datasets are real queries, but not support tickets from one company.
• Self‑reported confidence. GPT‑6 and Gemini don’t return token probabilities, so their confidence is a number the model writes down. It is not a calibrated probability.
• Human review cost is an assumption. Total‑cost figures use $1.00 per escalated ticket.
• Local latency is one machine: an Apple M4 Max with 64 GB, Ollama, 4 requests in parallel. Local cost is $0 per token, not $0 to operate.
• Gemini and 151 labels. Gemini rejects a response schema listing all 151 CLINC150 intents, so for that dataset it answered from the prompt’s list; 2 of 3,000 Gemini answers fell outside the list and counted as wrong.
• Prices change. All prices are list prices checked on 24 September 2026.
• One run per configuration. Intervals are 95% bootstrap intervals over tickets, not over repeated runs. Intervals on “share automated” are wide when a model’s confidence values cluster.

Method notes

The dataset generators use fixed seeds, every run keeps row‑level results, and every number and chart in this post is computed from those rows by one analysis script. The charts are rendered from Vega‑Lite specs with their data inlined.

The one‑page version