LLM Classification Cost vs Accuracy: We Tested TypeSafe AI's Jev

At Anansi News, the AI-powered news organisation for Ghana and a side venture of Kiseki Labs, an LLM has filed every story into one of nine sections for more than a year. In September 2026 we tested that LLM classification step against Jev, a new model from TypeSafe AI that returns a choice rather than text. On 540 real articles, Jev matched our Claude Opus 5 reference 92.6% of the time, against 79.8% for gpt-4o-mini, at about a quarter of the cost.

The same kind of decision sits inside plenty of mid-market and enterprise systems, from ticket routing to invoice checks, and it can easily end up running on a far more expensive model than the task needs.

Why LLM classification gets expensive quietly

Which queue should this ticket go to? Is this invoice a duplicate? Does this clause deviate from the standard? Is this customer email a complaint? Many of these questions are still answered by rigid code: if-else rules, keyword lists and regular expressions. That holds up until the decision needs what we call adaptive intelligence, meaning judgement that isn't deterministic and can't simply be coded.

Increasingly, those decisions go to an LLM instead. The input is sent with a list of labels, the model is asked for one word back, and the reply is parsed. It works, which is why nobody revisits it. But the bill scales with every document, and when the classifier reuses whatever model the rest of the product runs on, a simple labelling job ends up priced like reasoning.

What we tested at Anansi News

We follow the same rule at Anansi News that we apply on client work: evaluation before architecture. We built a test set of 540 articles published between 24 August and 22 September 2026, drawn from eight Ghanaian sources with every section represented.

The labels already stored in production came from gpt-4o-mini, so they could not serve as the answer key. Instead, Claude Opus 5 labelled every article against an explicit filing policy (file by the main subject, and use the most specific section that fits) and listed any other section an editor could defend. We scored gpt-4o-mini's production labels and Jev's answers against that reference.

Jev's median response time was 235 milliseconds, about a quarter of a second. Two caveats matter. The reference is a frontier model rather than a human editor, so these figures measure agreement with Opus 5, not ground truth. And gpt-4o-mini never had the filing policy in its production prompt, so we also ran Jev with a bare one-line question and no policy. It still reached 90.9%.

What TypeSafe's Jev is, and how it differs from an LLM

Jev is the first of what TypeSafe calls System One models, which we refer to here as decision models. You give the model the material to judge and a question with a fixed set of answers. It returns a probability for every answer, and the answer is always one of the options you supplied.

LLM APIs can also force an answer onto a fixed list if you configure them to. Anansi's older prompt didn't, and an unrecognised reply meant the article was dropped. What an LLM doesn't give you cheaply is a probability for every option, and that turned out to be the most useful part of the result.

What LLM classification costs per million decisions

The table shows the list-price cost of classifying one million items of about 1,100 tokens each (roughly an 800-word article), with a one-word answer. Jev costs $0.042 per million input tokens, and output is free.

Real LLM bills tend to run higher than these sums, because production prompts carry instructions and examples, and reasoning models bill for thinking before they answer. Where the work can wait, batch APIs roughly halve the LLM figures.

Against the cheapest OpenAI models, GPT-6 Luna and gpt-4o-mini, Jev is two to four times cheaper, so the stronger arguments there are accuracy and a usable confidence score. Against Claude Haiku 4.5 it is about 25 times cheaper, and against mid-tier and frontier models 50 to 250 times. At a million classifications a month, that is roughly $27,000 a year on Claude Sonnet 5 against about $550 on Jev, and Sonnet 5 still costs around 25 times more at batch prices.

Confidence scores: automate most decisions, review the rest

Every Jev answer carries a confidence score between 0 and 1. TypeSafe works it out from how the probability is spread across the options: all of it on one answer gives 1.0, and the more evenly it is spread, the lower the score. On the Anansi data, that score tracked accuracy closely.

In practice, that means setting a threshold per action. Decisions above it are automated, and the remainder go to a person or a larger model, with each threshold set according to what an error costs. TypeSafe's own guidance is to act automatically above 0.9 and route anything below 0.5 to a human.

The data also ruled out a design we might otherwise have built. Sending Jev's unsure cases to gpt-4o-mini for a second opinion would have made results worse. Below 0.7 confidence, across 83 articles, gpt-4o-mini matched the reference 37% of the time against 69% for Jev.

When to move LLM classification to a decision model

Start with the output. If it's a label, a yes or no, or a score from a fixed set, a decision model is a candidate. If it has to be written text, such as an explanation, a summary or a reply, keep the LLM.

Then look at what the workload runs on today. On a mid-tier or frontier model at meaningful volume, the cost case alone justifies a pilot. On a small, cheap model, switch only if your own evaluation shows an accuracy gain. And wherever a mistake is expensive, deploy with confidence thresholds and human review from day one.

Risks to check before you commit

Jev is a closed, early-access model behind an API, launched in September 2026 after two years in stealth. Commercially, check where data is processed. TypeSafe says its service is currently based on the US West Coast, so UK and EU organisations need the right terms in place before sending anything sensitive. TypeSafe also says openly that it cannot yet prove its pricing isn't subsidised, though it expects prices to fall. Its published limit of 1,200 requests a minute, about 1.7 million a day, is worth checking against your peak volumes.

Technically, TypeSafe publishes a candid list of Jev's known weaknesses, including taking negations at face value and falling accuracy when inputs fill with irrelevant material, which is why our implementation trims each article to 8,000 characters. Calibration, meaning whether a 0.9 confidence really is right about nine times in ten, held up on our data. A competitor cites third-party benchmarks where it held up less well, so measure it on yours.

The change at Anansi News is in code review and not yet live, but it's built to be reversible. The model is fixed to one version, a single setting switches back to gpt-4o-mini, and gpt-4o-mini takes over automatically if TypeSafe's API fails. We would put the same safeguards around any client deployment.

For transparency, Kiseki Labs has no commercial relationship with TypeSafe AI. As practitioners, we're simply glad to have more kinds of intelligence to choose from.

How we would run the pilot

Four weeks is enough to reach a go or no-go decision on one workflow. We inventory your LLM calls, flag those that are really classification, and build a reference set for the highest-volume one. We then run a decision model in shadow, alongside the live system without affecting its output, and measure agreement, accuracy at each confidence level, response time and cost. Finally, we set thresholds per action and ship behind a switch with a fallback. It is the same evidence-first approach behind our Pilot to Production engagements.

Frequently asked questions

What is TypeSafe AI's Jev?
Jev is the first System One model from TypeSafe AI, launched in September 2026. It answers structured questions with a probability for each allowed answer instead of generating text, and costs $0.042 per million input tokens.

Is a decision model as accurate as an LLM for classification?
On our data, Jev beat the small LLM it would replace (92.6% against 79.8%) and agreed with Claude Opus 5, a frontier LLM, on 92.6% of articles. Opus 5 was our answer key, so we can't claim Jev beats frontier models. Only an evaluation on your own inputs settles it.

How much does LLM classification cost?
At list prices, classifying a million items of about 1,100 tokens costs about $113 to $168 on the cheapest OpenAI models, $1,125 on Claude Haiku 4.5, $2,250 on mid-tier models and $4,500 to $11,250 at the frontier. Jev costs about $46.

Couldn't we just fine-tune a BERT classifier?
You could. Fine-tuned BERT-style classifiers have been fast and cheap to run since 2018, but they need labelled training data, a training pipeline, hosting, and retraining whenever labels change. A decision model gets you close to that per-call cost without the training work, because the labels are passed in with each request. What you give up is control, since a fine-tuned model runs on your own infrastructure. Either way, re-run your evaluation whenever the labels change.

If you suspect some of your LLM spend is going on multiple-choice questions in disguise, request an executive briefing and we will help you find out.