The Hidden Cost of Cheap Decisions
TypeSafe AI recently made a splash in the AI industry with their release of Jev, a System One Decision Model. Since then, a number of new decision models have popped up - Bespoke’s Nimble, Cloudflare’s Clef, OpenAI’s Decisions API, Perplexity’s Decider, and more.
Decision models are all the rage right now, and it’s for good reason. They’re cheap, fast, and give you a structured way to get a decision from an AI model, without the overhead of generating long responses.
The basic idea is that you pass some context, a question, and some potential answers to a decision model. The model will evaluate those answers against the context, and give you back some information (a choice, score, probability, etc.) that you can use in your application.
This is a really interesting development, if nothing else because it’s a very flexible way of performing zero-shot classification on a broad range of inputs, with a set of predefined output types, and minimal inference overhead.
So all is good in the world, we can make decisions really quickly now, right?
Well, maybe not.
I work for a company called Parsewave, which primarily deals with datasets used in AI. So, I’m very familiar with the models, the training data, and the relationship between those two things.
And I have to say, there’s a lot of room for developers to misuse these models.
I think there’s a real danger that some developers will see these decision APIs as the end-all, be-all of decision-making, and simply throw inputs at them with a set of predefined outputs.
Imagine you’re building a system that screens resumes, for example.
You might decide to use a decision API for this, passing in a resume and a job description with a question like this:
“Is this a good candidate for this job?”
What do you think the possible answers will be? Probably something like yes/no/maybe.
Or, maybe you want to know if an image was generated using AI or not, so you use a multimodal decision API and pass in an image along with the question:
“Was this image generated by AI?”
And again, the possible answers might be yes/no/maybe.
Now, the thing is, both of these implementations will likely work.
When you call the API, you’ll get a nice, valid response back. It may even come with a high confidence score, and there’s a decent chance that it’ll be correct.
But I think there’s a danger here. Just because a model can answer a question, it doesn’t mean that it’s qualified to make the decision that you’re asking it to make.
Let’s go back to the resume-screening example. What makes a good candidate for a job? Is it their background? Their experience? Their ability? Their familiarity with a certain tech stack? Their ability to learn?
All of these things might be important to different organizations, or even different people within the same organization. But by asking the model a poorly-defined question, we’re essentially asking the model to make a decision for us.
Another example would be image classification, such as detecting if an image is AI-generated. There may not be enough information within the image to know the origin of it, so the model will likely just be making a guess on where the image came from.
In these cases, the model is drawing conclusions without clearly defined criteria or necessarily sufficient evidence.
Crucially, the output of the model can then trigger an action.
Why does this matter?
Whose decision is it?
The model will use its preexisting capabilities and behavior to make a prediction. This is based on a variety of factors, such as the representations it has learned, its training objectives, and the distributions of data that it has seen.
Although the model may have the ability to generalize well to make good decisions (e.g. it doesn’t need to see thousands of resumes to make a good decision on hiring someone), that doesn’t mean the model is using the same criteria for making a decision as the developer.
For example, say we value a certain type of experience highly, such as seeing that someone has built their own stuff, and we would prefer to hire someone who had 2 years of great experience compared to someone with 5 years of industry experience.
There is no guarantee that a general decision-making model would make the same decision.
It may even understand both candidates, but make a different choice based on its own criteria.
In that case, the decision may still be reasonable, but is it the one that we would have made?
How would we know?
I think this is an important realization for developers to understand.
When we are using models, we are outsourcing part of our decision-making policy to the model.
What does the probability mean?
Even if the model is outputting a probability, what does this represent? Is it a prediction for how likely it is that the output is correct? Or how confident is it in making a certain decision? How closely do those probabilities match how often the outputs are actually correct?
Models can make mistakes, even if they are well calibrated.
Calibration describes how closely predicted probabilities correspond to actual outcomes across a set of predictions, but the model can still be wrong if the criteria are not well specified, or the model has not seen certain types of inputs before.
Even though the model’s output is constrained to a predefined format, the underlying decision-making process is not necessarily deterministic.
Aren’t we already solving this for LLMs?
To some extent, yes.
We have made many improvements to LLMs over the past few years, such as better post-training techniques, RL, evals, and workflows that allow LLMs to be run in certain ways at inference time.
This has led to the frontier LLMs having much better capabilities, such as following instructions and reasoning well.
This doesn’t remove the possibility of hallucination or reasoning errors entirely, but it gives you a lot more flexibility when constructing the process around the model.
Why is this important? Well, let’s consider an example. Let’s say we’re using a reasoning LLM to evaluate a resume.
Instead of just asking it “should we shortlist this candidate”, we might ask it to find relevant experience, evaluate that experience against a set of criteria, check that evaluation against a predefined rubric, and explain why it’s making its recommendation.
If the model tells us that the candidate isn’t a good fit because they don’t have a CS degree, and that doesn’t match our hiring philosophy, we can change the process.
It’s worth noting that we don’t necessarily have direct access to the model’s internal reasoning – a lot of reasoning models don’t give you access to the reasoning tokens, and even if they did, the model could be explaining its reasoning in a way that doesn’t match the actual process it used to arrive at its answer.
LLMs are really good at explaining answers, even when they’re wrong. However, by constructing a workflow around the model, we can make parts of its evaluation process visible and verifiable.
This gives us a lot more to work with when debugging problems, than if we just had a model that we asked to make a decision with zero-shot prompting.
Of course, much of this observability comes from the workflow itself, and we can build similar workflows around decision models.
With a zero-shot approach, we give the model some context, a set of options to choose from, and then it gives us a prediction. Let’s say it tells us that the candidate has a 0.23 probability of being a good fit.
Why? Was it because they don’t have a degree? Did it miss a skill? Did it misunderstand the role? Or did it understand everything, but has a different idea of what makes a good candidate?
We can try and tweak the prompt, or change the options we’re giving the model. We can evaluate it, calibrate it, and given a provider like Cloudflare that supports fine-tuning on Clef, we can even fine-tune it on our own data.
These models are very tunable. However, because the models don’t typically explain their reasoning, it can be harder to debug them – you have to do it by experimenting with the model and evaluating its behavior.
When fine-tuning, you have to consider an additional question – whose decisions are you fine-tuning on?
Fine-tuning can help the model adopt a particular decision-making policy, but it can’t tell you whether that policy is a good one. That’s still up to you.
Making subjective decisions less subjective
It can be tempting to avoid using these models for problems where there’s subjectivity involved, but I think that’s a mistake.
Instead, we should try to break our evaluation down into multiple smaller evaluations, as much as possible.
Going back to our resume-screening system, an example of a bad implementation would be the one we wrote above, where we just pass the resume into the decision-making model with the job description.
An example of a better implementation would be to list out very specific criteria.
Instead of asking whether the candidate has above-average ML skills, we could ask the model:
Does the resume indicate that the candidate has experience deploying ML models in production environments?
- Yes, the resume explicitly indicates this
- The evidence is unclear or incomplete
- No, the resume does not indicate this
And then we could repeat this for each of the specific skills we’re looking for.
Does the resume indicate that the candidate has experience building evaluation pipelines for ML models? Does the resume indicate that the candidate has experience owning production infrastructure? Does the resume indicate that the candidate has experience designing or working on agent harnesses?
Now, even though these are not deterministic questions, they’re hopefully easier for us to define, test, and evaluate.
One key thing to note here: we’re not asking if the candidate has the capabilities, we’re evaluating the resume as a piece of evidence. It’s possible that the candidate has experience deploying ML models in production, but their resume doesn’t indicate it.
If we’re automating this evaluation, we need to be able to delineate between “this candidate has these capabilities” and “this resume provides evidence that this candidate has these capabilities”.
We could even go a step further and define rubrics for how we’ll score each of these questions.
For our first question about experience deploying ML models in production, we could say that if the candidate has only experience working on academic projects involving ML, they’ll get a 0. If the candidate has experience working on projects which involved deployment to production, they’ll get a 1. And if they were specifically responsible for getting ML models into production, they’ll get a 2. We can define what each of these scores looks like and provide examples.
If the resume doesn’t provide enough evidence to score, we could mark it as unknown rather than assigning a zero.
Then, we could use more “traditional” application logic to combine these values. Perhaps certain skills are more important than others and so we weight them more heavily. Or maybe some of them are required and others are used to prioritize candidates if there are multiple candidates who meet the minimum criteria.
Even though our individual evaluations may not be deterministic, the way in which we combine our evaluations to make a final decision could be.
This is hopefully a better architecture, because we’re no longer leaving it up to the decision-making model to determine what makes a good candidate. Instead, we’re defining the criteria for a good candidate and asking the model to help us evaluate candidates against our criteria.
Note: we may not need to ask each of these questions to the model in a separate request. Depending on the API we’re using for our decision-making model, we might be able to ask all of these questions in a single request, as long as the answers to each of them are independent.
In general, our goal should be to keep as much of our decision-making policy as possible under our control.
The problem with decomposition
The thing is, it’s possible that we decompose our problem into multiple smaller problems but we don’t actually end up answering our original question better.
If the model is doing a poor job of understanding resumes in general, then it may make multiple incorrect evaluations for a given resume. Adding up multiple incorrect evaluations doesn’t give us a correct evaluation.
If our individual predictions are not trustworthy, then it probably doesn’t matter how we weight and sum them up - we can’t assume that our final prediction will be trustworthy either.
Also, if we decompose our problem into multiple subproblems, but there’s no way to answer the subproblems given the information we have, then it probably doesn’t matter how we decompose it - we still won’t be able to answer our original question.
At some point, we just need to accept that there’s not enough information for our decision-making model to answer a given question, no matter how we ask it.
Evaluation
I put this one in its own section because it’s extremely important. It is not optional. It’s also sort of obvious.
If we’re going to deploy a decision-making system, we need to have some sense of what success looks like for that system.
Going back to our resume-screening example, we could put together a representative dataset of resumes and have subject matter experts evaluate them based on our criteria. Then, we can see how our model’s predictions compare to our subject matter experts’ evaluations. We can see how often the model makes incorrect predictions and whether there are any kinds of resumes for which it consistently does a poor job. We can also see whether there are any criteria on which our subject matter experts do not agree, which might indicate that we need to better define those criteria.
(Note: there are also fairness and privacy considerations for a system like this, because it’s helping us make hiring decisions, but I’ll leave those as an exercise for the reader.)
It’s one thing to incorrectly route a support ticket. It’s another to disqualify someone from a potential job.
How much error can we tolerate? What does a false positive or negative cost us? Do we need a person involved?
What confidence thresholds should we use? Do we have enough examples from the work we expect the model to do to determine sensible thresholds?
Otherwise, what good is the confidence data?
(As a side note, this is equally applicable to reasoning LLMs.)
It’s still very possible for the model to reason beautifully to an incorrect answer.
Finding the sweet spot
The real question is:
At what point does a decision-model pipeline become more expensive than using a reasoning LLM?
If we’re decomposing subjective questions into many individual evaluations, extracting evidence, and creating multi-step decision-making pipelines, we may end up spending more.
And that’s okay.
The goal isn’t to use a decision model, it’s to use the right system for the job.
If we’re routing support requests, categorizing documents, evaluating conditions for agentic workflows, or doing some other simple task at volume, a decision model might actually be the right choice.
It’ll likely be faster and less expensive than other solutions for well-defined questions with constrained outputs where errors have limited impact.
If we’re trying to solve harder problems where the decision is harder to define and errors could be costly, a system combining evidence extraction, a series of simpler evaluations, some deterministic logic, and human oversight might make more sense.
In some cases, it might even make more sense to just use a reasoning LLM.
Not necessarily because it’ll do a better job of answering the question correctly, but because we might want the ability to inspect intermediate assessments, revisit different steps in the process, and verify the model’s work.
And in many cases, we may want to just write some code or use a more specific classification model to do the work.
It comes down to balancing cost, latency, accuracy, observability, and the impact of incorrect decisions.
If adding complexity gets us what we want, it’s worth it. If a simpler model can do the job, it might make more sense to go that route.
We still have the job
Regardless, I think decision models are interesting.
They make it easier than ever to include learned judgment in our application logic at extremely fast speeds and low costs.
But I also think they’re dangerous.
It’ll be easy to use them without considering how much of the decision-making process we’re leaving out. We’ve made learned judgment extremely inexpensive, and it’ll be easy to incorporate it into our software without considering the potential for error.
The API call will succeed, the response will validate against the expected schema, and our application will continue operating.
Whether we’re making the right decisions, though, is another question entirely.
It’ll be up to us as developers to be thoughtful about how and where we use these models. It’ll be up to us to define decision-making criteria, consider the limitations of the available evidence, evaluate the model’s decisions, and control the policies we’re asking the models to apply when making consequential decisions.
It’s not that decision models aren’t valuable or that reasoning LLMs are a better choice.
It’s that neither of them relieves us of the responsibility of decision-making. We’ve made decisions extremely cheap. It’s still up to us to make sure we’re making the right ones.
Originally published on Substack