I built a pipeline that finds and ranks job postings for me. For every posting it needs the same small answers. Is it remote? Where is the office? What seniority? I used to send an AI agent to answer them. Now a decision model does.
To check that choice, I gave the decision model, Jev, and two AI agents, Luna and Haiku, the same ten questions about the same 40 postings. These postings were held out: I never tuned anything on them. I scored every answer against labelled answers. One Jev call per posting got 95.3% right, in 0.6 seconds, for $0.00011. One agent call got 94.0%, in 10 to 14 seconds, for 10 to 41 times the cost.
What a decision model is
A decision model takes some text and a list of typed questions. It returns typed answers with probabilities. A typed question has a fixed answer shape: yes or no, one choice from a short list, or a score on a fixed scale. The model does not write text, call tools or explain itself.
The one I use is Jev, from TypeSafe. They call it a “System One” model, after the fast, automatic thinking in Daniel Kahneman’s Thinking, Fast and Slow. Jev charges $0.042 per million input tokens, and output is free. A token is a chunk of text, about three quarters of a word.
An agent is the opposite end. It is a general language model inside a harness, the program that gives it a system prompt and tools so it can plan, read files and act. That is what you want to build something. It is a lot of machine for “is this job remote?”
The test
I already had a labelled set: 100 public job postings, split 60 for tuning and 40 held out. Four Sonnet labellers wrote the answer key, with a quote for each answer. A blind re-label of 20 postings, which I use as the ceiling, scored 96.7% on the location questions. The ten questions cover location facts, work mode, seniority, role type and whether the job is still open.
I ran each system once on the 40 held-out postings:
- Jev, one call per posting, all ten questions at once, the way my pipeline asks its fact questions.
- Luna (GPT-5.6 Luna), one call per posting, through Codex, OpenAI’s agent harness, using the same code my pipeline uses for its agent work. Effort is how long the model may think before it answers. Low effort was its cheapest and best result here.
- Haiku (Claude Haiku 5.5), one call per posting, run as a Claude Code agent.
The agents got the same rules as Jev, in my pipeline’s own prompt, and gave valid JSON every time. I also ran Jev and Haiku with one question per call. Costs are list prices. Each call’s tokens, time, cost and answers are logged.
| Per posting | Tokens in | Seconds | Cost | Right |
|---|---|---|---|---|
| Jev, 1 call | 2,669 | 0.6 | $0.00011 | 95.3% |
| Jev, 10 calls | 11,759 | 5.2 | $0.00049 | 95.3% |
| Luna, low | 13,951 | 9.9 | $0.0011 | 94.0% |
| Luna, high | 13,951 | 11.1 | $0.0013 | 93.5% |
| Haiku, Claude Code | 43,592 | 13.5 | $0.0046 | 94.0% |
| Haiku, bare | 8,712 | 9.9 | $0.0020 | 92.0% |
“10 calls” means one call per question. Luna and Haiku make one call per posting. “Bare” means a short system prompt and no tools. “Right” is the share of answers that match the key. Seconds are summed per posting, as if the calls ran one after another.
Quality is a near tie. I expected the agents to do worse. They did not, on these questions. 95.3% against 94.0% is 5 answers out of 400. One run of 40 postings cannot separate that from noise.
Cost and time are not a tie. The cheapest agent cost 10 times more and took 16 times longer. Haiku through Claude Code cost 41 times more.
Where the agent’s tokens go
The posting is the same in every call. The difference is what comes with it. Jev reads the posting and ten short questions. An agent call also carries its harness: a system prompt, tool descriptions and my longer rules prompt. Haiku through Claude Code read 43,592 tokens to answer ten multiple-choice questions. Stripped to a bare prompt with no tools, it still read 8,712, and scored 2 points lower, within noise.
One question at a time
What if each call answers just one question? I ran that on 10 of the postings: 100 calls per system.
An agent call has a fixed floor. Asking it one small question costs almost as much as asking it ten: 42,309 tokens against 43,592. A decision model has almost no floor, so its cost follows the size of the question.
Splitting also cost Jev something. Ten single calls scored the same as one combined call, but read 4.4 times the tokens, because each call resends the posting. TypeSafe’s own docs report the same effect for 13 questions. So the rule is: small questions, asked together, over one shared posting.
Where it sits in the pipeline
Agents still do the open-ended work: searching the web, reading pages and scoring companies. I call that part the agent tier. The decision model does the repeated sorting. Today it makes about three calls per posting and answers 28 questions in all.
Question engineering
A decision model is only as good as its questions. These are the rules I ended up with:
- One fact per question. “Is the office in country X?” and “Is it remote?” are two questions, not one.
- A short, fixed list of answers, plus “not stated”. The model can say the posting is silent instead of guessing.
- Say what “no” means. Silence is not a no. A vague region is not a no.
- Say what to ignore. For example, the title or the nice-to-have skills.
- Change one question per round, and re-run the same 100 postings before keeping it.
Two rewrites show why.
Location. The question “Is the job based in country X?” started at 92.5% on held-out postings. I loosened it so that remote work open to several countries, X among them, counted as yes. It fell to 78.3%. Then I wrote the opposite rule down: remote work open to many countries is a no. It rose to 97.5%, 2 more postings right out of 40 than at the start. Across all six location questions, wording fixes like this moved the held-out mean from 92.6% to 96.4%.
Role type. The old question asked “What kind of work is this job mainly?” and gave bare names. The model leaned on job titles. The new one asks where the core of the work is, judged from duties and must-have skills, not the title. On a reference set of 104 postings, labelled by AI reviewers and not a random sample, it filtered out 33 of 38 wrong-type jobs instead of 22. The price was 6 good fits wrongly dropped out of 66. Testing it took 104 calls and $0.01. Re-running it on all 606 rated jobs cost $0.055.
Longer is not better. I once replaced a short five-step score with a longer one that described each step. Ranking quality on held-out postings dropped. Rank correlation with the answer key, where 1.0 means the same order, fell from 0.88 to between 0.80 and 0.83. I kept the short one.
Cheap calls are what make this loop possible. My first tuning rounds took 2,817 calls and cost $0.35.
One real run
On 26 October the pipeline rated 441 postings. Jev made 1,264 calls for $0.23. The agent tier made 226 calls for $0.73. In the rating step, Jev’s 441 calls took about 23 seconds. The 150 agent calls in that step, which do a different job, scoring companies, took about 411 seconds. Speed matters more to me than dollars, because I pay flat fees for most AI tools. An agent call costs me quota and waiting time.
Where it does not fit
- The three-way label is weak. A script turns Jev’s answers into apply, stretch or skip. That label agreed with the key 78% to 83% of the time. On the tuning set, answering “skip” for every posting does as well. It sorts my reading list. It does not decide.
- Most small decisions are already free. Inside a working agent session, the context is already loaded. A separate model saves nothing there. The savings come where one cheap call replaces a whole agent call, many times over.
- A script still beats a model. If a rule can decide, write the rule.
- It is hosted. I send it public text only, like job postings, and never my private notes.
Limits of this test
One run of 40 postings, ten questions, and an answer key written by AI labellers, checked by a blind re-label of 20 postings. Jev’s questions and the agents’ prompt were both tuned on this same set of 100 postings, so neither side is a stranger to it. The role question in the test is the older version. All runs made 8 calls at once, and the agent runs also shared the machine with each other, so their times include some waiting. The one-question test used only 10 postings. Prices are list prices. A harder question, one that needs reasoning across a whole page, could favor the agent.
The rule I use now
Before I send an agent, I ask three things. Is the answer a yes, a choice or a score? Does it repeat across many items? Can I build a labelled set to check it? If all three are yes, I write small typed questions, ask them together, and try a decision model first. A script comes before that.
I wrote more about measuring this way in What measuring my AI pipeline changed.