In 1865, English economist William Jevons noticed something worrying. As steam engines increasingly became more efficient, Britain increasingly burned more coal. Making steam power cheaper only created more reasons to burn coal.

This counterintuitive idea became Jevons Paradox, and was the motivation behind TypeSafe's latest model: Jev.

Jev is interesting not just because it's cheap, but because it’s refreshingly good at classification. It marks the beginning of a new era of agent evals.

Today, LLM-as-a-Judge evals are expensive enough that engineers naturally limit themselves. We run evals on only a fraction of prod data, and we're constrained to running just a handful of judges at a time. This system actively pushes teams to run fewer evals than they actually want to.

Jev, however, is a meaningful step towards a future that actively fights this system.

For the first time ever, we're getting an early glimpse at genuinely general-purpose classification that's cheap, fast, and flexible enough to replace LLM judges for a surprisingly large class of evals.

This is super exciting for evals because, at its core, an agent eval is really just a difficult classification problem: given an agent run, classify if it passed or failed some test.

victor@stokebuilder

So jev is basically 2022 machine learning, without the need for a minimum training dataset.

Sep 19, 202662642.7K

In an ideal world, every AI team can hand-train classification models for evaluating agent behaviors. Unfortunately, this isn't feasible -- data is expensive, and time is even more expensive.

Jev was designed for this exact problem, making it a truly phenomenal fit for agent evals. Because it's so cheap, we can track behaviors, intents, and patterns across millions of agent runs.

In other words, Jev makes running evals a lot cheaper and efficient.

What does this mean for evals? Jevons Paradox suggests a particularly exciting future: we're going to run a lot more evals.

Pause for a moment and imagine a future where every single agent run gets evaluated by dozens of judges.

Was the user frustrated? Did the agent actually complete the task? Did it use the right tool? Was the user asking for something the product doesn't support? Did the agent recover from a mistake? Did it violate a company-specific policy?

We're heading toward a world where agents execute a truly ridiculous number of actions on our behalf: writing software, moving money, answering customers, researching drugs, talking to other agents, and controlling increasingly large pieces of the economy.

At that scale, we need an equally ridiculous amount of measurement.

If AI eventually makes billions of decisions every minute, running a few sampled evals is going to look insane in retrospect. We should be evaluating everything, and we shouldn't settle for anything less.

This is the future that I'm so passionate about building. At my company, we don't even like the word "eval" because it connotes a level of constraint. Our goal, quite literally, is to let you eval everything in prod.

1. Make your evals smaller, not bigger

Good evals are atomic. When model calls are expensive, there's an incentive to cram everything into one judge. This is convenient, but it's not the most effective. With Jev, you can build atomic evals that are performant yet affordable at scale.

2. Lean on Jev's probabilities

Have Jev answer your eval questions with confidence probabilities. Getting a probability (rather than a single label) from your evals lets you analyze a new dimension of information. For example, you can accept high-confidence evals, and drop the rest or send them to a human to review.

3. Connect the dots to your product

It's important not to confuse more measurement with better measurement. If your "user frustration" classifier doesn't correlate with anything your users actually care about, running it on every single production trace doesn't make it more useful. You need to actually connect your evals back to product and business metrics. Does this failure mode predict churn? Does fixing it increase task completion? Do conversations classified as successful actually lead to the user coming back?

Jev is most likely not the model we'll be talking about in five years. But that's not really the point.

The important thing is that Jev unlocked an entirely new paradigm of classification. Instead of maintaining five expensive judges and being constrained to sampling production traffic, teams will run hundreds of tiny, specialized classifiers on every single run.

Soon, we'll live in a world where measuring an agent is nearly as cheap as running one, and I'm incredibly excited to reach that future.

Eval everything.

---

If working towards this future sounds fun, I'd love to chat with you. We're solving this problem at The Context Company and love working with like-minded people.