Making decisions with Jev

Unless you live under a rock (i.e. are not terminally online), you must have heard of TypeSafe.ai’s new AI model Jev. Their annoucement spread quickly within the AI-hyped communities and less than a week later, countless blog posts, projects and applications have been built with this model. So, since it’s a grey Sunday here, I thought why not jump on a bandwagon for once!

What is Jev?

If you really want to understand Jev in details, I can really only refer you to their documentation page. The important thing to know is that Jev isn’t a new LLM model release like we’re used to, promising higher and higher intelligence.¹ Jev represents both a departure from the bigger-is-better paradigm, instead focusing on cheaper and faster, as well as a shift from the “AI as a chatbot” concept. In their lingo, this is a System One model, instead of the usual System Two we’ve become accustomed to.

The way I like to think about it, Jev is to ML classification what ChatGPT was to basic NLP. You can think of it as a generalized classification model trained on a lot of data. All Jev requires is a custom state (can be a string, a JSON object, or a list of those) and a bounded output space and Jev will return an answer. There are three types of questions you can answer with Jev:

• choices: give Jev a list of choices, and you’ll get a selected choice, with a confidence score and a probability for each of the choices,
• noul: to a yes/no question, Jev answers with a number between 0 and 1 indicating the probability Jev estimates for a yes,
• score: give Jev a scale definition and you’ll get a score on that scale, with a confidence level, a legend for each level and a probability of each step in the legend.

Examples

It’s a bit easier to illustrate all this with some examples. I’m running the below examples using the jev-cli tool. Let’s say we are responsible for a small town public services and we receive an email like this

Dear Sir stroke Madam.

Fire! Fire! Help me!
123 Carrendon Road.

Looking forward to hearing from you.
All the best,
Maurice Moss.

Okay so first we can figure out if this is an emergency?

$ jev noul -s email.txt -q "Is this email an emergency?" | jq '.answers'

{
    "answer": {
        "type": "noul",
        "noul": 0.68
    }
}

With a value of 0.68, Jev would rule it an emergency, but not confidently. If instead we use a more ‘natural’ way of phrasing this, you can see the model is a lot more confident that there is an emergency.

$ jev noul \
    -s "Fire! I need help!" \
    -q "Is this email an emergency?" | jq -r '.answers'

{
    "answer": {
        "type": "noul",
        "noul": 0.88
    }
}

Next we can also figure out which department is needed by giving Jev choices

$ jev choice -s "$(cat email.txt)" \
    -q "Which department should handle this request?" \
    -o police="Send the police" \
    -o fire="Send the firemen" \
    -o medics="Send the paramedics" | jq -r '.answers'

{
    "answer": {
        "type": "choice",
        "choice": "fire",
        "confidence": 1.0,
        "probabilities": {
            "fire": 1.0,
            "police": 0.0,
            "medics": 0.0
        }
    }
}

Phew, no doubt here the fire department is needed. If we tweak the state, you can see how other departments might also start showing up, for example paramedics if someone is injured.

$ jev choice -s "Fire! Fire! My brother is unconscious! Help me!" \
    -q "Which department should handle this request?" \
    -o police="Send the police" \
    -o fire="Send the firemen" \
    -o medics="Send the paramedics" | jq -r '.answers'

{
    "answer": {
        "type": "choice",
        "choice": "fire",
        "confidence": 0.79,
        "probabilities": {
            "police": 0.0,
            "medics": 0.14,
            "fire": 0.86
        }
    }
}

Finally, we can score the state somehow on a provided scale

$ jev score -s "$(cat email.txt)" \
    -q "How panicked is the sender?" \
    -l "Calm" -l "Anxious" -l "Full-blown panicked" | jq -r '.answers'

{
    "answer": {
        "type": "score",
        "score": 1.78,
        "confidence": 0.67,
        "legend": {
            "0": "Calm",
            "1": "Anxious",
            "2": "Full-blown panicked"
        },
        "probabilities": {
            "0": 0.05,
            "1": 0.11,
            "2": 0.84
        }
    }
}

This is interesting: Jev is clearly missing the joke here. The email is not really supposed to convey panic, yet Jev places it solidly close to ‘Full-blown panicked’. My guess, it’s the exclamation marks. Let’s see if we dramatically tone it down

$ jev score -s "$(cat <<'EOF'
Dear Sir/Madam, there is a fire in this office.
Please come at your earliest convenience.
Best regards, Maurice Moss.
EOF
)" \
    -q "How panicked is the sender?" \
    -l "Calm" -l "Anxious" -l "Full-blown panicked" | jq -r '.answers'

{
    "answer": {
        "type": "score",
        "score": 0.06,
        "confidence": 0.92,
        "legend": {
            "0": "Calm",
            "1": "Anxious",
            "2": "Full-blown panicked"
        },
        "probabilities": {
            "0": 0.95,
            "1": 0.05,
            "2": 0.0
        }
    }
}

Bingo!

Why use Jev?

Hopefully, after all these examples, it should be pretty clear how Jev works and what kind of questions/answers you can use it for. What should also be pretty clear here is that all of this could already be done using either a fine-tuned classification model or your favourite LLM. What’s the added value here?

From a conceptual point of view, Jev’s architecture brings back type safety and confidence levels to the foreground, which is a welcome departure from the usual LLM paradigm of working with free-form text everywhere. You don’t need to parse Jev’s output, it can’t hallucinate a malformed output, and its reinforcement training is focused on getting its confidence level right rather than its accuracy.

However, the real marketing ace in the hole is its price and inference time. Per TypeSafe’s own numbers, Jev is 40-200x faster than traditional LLM inference while being close to 500x cheaper. What this means is you can use Jev for real-time applications requiring many decisions per second and still come ahead on your budget.

The cherry on top of the cake is that you can batch as many questions as you want per API call. Jev’s architecture will handle these in parallel, unlike an LLM which would require independent calls for each of them.

On the other hand, the context size is much smaller (~10-100x smaller than typical LLMs) so Jev is limited to small isolated questions right now. And of course it can only handle cases where the output is part of a bounded space, by construction.

Example project: a code-quality gate CLI tool

To try it out, I made a CLI tool which analyses code quality using custom rules. The idea here is to fill the void between automated linting rules and full-blown LLM/human code review. Best practices, project-specific no-no’s, underperforming code: all these can be identified by Jev simultaneously and returned as a report, either to you or your agent or whoever writes your code these days.