Rendered at 17:15:01 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
TeMPOraL 9 hours ago [-]
Now this is how[0] we get some of the most magical Star Trek technology that eludes us to this day, such as automatic doors. Because if you notice, they work much, much better than real-life ones, because they seem to be doing something like this:
if(within 10 meters of door then) {
if(Jev(
[A] Intends to go through, expects doors to open
[B] Approaches with no intent to pass
[C] Passing by, loiters, or otherwise
[D] Other
) == most definitely A) {
// open doors, +/- identity/security/interlocks check
} else {
// ignore
}
}
Keywords: ambient awareness, understanding of intent.
Most interactive tech on Star Trek is like this - from phasers to consoles to communicators to voice interactions with the ship's computer. The computer seems to be aware of the user and surrounding, and actively infers intent from context, to DWIM ("do what I mean") and when they mean it, instead of doing dumb things[1] on simple triggers.
--
[0] - The direction, not final implementation - surely we can work out how to do it more efficiently than wrapping around final stage of LLM. But the point is, multimodal.
[1] - Obviously it's a fictional show, but in this, both Watsonian and Doylist explanations align near-perfectly: this is/portrays advanced technology, that Just Works and doesn't do stupid shit. Same intent recognition algorithm is there - fictionally in the computer, in reality in the minds of on-set technicians.
ricardobeat 7 hours ago [-]
To do this you need video/motion understanding, the intent cannot be judged from still images or state descriptions.
We’ve had the tool to do this since mid 2025, V-JEPA2 [1], Yann Lecun’s last work at Meta.
It runs at several FPS on a macbook and can even be trained locally. Chaining it with Jev for decision-making would probably work great!
"doing dumb things" and "stupid shit" is an odd choice to describe tools that only trigger on explicit activation. Is a windshield being lowered by a switch being held a "dumb thing"?
Dumb things start to happen when you try to build Star Trek interfaces. When you build DWIM interfaces in real life, they are annoying and trigger unwanted and the implementation is without exception, by necessity, a growing ball of spaghetti.
TeMPOraL 12 minutes ago [-]
> Dumb things start to happen when you try to build Star Trek interfaces. When you build DWIM interfaces in real life, they are annoying and trigger unwanted and the implementation is without exception, by necessity, a growing ball of spaghetti.
This is what I'm talking about.
"Growing ball of spaghetti" happens because system cannot recognize intent. That problem, itself, was something most engineering teams still seem to fail to recognize.
Automated doors are my favorite example, because the "simple solution" is ubiquitous and wrong and we got used to it, and complex solution is usually leading people the wrong path. In short:
Current doors: if(user triggers proximity detector) { open(); }
Failed attempt at DWIM: if(user triggers proximity detector && this && not that && except when ...) { open(); }
Star Trek: if(user intends to walk through the door) { open(); }
LLMs are the first tool we have that allow us to infer user intent directly, and use that as an input.
And recognizing intent itself cannot be done with a single sensor. It requires both general understanding of how humans behave, and awareness of surrounding and subjects - their movements and behavior, as well as who/what they are, and what they are doing.
moregrist 8 hours ago [-]
> Most interactive tech on Star Trek is like this - from phasers to consoles to communicators to voice interactions with the ship's computer.
Almost like the Star Trek mechanisms can infer perfect intent.
Like there’s a hidden script or something.
More seriously, I think there’s real value in an automatic door that behaves consistently rather than one that tries to infer messy human intent. Real life isn’t a TV show and there’s both ambiguity in how people behave and how they even intend to behave. It’s mostly not hard to understand how a proximity sensor door will function. Using a black-box classifier to improve that won’t necessarily make people like it more. And calling up to the cloud for every sensor event, ignoring privacy issues, adds weird latency and a huge failure mode during data center outages.
philbo 57 minutes ago [-]
Spoken like a person who's never had to queue in a shop with an automatic door and then the queue reaches too close to the door and then you're the unfortunate person who keeps on accidentally opening the door while standing at the back of the queue and then everyone else in the queue glares at you.
7 hours ago [-]
4 hours ago [-]
rafaelmn 8 hours ago [-]
How does the model get context to decide ABCD ?
levmiseri 8 hours ago [-]
As described in the blog post – e.g. image via a camera feed.
prathje 9 hours ago [-]
Nice! I would love to use it for images as well.
Then again is using Grammar-Based Decoding with a json response not the same? Is Jev just that with nice caching?
Because then I have been using that already…
ulam2 7 hours ago [-]
Yes, that is my question too. Someone knowledgeable can comment
frabcus 9 hours ago [-]
Presumably this is much less good than Jev, because the normal LLM models have been trained with RLHF and to be agents. Especially on a large model, I'd expect it to decide in an earlier layer.
I'd hope whatever Jev's Reinforcement Learning for Calibrated Decisions (RLCD) does is better at training the models to give accurate probabilities in the weights.
jampekka 8 hours ago [-]
Empirically this approach is more accurate, faster and about the same price as Jev.
Ah sweet it’s like Jev but several order of magnitude more expensive, and slower too.
jampekka 8 hours ago [-]
Jev doesn't support images, so it's hard to compare this directly. But in general this approach beats Jev in its own benchmarks for accuracy and speed and is about the same price.
one day i was about the tell my girlfriend the wonders of ai and how it works underneath. She stopped me about 30 seconds in " so hotdog, not hotdog? " i was like "yep".
I never bought that up again.
CROON_tv 6 hours ago [-]
What I'd want to see next to accuracy is tail latency. In a real-time use, deciding when a spoken sentence is finished, a general LLM with the same prompt was slower and more hesitant for us than Jev, even though both cost about the same.
hatimmoxs 23 minutes ago [-]
That is the most intuitive way of doing it.
The hype is insufferable.
> Answer with the letter of the best option only [A, B, C]
What if it says "D"? What if it tries to say "Additional details needed"?
(Also no calibration, etc.)
jermaustin1 5 hours ago [-]
I don’t know how this wrapper works, but if it is like any of the classifiers I’ve had Claude build off an LLM in the past, it grabs the probabilities of the tokens you are looking for, and then computes their relative probs against each other.
Even if the LLM thinks it’s made up D is the highest probability, that isn’t part of the set.
You never actually generate the prose, only the first pass, and grab the probabilities. It couldn’t ask for more details even if it wants to. It gets stopped before the first token renders.
roger_ 1 hours ago [-]
I guess “A” by itself would be a seperate token but my point is the model might be trying to say something that begins with that letter rather than actually answering.
You’d need to use an approach that links the output to a closed set.
jermaustin1 1 hours ago [-]
A is significant in it's own right. But most models follow instructions well enough when you prompt it, "Choose one of the following answers:", it will follow that 90% of the time. Use temperature tuning, and a LoRA, and you are 99% of the way there, just without the speed that Jev has.
Yesterday, just to prove to a friend that Jev isn't that "revolutionary" I extracted some image classification code that Claude had written for my private image organizer that used Qwen3-VL, and stopped the output at a single token, then used the probabilities. Input processing on my GPU was somewhere around 1000ms per image, so not too fast, but each question used the prompt cache, so followups were 100ms-ish.
That was my baseline of an untrained, non-optimized single pass classification. It would connect to my llama-server, and use the logprobs for the choices.
After that, I had Claude remove llama-server from the solution, and write it directly to the transformers, then I kept prompting it to profile and find more speed. Eventually my "Decision Engine" running locally on a trained 1B model got to just under 85% accuracy across my 500 validation prompts (images and text) not used or derived from the training set, and an 8MB image, with 10 questions with 5 choices per question, got down to just under 500ms. Pure text prompts and questions are below 100ms for 300tokens + 10 questions + 5 choices per question (average).
It did better on text than images, just because my training set included 90% text. I'll do more training and validation for images when I get home, but for now, I'm more than happy that I can get a local "decision engine" running in 4GB of VRAM and responding in under 30ms for most use cases I've had.
tomerbarm 5 hours ago [-]
[flagged]
Havoc 11 hours ago [-]
Likely works even better with fireworks ai since they have proper grammar support
Of course it works, Jev is nothing but an API breakthrough
dist-epoch 9 hours ago [-]
Jev is rumored to be a 30B model, and it's input price is MUCH cheaper than similarly sized models. The maker is also heavily focused on having a profitable product, so it's unlikely to be subsidizing the cost, especially since they say they have more demand than what they can serve.
jampekka 8 hours ago [-]
You can get Gemma 4 26B A4B at the exact same token input price of $0.042/M. GPT-5 nano is not much more expensive at $0.05.
Exactly, imo it’s not even that cheap if you look into perspective and consider the fact that providers could subsidize the cost of cached input tokens to virtually zero if they would allow for a more flexible API (e.g. tree of message blocks instead of chain). Most of the cost is the infrastructure around keeping KV caches, estimating their lifetimes, etc. When mist people just want to run one context block with multiple subsequent variants of a second block in parallel. I still stand by my statement.
imtringued 7 hours ago [-]
That's an interesting point, if you send a batch with a shared prefix you basically only end up paying for the sequence length difference effectively.
There is still some minor memory bandwidth issue on outputting more tokens, but the truth is that if you process e.g. 16 messages at once you wont end up being much slower than Jev even though you have to perform several autoregressive passes.
bicsi 21 minutes ago [-]
"you wont end up being much slower than Jev" -- I would even go as far as saying that "you will end up being Jev".
_flux 5 hours ago [-]
With this cost, does it perform the same quality and speed as Jev?
I'm quite interested in this; my current understanding is though that Jev is great when scored with response quality and latency metrics.
jampekka 5 hours ago [-]
Seems to be slightly higher quality and substantially faster, although this compares remote API vs local deployment.
The prevalent idea of Jev's superiority in price, speed and accuracy seems to come from TypeSafe's marketing and their, I'd say even bad faith, benchmarking. In independent benchmarks the relative numbers tend to be very different.
They can't overturn the economics of attention by restricting themselves to a single token output.
Sure they are no longer memory bandwidth bound thanks to that but someone could add a similar projector to a conventional model, train with a Jev style dataset and call it a day.
Whatever they are doing on inputs must either mean they intentionally chose a Mamba successor or they suffer from the same compute costs as everyone else.
dist-epoch 6 hours ago [-]
Jev claims 70-500 ms latency, including for the first request. This requires some clever engineering at least, which will take a little to duplicate.
Maybe first request is unbatched, to have fast prefill, and the subsequent ones are batched.
They also don't restrict your prompt. You can have a dumb one, where you put the variable data at the front, and the details on how to process it at the back, thus you bust the user-part of the KV cache every request.
Most interactive tech on Star Trek is like this - from phasers to consoles to communicators to voice interactions with the ship's computer. The computer seems to be aware of the user and surrounding, and actively infers intent from context, to DWIM ("do what I mean") and when they mean it, instead of doing dumb things[1] on simple triggers.
--
[0] - The direction, not final implementation - surely we can work out how to do it more efficiently than wrapping around final stage of LLM. But the point is, multimodal.
[1] - Obviously it's a fictional show, but in this, both Watsonian and Doylist explanations align near-perfectly: this is/portrays advanced technology, that Just Works and doesn't do stupid shit. Same intent recognition algorithm is there - fictionally in the computer, in reality in the minds of on-set technicians.
We’ve had the tool to do this since mid 2025, V-JEPA2 [1], Yann Lecun’s last work at Meta.
It runs at several FPS on a macbook and can even be trained locally. Chaining it with Jev for decision-making would probably work great!
[1] https://ai.meta.com/research/vjepa/
Dumb things start to happen when you try to build Star Trek interfaces. When you build DWIM interfaces in real life, they are annoying and trigger unwanted and the implementation is without exception, by necessity, a growing ball of spaghetti.
This is what I'm talking about.
"Growing ball of spaghetti" happens because system cannot recognize intent. That problem, itself, was something most engineering teams still seem to fail to recognize.
Automated doors are my favorite example, because the "simple solution" is ubiquitous and wrong and we got used to it, and complex solution is usually leading people the wrong path. In short:
Current doors: if(user triggers proximity detector) { open(); }
Failed attempt at DWIM: if(user triggers proximity detector && this && not that && except when ...) { open(); }
Star Trek: if(user intends to walk through the door) { open(); }
LLMs are the first tool we have that allow us to infer user intent directly, and use that as an input.
And recognizing intent itself cannot be done with a single sensor. It requires both general understanding of how humans behave, and awareness of surrounding and subjects - their movements and behavior, as well as who/what they are, and what they are doing.
Almost like the Star Trek mechanisms can infer perfect intent.
Like there’s a hidden script or something.
More seriously, I think there’s real value in an automatic door that behaves consistently rather than one that tries to infer messy human intent. Real life isn’t a TV show and there’s both ambiguity in how people behave and how they even intend to behave. It’s mostly not hard to understand how a proximity sensor door will function. Using a black-box classifier to improve that won’t necessarily make people like it more. And calling up to the cloud for every sensor event, ignoring privacy issues, adds weird latency and a huge failure mode during data center outages.
I'd hope whatever Jev's Reinforcement Learning for Calibrated Decisions (RLCD) does is better at training the models to give accurate probabilities in the weights.
https://github.com/Mushroom-Systems/lichen
https://github.com/Mushroom-Systems/lichen
[B] Not a hotdog
I never bought that up again.
What if it says "D"? What if it tries to say "Additional details needed"?
(Also no calibration, etc.)
Even if the LLM thinks it’s made up D is the highest probability, that isn’t part of the set.
You never actually generate the prose, only the first pass, and grab the probabilities. It couldn’t ask for more details even if it wants to. It gets stopped before the first token renders.
You’d need to use an approach that links the output to a closed set.
Yesterday, just to prove to a friend that Jev isn't that "revolutionary" I extracted some image classification code that Claude had written for my private image organizer that used Qwen3-VL, and stopped the output at a single token, then used the probabilities. Input processing on my GPU was somewhere around 1000ms per image, so not too fast, but each question used the prompt cache, so followups were 100ms-ish.
That was my baseline of an untrained, non-optimized single pass classification. It would connect to my llama-server, and use the logprobs for the choices.
After that, I had Claude remove llama-server from the solution, and write it directly to the transformers, then I kept prompting it to profile and find more speed. Eventually my "Decision Engine" running locally on a trained 1B model got to just under 85% accuracy across my 500 validation prompts (images and text) not used or derived from the training set, and an 8MB image, with 10 questions with 5 choices per question, got down to just under 500ms. Pure text prompts and questions are below 100ms for 300tokens + 10 questions + 5 choices per question (average).
It did better on text than images, just because my training set included 90% text. I'll do more training and validation for images when I get home, but for now, I'm more than happy that I can get a local "decision engine" running in 4GB of VRAM and responding in under 30ms for most use cases I've had.
https://openrouter.ai/google/gemma-4-26b-a4b-it
There is still some minor memory bandwidth issue on outputting more tokens, but the truth is that if you process e.g. 16 messages at once you wont end up being much slower than Jev even though you have to perform several autoregressive passes.
I'm quite interested in this; my current understanding is though that Jev is great when scored with response quality and latency metrics.
The prevalent idea of Jev's superiority in price, speed and accuracy seems to come from TypeSafe's marketing and their, I'd say even bad faith, benchmarking. In independent benchmarks the relative numbers tend to be very different.
https://github.com/Mushroom-Systems/lichen
Sure they are no longer memory bandwidth bound thanks to that but someone could add a similar projector to a conventional model, train with a Jev style dataset and call it a day.
Whatever they are doing on inputs must either mean they intentionally chose a Mamba successor or they suffer from the same compute costs as everyone else.
Maybe first request is unbatched, to have fast prefill, and the subsequent ones are batched.
They also don't restrict your prompt. You can have a dumb one, where you put the variable data at the front, and the details on how to process it at the back, thus you bust the user-part of the KV cache every request.