10,000 Skittles sorted one at a time: Jev vs Claude Sonnet 5

10,000 Skittles, sorted by colour one at a time. Each one is a short description that never names the colour, like 'coloured as a stop sign is', so the model has to actually decide. Two models, the same 10,000 items in the same order, one item per request, 20 requests in flight on both sides.

Jev is TypeSafe's new decision model. It does not write text; it returns one of the options you give it, with a probability. It cleared the bowl in 3m41 for 15 cents. Claude Sonnet 5, with reasoning switched off so it was as fast and cheap as it can be, took 17m55 for $1.86. That is 4.9 times faster and 12 times cheaper.

Jev got 300 of the 10,000 wrong and Sonnet got 57, so Jev was 2.4 points less accurate. After posting I went back through the misses, and most of Jev's came from one description I wrote. I now think that was a faulty input on my side rather than the model, and I am re-running the test. Until that is out, treat the accuracy gap as unsettled.

They answer on different endpoints, so this compares two products, not two identical setups.