Jev and open System One models

I tested whether typed decision models can do a few small semantic jobs that an ecommerce search stack might turn into features. Each model takes a compact description of a query or a product pair, answers bounded questions, and returns a label with probabilities. They do not replace retrieval or ranking.

The work followed an essay on Jev, TypeSafe's hosted System One model. This project is the test I then ran. It is not a product, and it is not a leaderboard with one winning score.

What was measured

Four tasks, kept separate because they fail in different ways:

• Whether an accessory actually fits a specific product, which is not the same as whether the two are often bought together.
• What a query is trying to do, what it refers to, how specific it is, and whether it belongs in a product catalog at all.
• Brand and category, judged against a short list of candidates that was already retrieved.
• ESCI relevance on a balanced slice of Amazon's public human labels: Exact, Substitute, Complement, and Irrelevant.

Compatibility, query, and brand/category labels are agreement between two language-model judges. Disagreements were left unscored. ESCI uses the existing human labels. A constant majority-class predictor is included, because on some of these tasks a high accuracy only reflects an uneven label mix.

What ran

The same frozen questions went to a majority prior, Laya, hosted Jev (jev-1.13.0), and Kev-0.8B on a laptop, then to Kev-4B and JevLite on a Colab A100. Every run was zero-shot: no examples in the prompt and no fine-tuning. Accuracy, macro-F1, calibration, latency, memory, and cost are reported separately. There is no NDCG, click, or conversion result.

The published comparison is that the open models are not interchangeable with each other or with Jev. JevLite is close on some fields and not on others. Kev-0.8B is mainly usable on the brand shortlist. Laya is fast and weak on these schemas. The full numbers, and the limits of that reading, are in the two posts linked from this page.

Where it stands

The zero-shot comparison is finished. The harness, rubrics, and result summaries are in the public book repository. Few-shot prompts, a trained student model, hierarchical category classification, and any measurement of ranking impact were left for later and have not been run.