I Tested Laya, the Viral 421M "System 1" Model, Against My Family's Finances. It Lost 18 to 2.

An honest experiment with the open-source decision model everyone is talking about — deployed on my own server, measured against real data, no cherry-picking.

The hype

This week my feed exploded with Laya: a non-autoregressive decision model from Convai Innovations, open-source (Apache 2.0), 421 million parameters, promising structured answers in ~33 ms without generating text. The head-to-head against TypeSafe AI's Jev — 7.8× faster, free, self-hostable — made it the story of the week.

The premise is seductive: "not every AI problem requires an autoregressive chatbot." For classification, routing, and guardrails, a bidirectional encoder returning calibrated probabilities in a single forward pass sounds like the right tool.

But there was a detail buried in the author's own documentation that almost nobody quoted:

"Laya is a fast base to specialise, not a zero-shot decision engine."

I had the perfect testbed to verify that sentence. Here's the result.

The use case: my family's financial command center

My homelab runs two servers on my LAN hosting what we call the Family Wealth Command Center: a FastAPI + HTMX dashboard consuming a live NocoDB database with my family's finances — 85 transactions, 56 custom categories in Spanish (from "Weekly Supermarket / Groceries" to "Kids' Workshops / Extracurriculars"), 12 dimensions, accounts in two currencies.

A second server runs Hermes (NousResearch hermes-agent): I message it on Telegram — "Tottus 243.28 weekly groceries" — and it logs the expense into NocoDB.

The weak link: categorization. Hermes, like every LLM, is creative — one day it files something under "Food", the next under "Restaurants". My dashboard needs taxonomic consistency, not creativity. Laya promised exactly that: deterministic decisions against a closed list of options.

The experiment

Deployment

Laya runs in a Docker container on my second server (Ryzen 5 4600G, 14 GB RAM, CPU-only). A small FastAPI microservice exposes a single POST /clasificar endpoint that takes a transaction and returns dimension + category + calibrated confidence. The 56 categories sync from NocoDB at startup — add a new category and the classifier picks it up automatically.

Since 56 options exceed the model's practical limit (~20 options per question, per the author's own docs), I implemented the recommended hierarchical pattern: first pick the dimension (12 options), then the category within that dimension.

The measurement

Simple, trap-free design: I took 20 real transactions I had already categorized myself (the ground truth), fed them to Laya without revealing the category, and counted exact matches.

The results

Ten percent. On a task where picking the dimension at random already gives you ~8%.

The failures are instructive:

• "Fruit purchase" → predicted Weekend Food Delivery (actual: Fresh Market)
• "Helloween concert ticket" → predicted School Enrollment & PTA Fees (actual: Movies, Events & Entertainment)
• "Dinner at Siete Sopas" → predicted Personal Care & Barbershop (actual: Restaurants & Family Outings)

These aren't absurd errors — the model understands fruit is food and dinner is an outing. What it can't do is map that onto my specific taxonomy, with my local merchants (Tottus, Pastipan, Siete Sopas), in Spanish, across 56 options it never saw in training.

The unexpected finding: calibration actually works

Here's what almost nobody is measuring about this model. Look at the reported confidence:

• When it was right, it reported confidence of 0.72 and 0.78.
• When it was wrong, confidence was almost always < 0.65 (failure mean: ~0.3).

The RLCD training (RL against strictly proper scoring rules) delivered what it promised: the model knows when it doesn't know. A 0.7 threshold would have filtered nearly every error and escalated to a human — exactly the "act-or-escalate" pattern it was designed for. In production, a classifier that fails 90% of the time but admits it is infinitely safer than an LLM that fails 10% with absolute confidence.

Why it failed (and why it's not the model's fault)

1. Zero-shot ≠ what the headlines sell. The 0.766 accuracy making the rounds belongs to the checkpoint fine-tuned on that benchmark's own train split. The base checkpoints score ~0.35 on it — near random. The author states this explicitly; the hype omitted it.
2. My taxonomy is fully out-of-distribution. 56 custom categories in Spanish, with family scopes ("Mom", "Kids") that exist in no training corpus.
3. Fine-tuning wasn't realistic for me (yet). Laya needs dozens of examples per category; I have ~85 transactions spread across 56 categories — 1–2 examples each. In a few months, with more history, it'll be viable. Not today.
4. Few-shot in the criteria didn't help — it actually hurt (5%). Decision encoders aren't LLMs: stuffing examples into the criteria text dilutes the signal instead of guiding it.

What I did instead

I went back to the right architecture for my volume (a handful of transactions per day):

• Hermes (the LLM) keeps classifying, but now with the closed 56-category taxonomy in its system prompt and a strict instruction to never invent categories outside the list. LLMs are genuinely good at zero-shot classification — this is their task.
• Duplicate detection with a simple NocoDB query before writing. No AI needed.
• Laya stays deployed on the server. Once I have enough history to fine-tune it (or to leverage the typed-decisions checkpoint), the microservice is ready and the benchmark script is in place to measure again.

The lessons

1. Viral benchmarks are measured on the benchmark's distribution. Your real problem is a different distribution. Always measure against your own data, even if it's 20 rows.
2. "Open-source and 7.8× faster" doesn't answer the right question. The question is: what's the accuracy on my task, untrained?
3. Honest calibration is a feature, not a detail. It's what impressed me most about the model and what's discussed least.
4. A well-measured negative result saves months. One afternoon of experimentation saved me from shipping a classifier that would have silently degraded my financial reports.
5. System 1 and System 2 don't compete — they complement each other. For high volume with fine-tuning, Laya will be the right layer. For five expenses a day with an exotic taxonomy, the LLM wins today.

The full stack (FastAPI + NocoDB + Docker + Hermes) runs on two home servers on my LAN. The Laya microservice, the benchmark script, and the raw results are available to anyone who wants to replicate the experiment. If you also tested Laya against a real use case, I'd love to read your numbers in the comments.