Tagged “production”

A CLI tool that asks a decision model which files are relevant instead of pattern-matching for them. The benchmark numbers come with a real caveat worth reading before you trust them.

OpenAI's Decisions API returns predicates, choices and scores instead of free text. What's confirmed about it, how it differs from Jev under the hood and what I'd do about it in a production system.

TypeSafe AI's Jev returns typed answers with calibrated probabilities to a fixed set of questions and generates no tokens. How it works and where I'd put it in a production system.

Why my thesis evaluation pipeline scored every run with both a deterministic benchmark and an LLM judge. The disagreements between them turned out to be the useful part.

In my thesis work the model only routes questions to the right data partition and pushes predicates down to PostGIS and Neo4j. Inference token cost dropped 3x and routing-index quality ended up deciding answer accuracy.

A field guide to the failure modes that don't show up in tutorials and how to systematically debug them.