probrSee if there's a fit
Notes from the field

The blog.

Evaluation methods, failure analysis, and post-training lessons from work with teams running models in production.

Offline benchmarks rarely predict how a model behaves once real users, real edge cases, and real data drift show up. Here's what we look for first.

Open full post in new tab

SFT, RLHF, RLVR — before picking a method, there are a handful of questions worth answering. Skipping them is the most common way teams waste a training run.

Open full post in new tab

Not every cluster of bad outputs deserves the same fix. Some are data problems, some are prompt problems, and some genuinely need retraining.

Open full post in new tab