How to Measure AI Accuracy in Production
An AI system's accuracy on the day it ships is the highest accuracy it will ever have. Not because the model gets worse. The model is deterministic-ish under load. The world it operates in is n...
Notes from building the AI Operator Framework, shipping products, and designing open standards.
An AI system's accuracy on the day it ships is the highest accuracy it will ever have. Not because the model gets worse. The model is deterministic-ish under load. The world it operates in is n...
Framework · Sep 18, 2026 · 10 min read LLM Evaluation Metrics That Actually Matter Every evaluation framework ships with dozens of metrics. Mo...
Hallucinations are the failure mode where a model produces confident text that is not supported by evidence. The model is not lying in any meaningful sense. It is doing what it was trained to d...
A RAG system can score 0.95 faithfulness on the eval set and still ship wrong answers to production. The reason is almost always that the eval only measured the generator. The retriever failed silently. This is the field guide for evaluating both stages honestly.
The hardest part of advocating for infrastructure work is that the case looks weakest when the timing is best. When your AI program is small, the contract layer feels premature. The systems are...
If you are a CTO, VP of Engineering, or Head of AI evaluating whether to invest engineering time in an AI data contract layer, this post is for you. The frame is leadership-to-leadership. The argu...
Most AI systems are built as if they will run alone forever. Then six months in, someone needs them to talk to another AI system. The retrofit is harder than the original build. This post is th...
If you read the visibility stack post and accepted the three-layer model, the next question is the one I get in DMs every time. What do I actually do. This is that post. The 90-day playbook. Th...
Every leadership team I work with has the same problem. There are more AI use cases on the table than there is budget, attention, or organizational capacity to execute. The filter that decides whi...
An AI Operator is the dedicated role responsible for designing, governing, and scaling artificial intelligence inside a business. Not every company needs one yet. The signals that say it is time are...
The AI Operator has five core functions: Design, Govern, Scale, Measure, and Own. Each function exists because AI inside a business has to be designed before it ships, governed as it runs, scaled acro...
The artificial intelligence operator's toolkit is a focused set of capabilities, not an inventory of products. It includes: a model access layer, a workflow orchestration layer, a data envelope stan...