Notes from building the AI Operator Framework, shipping products, and designing open standards.
The Envelope Fidelity Score is a number between zero and one that measures how much meaning survived when data moved between AI systems and the software those systems talked to. One means nothi...
Trust is not something an AI system has by default. Every AI system a team ships is trustworthy up until the moment it produces its first wrong answer to a user. That moment happens sooner than...
An AI system's accuracy on the day it ships is the highest accuracy it will ever have. Not because the model gets worse. The model is deterministic-ish under load. The world it operates in is n...
People ask me why we need CORD-AEO. The answer is easier to show than to explain. Open ChatGPT. Or Claude. Or Perplexity. Ask any of them a specific question about your business. If your busine...
An evaluation harness is the piece of software that runs benchmarks against language models in a reproducible way. Before harnesses, every research paper had its own evaluation code. Two teams...
Framework · Sep 18, 2026 · 10 min read LLM Evaluation Metrics That Actually Matter Every evaluation framework ships with dozens of metrics. Mo...
Hallucinations are the failure mode where a model produces confident text that is not supported by evidence. The model is not lying in any meaningful sense. It is doing what it was trained to d...
AI observability is the practice of watching every step an AI system takes so you can debug it when it produces the wrong answer. Traditional observability answers questions like: is the server...
A RAG system can score 0.95 faithfulness on the eval set and still ship wrong answers to production. The reason is almost always that the eval only measured the generator. The retriever failed silently. This is the field guide for evaluating both stages honestly.
Agentic RAG puts the model inside the retrieval loop instead of running retrieval once and hoping. Standard RAG is a straight line. Retrieve once, generate once, done. That works when the answer...
Every team ships the same RAG demo on a Saturday. Three months later it is quietly failing in production, and nobody can say why. The arc is identical everywhere. A developer chunks som...
RAG is how you get an AI model to answer from your data instead of its memory. A language model knows what it was trained on. It does not know your contracts, your product docs, your support hist...