Allen Hutchison
The bug fix took an afternoon. The follow-up question took a week. I was deep in Gemini Scribe, my Obsidian plugin that drops a Gemini-powered agent into your vault, and I had just shipped a change to the way the agent picked its tools. It felt better. The few sessions I ran by hand showed...
Amir Teymoori
Customer support is one of the best use cases for AI agents. It’s also one of the easiest places to ship a bad one. A support agent can sound confident and still quote the wrong refund policy. It can answer in 800ms and still miss the customer’s real issue. It can retrieve the right document...
Alex Chernysh
A practical blueprint for legal QA, shaped in part by work around the Agentic RAG Legal Challenge: document identity, hybrid retrieval, structured answers, page-level grounding, telemetry, and evals. Read full article →
Alex Chernysh
Practical guidance on tool contracts, context engineering, evals, approvals, and telemetry. Read full article →
Alex Guglielmone Nemi
If you can’t measure it, you’re guessing. Here’s how I think about evals, practical examples at ai-evals.io. Start with pain, not tooling My eval approach is pain-point driven: I can’t compare what I can’t measure. I can’t trust an AI system to run on its own if I can’t quantify failure. If...
Alex Chernysh
LLM evals for continuous delivery: turn production failures into automated tests, grade traces with task-specific graders, and block bad releases with eval-driven gates. Read full article →
Alex Guglielmone Nemi
GenAI is non-deterministic and can fail or produce different results for the same input. A typical prompt-to-action flow involves many LLM calls. Each call is a chance for the model to misinterpret, hallucinate, or produce an unusable output. The question isn’t if errors happen. It’s what happens...
Amit Chaudhary
Synthetic data is a popular approach for bootstrapping an initial dataset when building LLM-based applications. We can find practical examples of synthetic data usage in the wild such as: Generating synthetic user queries from existing documents to evaluate RAG systems 1 Producing fake meeting...
Amit Chaudhary
Most software products we encounter today have some form of search functionality integrated into them. We search for content on Google, videos on YouTube, products on Amazon, messages on Slack, emails on Gmail, people on Facebook, and so on. As users, the workflow is pretty simple. We can...
Amit Chaudhary
When developing an NLP model, it’s a standard practice to test how well a model generalizes to unseen examples by evaluating it on a held-out dataset. Suppose we reach our target performance metric of 95% on a held-out dataset and thus deploy the model to production based on this single metric....