Workshop: Mastering AI Observability

Your agent is in production. Now you need to understand what it is doing, where it is failing, and what to improve next. In this hands-on workshop, you’ll use Braintrust to turn production traces into targeted evals. We’ll use Topics to surface patterns across real agent traces, identify the issues worth investigating, and build custom scorers to measure them directly. Then you’ll update your agent, rerun your evals, and compare the results in Braintrust. This is active observability: using production behavior to decide what to test, fix, and ship next. You’ll leave with a repeatable workflow for turning agent behavior into evals, measuring improvements, and catching regressions before users do. What you’ll build: A quality analysis using Topics A custom scorer for a real failure pattern An improved agent with traces and eval results to prove it Format: Guided hands-on workshop Level: Intermediate, prior experience with AI evals needed Bring: A laptop