Who Judges the Judge?
ππ» Thanks for coming to my talk at the She Bytes Back AI Series (18 Nov)!
Who Judges the Judge? What breaks when the thing checking your agent is also an AI is the story of how our team built online evals for a customer-facing AI assistant, and what we learnt when it came to trusting the LLM judge itself.
π Slides: coming soon, shortly after the talk.
π Read more
- How we supervise AI agents at Personio: our first post on the Personio tech blog.
- π§ Coming soon: more deep dives are planned on the Personio tech blog, including:
- how our AI platform is structured, and the trade-offs weβve made building it
- which measures are actually worth tracking
- a closer look at how we run online and offline evaluations for one of our products
Follow the Personio tech blog to catch them when theyβre out.
π§ͺ Eval resources I used to get started
- Your AI Product Needs Evals by Hamel Husain. Start here. Why evals matter, and why looking at your own data comes before any metric.
- Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge) by Eugene Yan. Read this before you build a judge. A practical survey of what works (and what doesnβt) when an LLM does the grading.
- Who Validates the Validators? by Shreya Shankar et al. The research version of this talk. How to align LLM judges with human preferences, and why your criteria keep shifting as you look at more outputs.
- Hex engineering blog. For evals in practice. Helpful for thinking about evals in general, and especially for evaluating AI analytics features like the ones we build.
ππ»ββοΈ Find me
- βπ» Coding in Tune, my Substack: music, machine learning, and the ideas in between
- πΌ LinkedIn
- π©π»βπ» GitHub
- π More about me