How can advanced agentic AI systems like Deep Research be customized on private enterprise data with just a few labeled examples? How can we evaluate and optimize these modular, non-differentiable systems to build high performance reasoning systems for complex tasks?

In this talk we will outline the architecture of this class of agents, show how LLM-as-a-judge can be used for Elo based evaluation in the absence of gold standard benchmarking data sets, and explore how sparse evaluation signals can be used for end-to-end optimization of AI agents to improve performance.