As clinical AI systems become integral to patient care and healthcare operations, organizations’ AI Governance duty to prove that their systems are safe, effective, and legal require robust evaluation and continuous monitoring. This session covers:

• Key challenges in effective evaluation of medical AI solutions include data contamination, fragile metrics, benchmarks that lack resemblance to real-world use cases, and the need for localize validation.
• Legal requirements including national and state laws, regulations, industry standards, and contractual obligations that require bias, safety, and quality evaluation.
• Medical red teaming gaps that must be addressed to uncover ethical, safety, and security vulnerabilities unique to healthcare applications.
• Clinical cognitive biases such as anchoring, framing, and sycophancy that impact LLMs as well as humans and can skew AI‑driven recommendations.
• MLOps requirements for continuous testing and monitoring, to ensure models remain safe, compliant, and clinically reliable as regulations and data drift evolve.

We’ll explore emerging open-source libraries like MedHELM and LangTest, new research datasets like MedSafetyBench and MedAgentBench, and evolving standards from CHAI and NIST aiming to help address these challenges. Whether you’re building EHR‑powered chatbots or diagnostic decision‑support tools, this session will equip you with the knowledge and tools available today to deliver effective AI Governance for medical applications.