BRKS216 - Measuring and Managing Healthcare AI Performance Long-Term

Course Description

After an organization deploys an AI tool at scale, the launch plan ends, the go-live celebrations are over, and the vendor has left the room. Now comes the harder question: Is the tool still working, and how would you know?

A launch plan typically accounts for milestones at 30, 60, and 90 days; Day 91 has no end date. The patient population shifts, the model ages, adoption erodes, and the data that would warn you is scattered across systems, departments, and people, each of whom owns a piece of it. Monitoring a scaled AI deployment is two tracks at once: the technical work of watching model performance, clinical outcomes, and operational impact, and the human work of learning what stakeholders are seeing that the numbers don't show.

In this course, you will use a fictionalized case study to map where monitoring data lives in an organization, who owns it, and where the gaps are. You’ll define the metrics behind technical, clinical, and operational monitoring dashboards: what is measured, the thresholds that trigger action, and who owns each number. You’ll socialize your monitoring strategy with stakeholders and decide what to measure, what to defer, and what to cut. The result is a continuous monitoring and optimization plan your organization can operate.

By working through a fictionalized healthcare case, you will develop the skills and frameworks to build a monitoring plan for your own organization and keep a deployed AI tool safe, effective, and worth its cost.

You are required to have completed the following courses or have equivalent experience before taking this course:

• Examining the AI Landscape in Healthcare Delivery
• Designing Clinical Workflows for AI Adoption
• Building an AI Governance Framework for Healthcare
• Evaluating Costs, Contracts, and ROI for Healthcare AI
• Scaling AI Implementations in Healthcare Beyond the Pilot Stage

Faculty Author

Benefits to the Learner

• Review a completed 30-60-90-day scaling plan to identify what must be operationalized and where friction will arise; map where monitoring data lives in an organization, who owns it, and where the gaps are; and evaluate technical monitoring for a scaled AI tool
• Design clinical and operational monitoring for a scaled AI tool; socialize it with the stakeholders who will live with it; respond to their feedback by deciding what to measure, defer, or cut; and assemble the complete continuous monitoring and optimization plan

Target Audience

• C-suite and senior healthcare executives (CEOs, CMOs, CIOs, CDOs, CNOs, COOs) responsible for strategic AI investment, governance, and enterprise-wide adoption
• VPs and directors of digital health, clinical operations, and innovation leading AI pilot design, workflow transformation, and scale-out execution
• Hospital and health system administrators overseeing operational transformation and resource allocation for AI initiatives
• Physician leaders and clinician-executives (medical directors, department chairs) serving as clinical champions and bridging practice with technology strategy
• Health IT and informatics professionals (directors of health IT, clinical informaticists, CTOs) moving from technical implementation into strategic and governance roles
• Healthcare consultants and strategy advisors serving delivery organizations who need credentialed expertise in AI procurement, governance, and ROI
• Regulatory, compliance, and legal professionals navigating HIPAA, FDA classification, algorithmic bias, and evolving U.S. state-level AI legislation
• Healthcare project and program managers, quality improvement directors, and technical product managers leading cross-functional AI implementation and outcomes measurement

Applies Towards the Following Certificates

• AI Strategy in Healthcare : Core Courses