Epic Releases Open-Source AI Validation Tool for Health Systems
Healthcare organizations are seeing a major increase of artificial intelligence (AI) features, which have the potential to improve clinician productivity and ultimately patient outcomes. Although predictive AI modes have been used for nearly a decade, generative AI and large language models (LLMs) are quickly emerging. In the healthcare arena, Generative AI can augment diagnostic exams, help research and develop new drugs, build patient medical records, and provide customized patient treatment plans. LLMs are similar to generative AI, except LLMs process large amounts of data to produce outcomes. For adoption of this technology, it is still a challenge to use the best methods in which to validate AI models for accuracy, consistency and safety.
Epic has released an open-source tool to allow healthcare organizations to test and monitor artificial intelligence (AI) models. Epic recognizes evaluation criteria for AI has to be meaningful for each individual healthcare organization: “the standards need to be implemented correctly and consistently and used to test models on the local patient populations and embedded in the specific workflows that will be used.” As such, this tool is free to the public on GitHub, and can be used with existing EHR systems (not just Epic) to validate and monitor an AI model’s performance on a continual basis. The goal is to provide consistency, while eliminating the need for healthcare organization data mapping.
In an interview with Seth Hain, Epic Senior Vice president of Research and Development, Hain expressed, “We’ll provide health systems with the ability to combine their local information about the outcomes around their workflows, alongside the information about the AI models that they’re using, and they will be able to use that both for evaluation and then importantly, ongoing monitoring of those models in their local contexts.”
