For Seattle Children's Hospital, a big clinical hurdle revolved around efficiently accessing and applying the ever-expanding universe of evidence-based information in a timely and contextually relevant manner. While the hospital has always prioritized delivering the most up-to-date care, the sheer volume and rapid evolution of medical knowledge presented a significant challenge.
THE CHALLENGE
"Our dedicated pediatricians must navigate a landscape where medical literature doubles every few months, all while striving to critically evaluate each new study's rigor and relevance to the unique needs of their patients," said Dr. Clara Lin, vice president and chief medical information officer at Seattle Children's Hospital. "The need to stay abreast of the latest findings, coupled with the pressures of a growing pediatrician shortage and increasingly complex patient cases, underscored the urgency for a streamlined solution to the problem.
"Our commitment to evidence-based practice led to the establishment of the Clinical Effectiveness program in 2010," she continued. "This vital initiative, now led by Dr. Darren Migita within our Center of Quality and Patient Safety, involves a significant investment of time and expertise from numerous providers and scholars."
Together, they meticulously review medical literature to develop and maintain the Clinical Standard Work Pathways, or CSWs, for more than 70 diagnoses. These CSWs have proven to be invaluable in improving patient outcomes and are adopted by healthcare professionals globally.
"However, even with these well-curated pathways, the sheer volume of information – thousands of pages of text, images, flowcharts and tables representing the collective wisdom of our experts – could still present a time-intensive search process for a busy clinician seeking specific guidance for a particular patient scenario," she explained.
"The emergence of generative AI presented an exciting opportunity to explore a potential paradigm shift in how we access and use this critical information," she continued. "We saw the potential for AI large language models to act as powerful tools, capable of sifting through our extensive CSW knowledge base with remarkable speed and precision."
The core challenge was to determine if this technology could achieve the level of accuracy and reliability demanded by clinical healthcare. Could an AI agent effectively and efficiently extract the precise, evidence-based insights a provider needs at a crucial decision point, thereby reducing the cognitive load and time spent searching, and ultimately enhancing the delivery of optimal patient care?
This fundamental question drove Seattle Children's work with Google to develop and rigorously test this technology.
PROPOSAL
The proposal centered on leveraging the power of the LLM-based AI agent to act as an intelligent and efficient navigator through the vast repository of CSW Pathways. Staff envisioned the agent as a dynamic interface capable of understanding natural language queries from healthcare providers and then rapidly pinpointing the most relevant information within the extensive pathway documentation.
"Instead of a clinician having to manually sift through dozens of pages within a single pathway, the AI agent would serve as a targeted search engine, capable of extracting specific answers based on the nuances of the question asked," Lin explained.
"The key to alleviating the information access challenge lay in the agent's ability to process and understand the intricate structure and content of our CSW Pathways, which includes text, images, tables and flowcharts," she said. "Each pathway meticulously details the standardized care for a particular condition, encompassing every stage from initial presentation and diagnostic workup to the complete treatment plan and follow-up."
Staff proposed the AI agent would be fine-tuned to recognize the different sections and decision points within these pathways, allowing it to contextualize the user's question within the overall clinical workflow.
"Furthermore, we anticipated the AI agent would engage in a conversational dialogue with the user to ensure accurate information retrieval," Lin noted. "If the initial query from the provider lacked sufficient detail to precisely locate the relevant section within a pathway, the agent would proactively ask clarifying questions.
"This iterative process would help narrow down the search and ensure the information provided was highly specific to the pathways," she continued. "By mimicking a conversation with a 'virtual consultant,' the agent aimed to provide a more intuitive and efficient experience compared with traditional keyword/text searches, ultimately saving valuable time and reducing the cognitive burden on our healthcare providers when seeking evidence-based guidance."
MEETING THE CHALLENGE
To address the challenge, Seattle Children's worked with Google to create the Pathway Assistant, leveraging their Gemini models on Google Cloud's Vertex AI. This agent is specifically trained on Seattle Children's CSW Pathways – thousands of pages of text, images and flowcharts.
"Providers can ask Pathway Assistant questions about clinical scenarios in a conversational way," Lin explained. "The agent then quickly retrieves relevant information directly from the CSW Pathways. If the initial question isn't specific enough, the agent will ask clarifying questions to pinpoint the needed information.
"More than 50 Seattle Children's providers participated in testing Pathway Assistant, submitting more than 1,300 prompts to validate its accuracy," she continued. "This rigorous testing led to an accuracy rate exceeding 98% after multiple iterations. We now are developing training materials for wider use and plan to pilot the tool in some clinical areas soon."
Currently, Pathway Assistant operates independently and is not integrated with other applications or systems. This deliberate approach during the development and testing phase allowed staff to focus solely on the accuracy and reliability of its core function, without the complexities or the need of integrating patient-specific data or other clinical applications.
"As we prepare for pilot implementation in select clinical areas, our focus remains on empowering providers with immediate access to the collective knowledge embedded within our CSWs, without needing patient-specific data," she said.
RESULTS
While the healthcare organization's primary endpoint with the AI Pathway Assistant is not on reducing readmission rates, the broader CSW Pathways have a proven track record of improving overall patient outcomes. The focus for this AI initiative revolves around enhancing the accessibility and adherence to these evidence-based, standardized care pathways.
"We will be evaluating the reduction in the time it takes for providers to retrieve critical information within the CSWs," Lin reported. "By making it faster and easier for clinicians to access and use the most up-to-date guidelines at the point of care, we anticipate a more consistent application of best practices, which, as demonstrated by the effectiveness of CSWs in general, can indirectly contribute to improved patient trajectories."
ADVICE FOR OTHERS
Seattle Children's has learned many lessons throughout its AI journey beyond the development of this tool.
"My advice to any healthcare provider organization considering leveraging AI agent technology is to first and foremost anchor their exploration in a clearly defined and pressing problem," Lin said. "Resist the temptation to adopt a cool technology for its own sake. Instead, meticulously identify a specific bottleneck or inefficiency that directly impacts patient care or operational workflows. This problem-first approach will help ensure the AI implementation translates into tangible clinical or operational value.
"Furthermore, be realistic and thorough in estimating the potential return on investment, both clinical and operational," she continued. "A well-defined use case with a clear path to measurable improvement will be crucial for prioritizing efforts and resources in this rapidly growing industry."
Second, recognize the application of AI, particularly generative AI, in clinical contexts demands an exceptionally rigorous approach to validation and testing, she advised.
"Inaccuracies in AI-generated information within clinical settings can have significant consequences," she said. "Therefore, organizations must commit to extensive and iterative testing to achieve a demonstrably high level of accuracy and reliability, and this often means investment in human expertise and hours, often unexpected in initial resource planning.
"Moreover, establishing a robust ongoing monitoring system is essential," she continued. "This should involve actively collecting feedback on the AI's performance in real-world clinical scenarios to identify and address any potential drift or areas for improvement. This commitment to continuous monitoring and validation is critical for responsible AI deployment in healthcare."
Finally, fostering trust and ensuring user adoption are critical success factors for this type of technology, Lin offered.
"This goes beyond the explainability of the technology; it involves comprehensive training and transparent communication about the tool's capabilities and limitations," she said. "Healthcare professionals will want to understand the AI's intended purpose, its data sources and the safeguards in place to ensure accuracy.
"Furthermore, consider the user experience carefully," she concluded. "An intuitive and seamlessly integrated tool is far more likely to be embraced by busy clinicians. Investing in user-centered design and robust training programs will be essential to maximizing the benefits of this technology and ensuring its successful integration into clinical workflows."
Follow Bill's HIT coverage on LinkedIn: Bill Siwicki
Email him: bsiwicki@himss.org
Healthcare IT News is a HIMSS Media publication.
WATCH NOW: How an IT exec can become a Chief AI Officer, and work well with the C-suite
