Anticipating your organization’s artificial intelligence risks

It’s good to be right. But being right means you sometimes have to take on additional responsibilities that require learning new competencies and skill sets or working closer with other departments. 

We’re talking about artificial intelligence and how AI’s exponential growth in healthcare will affect the responsibilities of internal auditors who work with their risk management, compliance, IT, and quality assurance colleagues within a hospital, health system, or medical practice. 

Prescience of Kodiak’s Top Risks report 

You can find the “right” part in our annual Top Risks report. We released the 2025 Top Risks report in December, and you can read and/or download it here. We identified AI in two risk domains: 

• In the Information Systems domain, we cited the specific risk of AI and emerging technologies governance. We said, “AI is rapidly being adopted across clinical, financial, and operational areas. These tools require access to massive datasets, increasing exposure to privacy and ethical risks. Governance frameworks for AI are still evolving.” 
  
  
• In the Clinical Operations domain, we cited the specific risk of clinical operations and AI. We said, “Clinicians may defer too quickly to AI recommendations, even when they conflict with clinical judgment, leading to missed diagnoses and inappropriate treatment. Workflows and algorithms incorporating AI need constant re-evaluation and validation to address accuracy and safety.”  

We were right, but we didn’t know how right we would be.  

ECRI recognizes AI as a top risk 

In January 2026, ECRI, the consumer patient safety organization, released its Top 10 Health Technology Hazards for 2026 report. No. 1 on ECRI’s list? The “misuse of AI chatbots in healthcare.”  

ECRI said AI chatbots are No. 1 because: “Commonly available LLMs (large language models)—tools like ChatGPT, Claude, Copilot, Gemini, and Grok—are not designed or regulated for healthcare purposes. Yet users may turn to these tools for quick answers to questions about medical conditions, treatments, or less clinical concerns, such as how to use a medical device or what supplies to buy. Such applications often seem innocuous but can have critical implications for patient safety.” 

Then, two months later, in March, ECRI released its Top 10 Patient Safety Concerns 2026 report. What topped this list? You guessed it: AI. Specifically, “navigating the AI diagnostic dilemma.”  

ECRI said using AI to diagnose patients topped its ranking of concerns because: “Placing too much trust in an AI model to diagnose patients without factoring in clinician expertise can lead to misdiagnosis—the very problem AI was intended to solve.” 

AI is getting real for internal audit 

So now, internal auditors have two scenarios to take into account in their daily risk assessment work and under the umbrella of their AI governance program: patients using AI chatbots offered by a third party or healthcare provider organization to diagnose themselves and providers using AI to diagnose patients. 

The first may seem like a stretch as the risk of patients using third-party or provider chatbots is less about technology and more about the downstream clinical and operational impact. Patients may present with preformed diagnoses and high confidence in AI-generated medical advice, requiring providers to recalibrate patients’ expectations and clinical understanding.  

AI tools like chatbots also can miscalculate the urgency of a medical situation, potentially delaying medically necessary care on one hand or driving unnecessary utilization on the other. Think patients demanding antibiotics when clinically contraindicated. Both can affect access and provider workload.  

Providers’ primary response is not to restrict use but to adapt clinical intake, patient education, and governance practices to account for AI-influenced patient behavior. 

This isn’t a made-up scenario. 

In March, KFF released the results of a survey of about 1,300 U.S. adults on their use of AI for healthcare. Thirty-two percent said they used an AI chatbot to get health information on their physical or mental health over the past year. The most common reasons for using an AI chatbot were looking up symptoms or getting general information about their conditions. Fifty-eight percent of those who looked up information on their physical health followed up with a doctor or other healthcare professional as did 42% of those who looked up information on their mental health. 

The second scenario internal audit should consider is much more straight forward, and it’s happening now. 

Also in March, the American Medical Association released results of its latest annual

The need for better AI risk assessment 

Getting ahead of these two risks—consumer-facing AI chatbots and provider-facing AI diagnostic tools—requires cross-functional governance. Cross functional is essential because both scenarios cut a wide path across a variety of departments: internal audit, IT, protected health information and data security, clinical, risk management, compliance, quality assurance, cybersecurity, and more. One team or department doesn’t have the full set of competencies or skill sets to oversee such complex and potential risks. 

That cross‑functional approach must also include continuous collection of data on AI transactions and interventions occurring within organizationally deployed or supported clinical workflows, continuous monitoring of that data, investigation and verification of the associated clinical outcomes, and the ability to flag anomalies and act to mitigate risks created by unexplained variations. This expectation applies to AI tools within the organization’s control or visibility, recognizing that external, patient‑selected tools may fall outside of this scope. 

We will be talking more about AI and other emerging risks at Kodiak’s annual Healthcare Summit to be held in person and virtually Sept. 22-24, 2026, in Fort Worth, Texas. Please come to our breakout session on Risk Intelligence as a strategic asset. You won’t want to miss it.