FolioStart free

Computer Science · Literature

Research papers on Explainable AI in healthcare

Recent and highly-cited academic work on explainable ai in healthcare, gathered from Semantic Scholar, CrossRef and OpenAlex.

Search all 200M+ papers on this topic, free →

  2. A survey of methods for explaining black box models

    Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, et al. · 2019 · ISTI Open Portal · 4,809 citations

    In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provi

  3. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models

    Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, et al. · 2023 · PLOS Digital Health · 3,638 citations

    We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.

  4. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods

    Professor Gary S. Collins, Karel G.M. Moons, Paula Dhiman, et al. · 2024 · BMJ · 2,447 citations

    The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) statement was published in 2015 to provide the minimum reporting recommendations for studies developing or evaluating the performance of a prediction model. Methodological advances in the field of prediction have since included the widespread use of artificial intelligence (AI) powered by machine learning methods to develop prediction models. An update to the TRIPOD statement is thus needed. TRIPOD+AI provides harmonised guidance for reporting prediction model studies, irrespective of whether regression modelling or machine learning methods have been used. The new checklist supersedes

  5. Explainability for artificial intelligence in healthcare: a multidisciplinary perspective

    Julia Amann, Alessandro Blasimme, Effy Vayena, et al. · 2020 · BMC Medical Informatics and Decision Making · 1,914 citations

    BACKGROUND: Explainability is one of the most heavily debated topics when it comes to the application of artificial intelligence (AI) in healthcare. Even though AI-driven systems have been shown to outperform humans in certain analytical tasks, the lack of explainability continues to spark criticism. Yet, explainability is not a purely technological issue, instead it invokes a host of medical, legal, ethical, and societal questions that require thorough exploration. This paper provides a comprehensive assessment of the role of explainability in medical AI and makes an ethical evaluation of what explainability means for the adoption of AI-driven tools into clinical practice. METHODS: Taking A

  6. The false hope of current approaches to explainable artificial intelligence in health care

    Marzyeh Ghassemi, Luke Oakden‐Rayner, Andrew L. Beam · 2021 · The Lancet Digital Health · 1,405 citations

    The black-box nature of current artificial intelligence (AI) has caused some to question whether AI must be explainable to be used in high-stakes scenarios such as medicine. It has been argued that explainable AI will engender trust with the health-care workforce, provide transparency into the AI decision making process, and potentially mitigate various kinds of bias. In this Viewpoint, we argue that this argument represents a false hope for explainable AI and that current explainability methods are unlikely to achieve these goals for patient-level decision support. We provide an overview of current explainability techniques and highlight how various failure cases can cause problems for deci

  7. Designing Theory-Driven User-Centric Explainable AI

    Danding Wang, Qian Yang, Ashraf Abdul, et al. · 2019 · 859 citations

    From healthcare to criminal justice, artificial intelligence (AI) is increasingly supporting high-consequence human decisions. This has spurred the field of explainable AI (XAI). This paper seeks to strengthen empirical application-specific investigations of XAI by exploring theoretical underpinnings of human decision making, drawing from the fields of philosophy and psychology. In this paper, we propose a conceptual framework for building human-centered, decision-theory-driven XAI based on an extensive review across these fields. Drawing on this framework, we identify pathways along which human cognitive patterns drives needs for building XAI and how XAI can mitigate common cognitive biases

  8. Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review

    Anna Markella Antoniadi, Yuhan Du, Yasmine Guendouz, et al. · 2021 · Applied Sciences · 588 citations

    Machine Learning and Artificial Intelligence (AI) more broadly have great immediate and future potential for transforming almost all aspects of medicine. However, in many applications, even outside medicine, a lack of transparency in AI applications has become increasingly problematic. This is particularly pronounced where users need to interpret the output of AI systems. Explainable AI (XAI) provides a rationale that allows users to understand why a system has produced a given output. The output can then be interpreted within a given context. One area that is in great need of XAI is that of Clinical Decision Support Systems (CDSSs). These systems support medical practitioners in their clini

  9. Explainable AI for Healthcare 5.0: Opportunities and Challenges

    Deepti Saraswat, Pronaya Bhattacharya, Ashwin Verma, et al. · 2022 · IEEE Access · 486 citations

    In the healthcare domain, a transformative shift is envisioned towards Healthcare 5.0. It expands the operational boundaries of Healthcare 4.0 and leverages patient-centric digital wellness. Healthcare 5.0 focuses on real-time patient monitoring, ambient control and wellness, and privacy compliance through assisted technologies like artificial intelligence (AI), Internet-of-Things (IoT), big data, and assisted networking channels. However, healthcare operational procedures, verifiability of prediction models, resilience, and lack of ethical and regulatory frameworks are potential hindrances to the realization of Healthcare 5.0. Recently, explainable AI (EXAI) has been a disruptive trend in A

  10. Explainable AI in Clinical Decision Support Systems: A Meta-Analysis of Methods, Applications, and Usability Challenges

    Qaiser Abbas, Woonyoung Jeong, S. Lee · 2025 · Healthcare · 170 citations

    Background: Theintegration of artificial intelligence (AI) into clinical decision support systems (CDSSs) has significantly enhanced diagnostic precision, risk stratification, and treatment planning. AI models remain a barrier to clinical adoption, emphasizing the critical role of explainable AI (XAI). Methods: This systematic meta-analysis synthesizes findings from 62 peer-reviewed studies published between 2018 and 2025, examining the use of XAI methods within CDSSs across various clinical domains, including radiology, oncology, neurology, and critical care. Model-agnostic techniques such as visualization models like Gradient-weighted Class Activation Mapping (Grad-CAM) and attention mecha

  11. Co-design of Human-centered, Explainable AI for Clinical Decision Support

    Cecilia Panigutti, Andrea Beretta, D. Fadda, et al. · 2023 · ACM Transactions on Interactive Intelligent Systems · 102 citations

    eXplainable AI (XAI) involves two intertwined but separate challenges: the development of techniques to extract explanations from black-box AI models and the way such explanations are presented to users, i.e., the explanation user interface. Despite its importance, the second aspect has received limited attention so far in the literature. Effective AI explanation interfaces are fundamental for allowing human decision-makers to take advantage and oversee high-risk AI systems effectively. Following an iterative design approach, we present the first cycle of prototyping-testing-redesigning of an explainable AI technique and its explanation user interface for clinical Decision Support Systems (D

  12. Explainability in medicine in an era of AI-based clinical decision support systems

    Robin Pierce, Wim Van Biesen, Daan Van Cauwenberge, et al. · 2022 · Frontiers in Genetics · 84 citations

    The combination of "Big Data" and Artificial Intelligence (AI) is frequently promoted as having the potential to deliver valuable health benefits when applied to medical decision-making. However, the responsible adoption of AI-based clinical decision support systems faces several challenges at both the individual and societal level. One of the features that has given rise to particular concern is the issue of explainability, since, if the way an algorithm arrived at a particular output is not known (or knowable) to a physician, this may lead to multiple challenges, including an inability to evaluate the merits of the output. This "opacity" problem has led to questions about whether physician

  14. Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis

    Mozhgan Salimparsa, K. Sedig, Dan Lizotte, et al. · 2025 · Informatics · 31 citations

    While Artificial Intelligence (AI) promises significant enhancements for Clinical Decision Support Systems (CDSSs), the opacity of many AI models remains a major barrier to clinical adoption, primarily due to interpretability and trust challenges. Explainable AI (XAI) seeks to bridge this gap by making model reasoning understandable to clinicians, but technical XAI solutions have too often failed to address real-world clinician needs, workflow integration, and usability concerns. This study synthesizes persistent challenges in applying XAI to CDSS—including mismatched explanation methods, suboptimal interface designs, and insufficient evaluation practices—and proposes a structured, user-cent

  15. A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systems

    Alessandro Gambetti, Qiwei Han, Hong Shen, et al. · 2025 · ArXiv · 17 citations

    Explainable Artificial Intelligence (XAI) is essential for the transparency and clinical adoption of Clinical Decision Support Systems (CDSS). However, the real-world effectiveness of existing XAI methods remains limited and is inconsistently evaluated. This study conducts a systematic PRISMA-guided survey of 31 human-centered evaluations (HCE) of XAI applied to CDSS, classifying them by XAI methodology, evaluation design, and adoption barrier. Our findings reveal that most existing studies employ post-hoc, model-agnostic approaches such as SHAP and Grad-CAM, typically assessed through small-scale clinician studies. The results show that over 80% of the studies adopt post-hoc, model-agnostic

  16. Explainable AI in Healthcare: Systematic Review of Clinical Decision Support Systems

    N. A. Aziz, Awais Manzoor, Muhammad Deedahwar, et al. · 2024 · 17 citations

    This systematic review examines the evolution and current landscape of eXplainable Artificial Intelligence (XAI) in Clinical Decision Support Systems (CDSS), highlighting significant advancements and identifying persistent challenges. Utilising the PRISMA protocol, we searched major indexed databases such as Scopus, Web of Science, PubMed, and the Cochrane Library, to analyse publications from January 2000 to April 2024. This timeframe captures the progressive integration of XAI in CDSS, offering a historical and technological overview. The review covers the datasets, application areas, machine learning models, explainable AI methods, and evaluation strategies for multiple XAI methods. Analy

  17. Multi-task reinforcement learning and explainable AI-Driven platform for personalized planning and clinical decision support in orthodontic-orthognathic treatment

    Zhiyuan Li, Liwei Wang · 2025 · Scientific Reports · 8 citations

    This study presents a novel clinical decision support platform for orthodontic-orthognathic treatment that integrates multi-task reinforcement learning with explainable artificial intelligence. The platform addresses the challenges of personalized treatment planning in complex dentofacial deformities by formulating treatment as a sequential decision-making process optimizing multiple clinical objectives simultaneously. We developed a comprehensive framework comprising: (1) a multi-task reinforcement learning core with specialized state-action representations for craniofacial structures; (2) complementary explainable AI components that render complex model decisions interpretable within clini

  18. Explainable AI for Trustworthy Clinical Decision Support: A Case-Based Reasoning System for Nursing Assistants

    Yen-Ku Liu, Yun-Cheng Tsai · 2024 · 2024 IEEE International Conference on Big Data (BigData) · 8 citations

    The increasing complexity of patient data and the need for reliable, transparent decision-making in healthcare drive the demand for advanced AI systems. This paper introduces a Clinical Decision Support System (CDSS) integrating Explainable AI (XAI) with Retrieval-Augmented Generation (RAG), designed to offer transparent, real-time, and evidence-based recommendations. HIPAA-compliant and grounded in historical case studies, the system facilitates trustworthy decision-making in high-pressure environments for nursing assistants. Although preliminary, simulations indicate the system’s potential to reduce cognitive load and enhance decision accuracy, with future clinical trials planned for compr

  19. Current state and promise of user-centered design to harness explainable AI in clinical decision-support systems for patients with CNS tumors

    Eric W. Prince, D. M. Mirsky, T. Hankinson, et al. · 2025 · Frontiers in Radiology · 7 citations

    In neuro-oncology, MR imaging is crucial for obtaining detailed brain images to identify neoplasms, plan treatment, guide surgical intervention, and monitor the tumor's response. Recent AI advances in neuroimaging have promising applications in neuro-oncology, including guiding clinical decisions and improving patient management. However, the lack of clarity on how AI arrives at predictions has hindered its clinical translation. Explainable AI (XAI) methods aim to improve trustworthiness and informativeness, but their success depends on considering end-users’ (clinicians') specific context and preferences. User-Centered Design (UCD) prioritizes user needs in an iterative design process, invo

  20. Explainable AI for Diabetes Risk Assessment: Enhancing Clinical Decision Support with Interpretable Machine Learning Models

    C. S. Reddy, Mohan Annamalai · 2025 · 2025 5th International Conference on Soft Computing for Security Applications (ICSCSA) · 7 citations

    This paper presents a comprehensive study on the application of Explainable Artificial Intelligence (XAI) for diabetes risk assessment, focusing on the interpretability of machine learning models in clinical decision support systems. While machine learning has demonstrated high accuracy in predicting diabetes, the lack of transparency in decision-making processes limits its adoption in healthcare. We employ interpretable models and model-agnostic explanation techniques such as SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-Agnostic Explanations) to enhance the understanding of predictive outcomes. Using real-world datasets including the PIMA Indians Diabetes Dataset

  21. Enhancing AI Clinical Decision Support Trust: Design Workshop Insights from General Practitioners

    R. A. D. L. M. K. Ranwala, Andre Q. Andrade · 2025 · Studies in health technology and informatics · 5 citations

    Artificial Intelligence (AI) predictive models are increasingly integrated into Clinical Decision Support Systems (CDSS). However, real-world implementation is lagging due to a lack of trust and acceptance by healthcare providers. To investigate factors related to trust and acceptance of AI-based CDSS, we conducted a workshop with General Practitioners to explore user, model, and organizational factors that could affect trust and recommendation acceptance. The workshop discussions revealed that while explainability is crucial for understanding AI decisions, current explainable AI (XAI) visualizations appeared to provide limited value to clinicians without technical backgrounds. Participants

  22. AXAI-CDSS: An Affective Explainable AI-Driven Clinical Decision Support System for Cannabis Use

    Tongze Zhang, Tammy Chung, Anind K. Dey, et al. · 2025 · 2025 International Conference on Activity and Behavior Computing (ABC) · 3 citations

    As cannabis use has increased in recent years, researchers have come to rely on sophisticated machine learning models to predict cannabis use behavior and its impact on health. However, many artificial intelligence (AI) models lack transparency and interpretability due to their opaque nature, limiting their trust and adoption in real-world medical applications, such as clinical decision support systems (CDSS). To address this issue, this paper enhances algorithm explainability underlying CDSS by integrating multiple Explainable Artificial Intelligence (XAI) methods and applying causal inference techniques to clarify the models’ predictive decisions under various scenarios. By providing deepe

  23. User-Centered Methods in Explainable AI Development for Hospital Clinical Decision Support: A Scoping Review.

    Bethany A. Van Dort, Thomas Engelsma, S. Medlock, et al. · 2025 · Studies in health technology and informatics · 3 citations

    Explainable Artificial Intelligence (XAI) offers promising advancements in enhancing transparency and usability of AI-based Clinical Decision Support Systems (CDSS) in healthcare settings. These tools aim to improve clinical outcomes by assisting with diagnosis, treatment planning, and risk prediction. However, integrating XAI into clinical workflows requires effective involvement of healthcare professionals to ensure that the explanations provided by these tools are comprehensible, relevant, and actionable. This scoping review aimed to investigate how (potential) end users were involved in the design and development of XAI-based CDSS for hospitals. A systematic search of Medline, Embase, an

Write your paper with these sources

Folio is the integrity-first research workspace: search 200M+ papers, save sources, and write with citations that format themselves. Free for students and researchers.

Start writing free →