Journal Of The American Medical Informatics Association[JOURNAL]
Sun
200 papers
Is one run enough? Reproducibility of flagship large language models across temperature and reasoning settings in biomedical text processing.
Windisch P, Koechli C, Dennstädt F … +4 more , Aebersold DM, Zwahlen DR, Förster R, Schröder C
J Am Med Inform Assoc · 2026 Jun · PMID 41911386 · Full text
BACKGROUND: To quantify run-to-run reproducibility of Gemini 3 Flash Preview and GPT-5.2 for trial-success classification across temperature and reasoning/thinking settings and determine whether single-run reporting suff...
BACKGROUND: To quantify run-to-run reproducibility of Gemini 3 Flash Preview and GPT-5.2 for trial-success classification across temperature and reasoning/thinking settings and determine whether single-run reporting suffices. MATERIALS AND METHODS: We utilized 250 trial abstracts labeled based on primary endpoint success. We evaluated Gemini across thinking levels (minimal, low, medium, high) and temperatures 0.0-2.0 and GPT-5.2 across reasoning-effort levels (none to x-high) with an additional temperature sweep when reasoning was disabled. Each setting was run 3 times. RESULTS: Reproducibility was high for Gemini (κ = 0.942-1.000; invalid outputs 0%-1.5%) and GPT-5.2 (κ = 0.984-0.995; no invalid outputs). F1 remained stable (mean/majority vote 0.955-0.971), with marginal gains from majority voting. CONCLUSION: For binary biomedical classification with tightly constrained outputs, both models were reproducible across decoding and reasoning settings, suggesting single runs are often sufficient, with minimal replication as a practical stability check.
Med.ai ASK: an agentic system for biomedical question answering.
Nguyen NTH, Lituiev DS, Liu Z … +10 more , Kashyap A, Jenkinson G, Kuhl K, Corrado C, Patel NM, Snigdha K, Saeedi S, Smith D, Baro N, Schultz T
J Am Med Inform Assoc · 2026 Jun · PMID 41911379 · Full text
OBJECTIVE: Intelligent agent-driven research co-pilots, leveraging advances in generative AI, are transforming how scientists access biomedical knowledge. This paper presents Med.ai ASK, an agentic question-answering sys...
OBJECTIVE: Intelligent agent-driven research co-pilots, leveraging advances in generative AI, are transforming how scientists access biomedical knowledge. This paper presents Med.ai ASK, an agentic question-answering system designed to address biomedical inquiries through dynamic retrieval augmentation and tool-driven reasoning. We aim to develop a system capable of parsing the nuance in biomedical scientists' research questions to provide reliable, grounded responses that are more accurate than other generative AI solutions. MATERIALS AND METHODS: We adopt the ReAct framework's tool-calling architecture and leverage atomic reasoning from Self-Discover to build Med.ai ASK. It selectively queries multiple biomedical knowledge bases and employs map-reduce tools for vector database retrieval, alongside external API and NER tool integration. We ingested 44 million biomedical documents from diverse sources. The agent is evaluated on a range of biomedical question-answering datasets. RESULTS: Human evaluation on an internal dataset shows strong performance and stability. Ratings from a large language model are aligned with human assessments, supporting its use in further experiments. Automatic evaluations indicate superior performance in long-form answers regarding accuracy, faithfulness, factuality, and reduced hallucinations. For short-form and multiple-choice answers, performance is competitive with state-of-the-art systems. The agent's detailed answers are more interpretable than other systems attributed to its agentic design. The agent effectively selects tools based on question type and is deployed in a production-level chat platform with over 1600 users and 25 000 answered questions. CONCLUSION: Med.ai ASK dynamically orchestrates biomedical information retrieval tools to deliver robust interpretative, accurate, and factual answers, which is crucial in the biomedical domain.
The Potential Implications of Informatics for Value-Based Bare.
Dun C, Hicks CW, Lehmann HP
J Am Med Inform Assoc · 2026 Apr · PMID 41854262 · Full text
OBJECTIVE: Value-based care (VBC) represents a fundamental shift from volume-driven reimbursement to models focused on improving patient outcomes and reducing costs. Informatics plays an essential, but often underappreci...
OBJECTIVE: Value-based care (VBC) represents a fundamental shift from volume-driven reimbursement to models focused on improving patient outcomes and reducing costs. Informatics plays an essential, but often underappreciated, role in enabling VBC. Traditional discussions of informatics emphasize data and technology; however, a broader sociotechnical view highlights how people, organizations, workflows, and policies interact with technology to influence the success of VBC initiatives. In this article, we apply the Informatics Stack as a heuristic framework to examine how informatics shapes VBC across 4 phases: research, policy setting, healthcare implementation, and local assessment within learning health systems. MATERIALS AND METHODS: We applied the Informatics Stack as a heuristic framework to analyze VBC across four phases: research, policy setting, healthcare implementation, and local assessment. To provide a grounded analysis, the study focused on the Healthcare Implementation phase, utilizing vascular claudication management as a primary illustrative case to demonstrate how high-level VBC policies are converted into granular clinical workflows and algorithms. RESULTS: We present "As-Is" characterizations of informatics in VBC at multiple levels of the Stack, ranging from world-level regulatory forces to organizational values, to business processes, workflows, information systems, modules, algorithms, data, and underlying technologies. We also outline "To-Be" opportunities, including computable clinical guidelines, interoperable data platforms, algorithm performance monitoring, and integration of multimodal data streams into decision support. To provide a grounded analysis, we narrow our focus to the Healthcare Implementation phase, using vascular claudication management as our primary illustrative case. Managing claudication in a VBC model requires preventing low-value care, such as early, aggressive peripheral vascular interventions, while optimizing patient-specific outcomes. We will used this clinical example to walk down the levels of the Stack, demonstrating how informatics converts high-level VBC policy into granular clinical workflows and algorithm. DISCUSSION: In this article, we apply the Informatics Stack as a heuristic framework to examine how informatics shapes VBC across four phases, specifically focusing on the Healthcare Implementation phase using vascular claudication management as an illustrative case. We present "As-Is" characterizations of the current state of informatics alongside "To-Be" opportunities, including computable clinical guidelines, interoperable data platforms, and algorithm performance monitoring. Concluding that VBC requires a socio-technical perspective beyond mere data and technology, we propose the Stack as a diagnostic tool for health leaders and offer a "VBC Informatics Gap Analysis Toolkit" to help organizations identify alignment gaps in their implementation strategies. CONCLUSION: In teaching informatics and in generating an assessment of where a problem or a field is at any point in time, we have found the Stack to help students, universally, and many alumni, to apply in their work. However, for the health system leader attempting to implement VBC, the Stack must be more than a theoretical model; it must be a diagnostic tool. It does not prescribe specific software purchases, but rather points out to an organization where they may have inconsistencies in their conception of their informatics infrastructure or frank gaps in that conceptualization.
Accuracy of an XGBoost-based privacy preserving record linkage system compared with an electronic health record patient matching module in identifying patients shared between nearby academic health centers.
Bell DS, Saleh T, Sanz Vidorreta FJ … +4 more , Pirani CN, Pevnick JM, Jenders RA, Soohoo SL
J Am Med Inform Assoc · 2026 Apr · PMID 41851042 · Full text
OBJECTIVES: Patients often receive health care from multiple organizations. Privacy Preserving Record Linkage (PPRL) is a technology for linking patient records without releasing personally identifiable information. We c...
OBJECTIVES: Patients often receive health care from multiple organizations. Privacy Preserving Record Linkage (PPRL) is a technology for linking patient records without releasing personally identifiable information. We compared a commercial PPRL tool that uses the XGBoost machine learning algorithm with Care Everywhere (CE), a widely used rule-based patient linkage module. MATERIALS AND METHODS: We matched the complete patient populations from Cedars-Sinai Health System and University of California, Los Angeles (UCLA) Health using the XGBoost PPRL tool at each of 3 score thresholds (98, 95, and 90), reflecting stricter vs more permissive matching. We compared PPRL matches with CE matches for the cohort of 849 157 patients who had been queried by CE from UCLA to Cedars-Sinai over 18 months. To classify proposed matches as false, uncertain or correct matches, 2 reviewers manually reviewed a random sample of 1200 patients representing each category of matches. RESULTS: Care Everywhere matched 18% of the cohort, whereas PPRL matched 9%, 27%, and 29% of the cohort using the 98, 95, and 90 thresholds, respectively. Projecting the false match rates from the manual review to the original populations, precision for CE was 99.6% (95% CI, 97.8%-100%). Precision for PPRL was 100% (95% CI, 99.2%-100%), 99.4% (95% CI, 97.4%-99.9%), and 98.7% (95% CI, 96.5%-99.4%) at the 3 thresholds, respectively. Using CE and PPRL matches together as a proxy gold standard, recall for CE was 61.5% (95% CI, 60.3%-61.9%) and for PPRL was 30.6% (95% CI, 30.3%-30.7%), 92.2% (95% CI, 90.2%-92.7%), and 96.8% (95% CI, 94.6%-97.5%) at each threshold, respectively. CONCLUSIONS: The precision and recall of PPRL matching differed substantially across the available match thresholds. Compared with the rule-based system, PPRL at the 95 threshold had 50% higher recall with similar precision. Privacy Preserving Record Linkage holds promise for improving research, but users must choose the precision vs recall needed for their application.
Life events extraction from healthcare notes for veteran acute suicide risk prediction.
Morrow D, Zamora-Resendiz R, Dhaubhadel S … +5 more , Beckham JC, Kimbrel NA, McMahon BH, VA Million Veteran Program , Crivelli S
J Am Med Inform Assoc · 2026 Apr · PMID 41842607 · Full text
OBJECTIVE: Predictive models of suicide risk have focused on features extracted from structured data found in electronic health records, with limited consideration of predisposing life events (LE) expressed in unstructur...
OBJECTIVE: Predictive models of suicide risk have focused on features extracted from structured data found in electronic health records, with limited consideration of predisposing life events (LE) expressed in unstructured clinical text such as housing instability and marital troubles. This study aims to expand upon previous research, demonstrating how high-performance computing (HPC) and machine learning methodologies can be used to extract and annotate 8 LE across all Veterans Health Administration (VHA) unstructured clinical text data with enriched performance metrics. Integration of the 8 LE with the structured features using different statistical and machine learning (ML) methods is also discussed. MATERIALS/METHODS: VHA-wide clinical text from January 2000 to January 2022 was pre-processed and analyzed using HPC. Data-driven lexicon curation enabled a rule-based annotator to extract LE, followed by machine learning for improved positive predictive value (PPV). NLP results were analyzed longitudinally and then integrated and compared to a baseline statistical model predicting risk for a combined outcome (suicide death, suicide attempt and overdose). RESULTS: First-time LE mentions showed a significant temporal correlation to suicide-related events (SRE) (suicide ideation, attempt and/or death) and are not associated with administrative bias. Predictive linear regression (LR) models integrating NLP-derived LE show an improved AUC of 0.81 and novel patient identification of up to 18%. DISCUSSION: Our analysis shows that these methodologies helped improve performance metrics significantly from previous work, while outperforming related works. These results demonstrated that NLP-derived LE served as acute predictors for SRE. CONCLUSION: NLP integration into predictive models may help improve clinician decision support. Future work is necessary to better define and integrate these and other potential LE.
Implementation of a clinical decision support tool for hypertension in a rural health system.
Kharbanda EO, Asche SE, Essien I … +9 more , Allen CI, Freitag LA, Ekstrom HL, Kromrey KA, Muthineni A, Saman DM, Thirumalai V, O'Connor PJ, Benziger CP
J Am Med Inform Assoc · 2026 May · PMID 41841357 · Full text
OBJECTIVES: Elevated blood pressure (BP) and hypertension are often overlooked in pediatric care. We adapted a pediatric hypertension clinical decision support (CDS) for a primarily rural health system and compared CDS i...
OBJECTIVES: Elevated blood pressure (BP) and hypertension are often overlooked in pediatric care. We adapted a pediatric hypertension clinical decision support (CDS) for a primarily rural health system and compared CDS impact across varied implementation approaches. METHODS: In this cluster randomized trial, 40 primary care clinics were randomized 1:1:1 to CDS with high-intensity implementation, CDS with low-intensity implementation, or usual care (UC). Low-intensity implementation was limited to online training. High-intensity CDS implementation included in-person and online training, monthly check-ins and feedback regarding CDS use. Patients 6-17 years with BP measured at a primary care visit from August 1, 2022 to January 31, 2024 were eligible. Outcomes were remeasurement of elevated BP during a visit and recognition of hypertension within 6 months of meeting criteria. Analyses adjusted for clustered study design and patient characteristics. RESULTS: Of 9155 patients with an elevated BP, remeasurement during the visit occurred for 51.5% at high-intensity, 23.6% at low-intensity, and 6.2% at UC clinics. Among 578 patients with incident hypertension, recognition was 42.8% at high-intensity, 24.5% at low-intensity and 14.4% at UC clinics. Patients attending high or low-intensity CDS clinics were more likely than those at UC to have elevated BP remeasured (adjusted odds ratio [aOR] 8.70; 95% CI 5.68-13.3) and to have their hypertension clinically recognized (aOR 2.94; 1.00-8.60). High-intensity implementation was more effective than low-intensity implementation for repeat BP measurement (aOR 3.45; 1.88-6.33) and hypertension recognition (aOR 2.31; 1.08-4.98). CONCLUSIONS: CDS improved pediatric BP care in a primarily rural health system while effectiveness varied by implementation approach.
Systematic review of foundation models for structured electronic health records.
Guo LL, Arciniegas SE, Yan AP … +3 more , Fries J, Tomlinson GA, Sung L
J Am Med Inform Assoc · 2026 Jun · PMID 41830958 · Full text
PURPOSE: Foundation models pretrained on structured electronic health record (EHR) data promise improved predictive performance, sample efficiency and resilience to distribution shifts. However, model design, scale and u...
PURPOSE: Foundation models pretrained on structured electronic health record (EHR) data promise improved predictive performance, sample efficiency and resilience to distribution shifts. However, model design, scale and use remain unclear. Objectives were to characterize foundation models pretrained on structured EHR data; examine temporal trends in model application and scale, architecture and design; and assess the extent to which publications omitted methodological details. METHODS: We searched MEDLINE and Embase (2018-October 2025) for foundation models pretrained on structured EHR data using self-supervised learning and applied to clinical prediction tasks. Study selection and data abstraction were performed in duplicate. Characteristics were summarized and stratified by median publication year. RESULTS: Fifty-three studies were included; publications increased over time. Most datasets (79%) originated from the United States. None pretrained exclusively on pediatric cohorts. Model architecture shifted towards transformers (P = .013) with longer context windows (P = .028), while application shifted from exclusively embedding-based toward generative or mixed use (P < .001). Choices regarding feature inclusion, temporal representation, self-supervised objective and downstream adaptation remained heterogeneous. Only 26% of studies evaluated transfer to external datasets, and none described clinical deployment. Key indicators of scale and compute were frequently unreported. CONCLUSIONS: EHR foundation models are proliferating and increasingly transformer-based and generative. Yet methodological choices and reporting remain fragmented, indicating design trade-offs and best practices for EHR foundation models have not yet been established. None describe clinical deployment. Future work should clarify which design choices improve performance, robustness and transferability, increase reporting transparency and identify if they can be implemented to improve patient-important outcomes.
Exploring approaches to computational representation and classification of user-generated meal logs.
Hu G, Anand A, Desai PM … +2 more , Urteaga I, Mamykina L
J Am Med Inform Assoc · 2026 Apr · PMID 41830956 · Full text
OBJECTIVE: This study examined the use of machine learning (ML) and domain-specific enrichment in patient-generated health data, in the form of free-text meal logs, to classify meals on alignment with different nutrition...
OBJECTIVE: This study examined the use of machine learning (ML) and domain-specific enrichment in patient-generated health data, in the form of free-text meal logs, to classify meals on alignment with different nutritional goals. MATERIALS AND METHODS: We used a dataset of over 3000 meal records collected by 114 individuals from a diverse, low-income community in a major US city using a mobile app. Registered dietitians (RDs) provided expert judgment for meal-goal alignment, used as the "gold-standard" for evaluation. Using text embeddings (TF-IDF and BERT) and domain-specific enrichment information (ontologies, ingredient parsers, and macronutrient contents) as inputs, we evaluated the performance of logistic regression and multilayer perceptron classifiers using accuracy, precision, recall, and F1 score against the gold standard and the individual's self-assessment. RESULTS: On average, individuals who logged meals achieved 0.576 accuracy of meal-goal alignment self-assessments. Even without enrichment, ML outperformed individual's self-assessments, with accuracies within 0.726-0.841 for different goals. The best-performing combination of ML classifier with enrichment achieved even higher accuracies (0.814-0.902). In general, ML classifiers with enrichment of parsed ingredients, food entities, and macronutrients information performed well across multiple nutritional goals, but there was variability in the impact of enrichment and classification algorithm on accuracy of classification for different nutritional goals. CONCLUSION: ML can utilize unstructured free-text meal logs and reliably classify whether meals align with specific nutritional goals, exceeding individuals' self-assessments, especially when incorporating nutrition domain knowledge. Our findings highlight the potential of ML analysis of patient-generated health data to support patient-centered nutrition guidance in precision healthcare.
Improved evaluation frameworks are required to move application of LLMs from research into clinical practice.
Bakken S
J Am Med Inform Assoc · 2026 Mar · PMID 41818773 · Full text
Abstract loading — click title to view on PubMed.
Current methods for analyzing time-series patient-generated health data to assess treatment response: a scoping review.
Banda M, Bladon S, Al-Attar M … +4 more , Cahuantzi R, Jenkins DA, Dixon WG, van der Veer SN
J Am Med Inform Assoc · 2026 May · PMID 41812143 · Full text
OBJECTIVES: We aimed to identify and map recent studies using high-frequency, time-series electronic patient-generated health data (ePGHD) to assess treatment response; characterize ePGHD types and collection methods; su...
OBJECTIVES: We aimed to identify and map recent studies using high-frequency, time-series electronic patient-generated health data (ePGHD) to assess treatment response; characterize ePGHD types and collection methods; summarize ePGHD-based definitions of treatment response; and describe analytical approaches used. MATERIALS AND METHODS: We systematically searched 4 databases for articles published between January 2022 and June 2024, supplemented by a forward citation search until June 2025. Peer-reviewed studies were eligible if ePGHD were collected outside clinical settings, and either reported at least weekly (ie, if actively reported by participants) or summarized discretely (eg, daily) if passively collected via wearables/sensors. We screened articles for eligibility independently in duplicate and synthesized extracted data descriptively. RESULTS: Our search yielded 4030 articles, of which we included 186. Most studies collected ePGHD using mobile applications or webforms (n = 133) over 4-12 weeks (n = 67). Prior to analysis, 132 studies excluded portions or condensed ePGHD into one or more summaries. Among 172 studies estimating treatment response, 98 applied longitudinal methods (eg, mixed-effects models) that accounted for repeated measures while capturing within- and between-subject variations, whereas 74 used cross-sectional approaches. Of 18 prediction modeling studies, 16 employed machine learning techniques, with only 4 explicitly modeling repeated measures. Five studies identified clusters of response trajectories generally without incorporating temporal dependencies (eg, using K-means). DISCUSSION AND CONCLUSION: Many studies in this review did not fully leverage the high-frequency, longitudinal nature of ePGHD. Future research should adopt more appropriate and readily available analytic methods to maximize the potential of time-series ePGHD for generating insights into treatment response.
MedRep: medical concept representations for general electronic health record foundation models.
Kim J, Lee N, Kim J … +1 more , Kim K
J Am Med Inform Assoc · 2026 Jun · PMID 41806382 · Full text
OBJECTIVE: Traditional electronic health record (EHR) foundation models fail to process unseen medical codes, limiting generalizability across institutions with different vocabularies. To address this problem, we introdu...
OBJECTIVE: Traditional electronic health record (EHR) foundation models fail to process unseen medical codes, limiting generalizability across institutions with different vocabularies. To address this problem, we introduce medical concept representation (MedRep), standardized medical concept representations for EHR foundation models, enabling recognition of semantically similar concepts regardless of their specific IDs. MATERIALS AND METHODS: We utilized Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) vocabulary covering 7.5 million concepts from 66 medical vocabularies. MedRep integrates large language model-generated concept descriptions and OMOP graph ontology using graph contrastive learning with knowledge distillation. We evaluated MedRep-based models on MIMIC-IV (internal validation) and EHRSHOT (external validation) across 9 prediction tasks including clinical outcomes, phenotypes, and in-hospital events. RESULTS: MedRep consistently outperformed baseline models, particularly in external validation with average improvements of 0.088 in area under the receiver operating characteristic curve and 0.208 in area under the precision-recall curve. Qualitative analysis demonstrated that MedRep-based models identified more clinically relevant concepts when making decisions than the baseline models. Performance improvements remained stable across diverse EHR foundation model architectures, including BEHRT, Med-BERT, and CDM-BERT. DISCUSSION: MedRep improves the generalizability of EHR foundation models by encouraging similar concepts to have similar representations. EHR foundation models developed at different institutions could cooperate through MedRep, merging knowledge from multiple hospital datasets. In addition, our approach could reduce healthcare disparities by enabling smaller institutions to benefit from models trained on larger datasets. CONCLUSION: MedRep improves EHR foundation model performance, interpretability, and generalizability, serving as a standard baseline representation for EHR foundation models adopting OMOP CDM.
Interactive active learning for literature screening: finetuning GPT with DeepSeek reasoning for cross-domain generalization.
Li Y, Plasek JM, Du X … +7 more , Wang Y, Zhou Z, Lian J, Chuang YW, Hong P, Hou PC, Zhou L
J Am Med Inform Assoc · 2026 May · PMID 41801982 · Full text
OBJECTIVE: Automated literature screening in biomedical research is often hindered by domain shifts and scarcity of labeled data, which limit model accuracy and generalizability. While large language models (LLMs) perfor...
OBJECTIVE: Automated literature screening in biomedical research is often hindered by domain shifts and scarcity of labeled data, which limit model accuracy and generalizability. While large language models (LLMs) perform well in zero-shot settings, they often fail to capture complex, domain-specific reasoning patterns. To address this limitation, this study investigates whether an interactive, weakly supervised learning framework combining GPT (generative pre-trained transformer)'s fine-tuning adaptability with DeepSeek's reasoning capabilities can improve literature screening performance across biomedical domains. MATERIALS AND METHODS: We developed an active learning framework that leverages model disagreement between GPT-4o and DeepSeek to improve literature screening performance. This process began with a labeled corpus of 6331 articles on large language models, from which a model disagreement analysis was performed to identify cases where GPT-4o misclassified and DeepSeek produced correct predictions. Three GPT variants-GPT-4o, GPT-4o-mini, and GPT-4.1-nano, were fine-tuned under standard supervised learning settings using these disagreement-based samples. Fine-tuning prompts incorporated classification labels and, when available, rationale traces generated by DeepSeek to provide reasoning-augmented weak supervision. Model performance was evaluated on an independent benchmark set of 291 annotated articles across 10 topic queries in cancer immunotherapy and LLMs in medicine, using standard evaluation metrics, with recall as the primary measure. RESULTS: Fine-tuning GPT models using disagreement-based examples significantly improved performance. GPT-4o-mini achieved the best overall results after fine-tuning, especially with the highest F1 score (0.93, P < .001) and recall (0.95, P < .001). Across the biomedical topics, fine-tuned models consistently outperformed their zero-shot counterparts without increasing reviewer workload. DISCUSSION: These findings demonstrate the effectiveness of disagreement-driven active learning in enhancing GPT-based biomedical literature screening. Lightweight models like GPT-4o-mini benefit most from targeted, reasoning-enriched training, highlighting their suitability for scalable deployment. CONCLUSION: This study introduces an interactive active learning framework that leverages fine-tuned LLMs with reasoning capabilities to enhance literature screening. The approach offers a scalable solution to more efficient and reliable information retrieval in systematic reviews.
Fairness aware subset selection for advancing equity in skin cancer detection.
Perry Y, Almuzaini AA, Adamson AS … +3 more , Dasgeb B, Foran DJ, Singh VK
J Am Med Inform Assoc · 2026 May · PMID 41801970 · Full text
OBJECTIVES: Skin cancer is the most common malignancy in the United States, with more than five million cases diagnosed annually among 3.3 million individuals. Melanoma, the deadliest form of skin cancer, accounts for ro...
OBJECTIVES: Skin cancer is the most common malignancy in the United States, with more than five million cases diagnosed annually among 3.3 million individuals. Melanoma, the deadliest form of skin cancer, accounts for roughly 200 000 new diagnoses each year and nearly 10 000 deaths. AI-based skin cancer detection is being developed and tested in laboratory and academic settings as a promising approach to improve access and reduce disparities. However, current models often underperform on darker skin tones (Fitzpatrick Types V and VI), creating fairness concerns that must be addressed prior to clinical deployment. Existing fairness-aware methods focus on algorithmic adjustments while neglecting data quality and representation. We introduce FAIR-SCAN (Fairness and Accuracy through Ranking-Based Subset Selection for Skin Cancer Detection), a data-centric framework that enhances fairness through subset selection guided by marginal contribution score (MCS) estimation. MATERIALS AND METHODS: FAIR-SCAN ranks data points by their contribution to both accuracy and fairness, then selects an optimal subset for training. We evaluated its effectiveness using images from Diverse Dermatology Images (DDI) and Fitzpatrick 17K. RESULTS: FAIR-SCAN improved balance in accuracy, True Positive Rate, and False Positive Rate across skin tones while reducing the training dataset by 50%, outperforming algorithm-focused fairness methods. DISCUSSION: These findings highlight the importance of strategic data selection in mitigating bias in AI-driven diagnostics. FAIR-SCAN's data-centric approach enhances both precision and equity in skin cancer detection. CONCLUSION: Strategic data selection is critical for equitable AI-driven diagnostics. FAIR-SCAN advances fairness and accuracy in skin cancer detection, supporting development of trustworthy clinical AI systems.
Evaluating the potential of fast healthcare interoperability resources for clinical registry data submission.
Tcheng JE, Finney D, Boone K … +6 more , Desai SP, Pyke DA, Shanbhag N, Srinivasan G, Ramsing N, Kelemen MD
J Am Med Inform Assoc · 2026 May · PMID 41801964 · Full text
OBJECTIVE: We conducted the Clinical Registry Extraction and Data Submission (CREDS) project to evaluate the readiness of HL7 Fast Healthcare Interoperability Resources (FHIR) for provisioning data from health informatio...
OBJECTIVE: We conducted the Clinical Registry Extraction and Data Submission (CREDS) project to evaluate the readiness of HL7 Fast Healthcare Interoperability Resources (FHIR) for provisioning data from health information systems for the American College of Cardiology Cardiac Catheterization Percutaneous Coronary Intervention (CathPCI) Registry. MATERIALS AND METHODS: The CREDS project had 3 workstreams: (1) evaluation of the readiness of clinical documentation for data transforms, (2) modeling of a FHIR-based clinical workflow for registry data submission, and (3) development and demonstration of a CREDS FHIR implementation for registry data submission. RESULTS: Of the 344 data concepts comprising the CathPCI Registry, only 111 (32%) were sufficiently discrete to be listed in the CathPCI Data Dictionary with a terminology mapping. Cardiologist informaticians identified an additional 42 concepts suitable for provisioning via a FHIR payload. The resulting notional workflow combined FHIR-based data assembly with manual chart abstraction of compound, summative, and complex clinical concepts. A CathPCI FHIR StructureDefinition artifact was authored, incorporated into a CREDS FHIR Implementation Guide, and balloted to Standard for Trial Use status. DISCUSSION: CREDS demonstrated both potential and limitations for using FHIR for registry data submission. The largest technical impediment was the volume of code (>11 000 lines) for the FHIR StructureDefinition. Lack of regularized clinical vocabularies, reliance of registries on complex clinical concepts, and absence of FHIR infrastructure must be overcome before CREDS can be used at scale. CONCLUSION: CREDS demonstrated proof-of-concept FHIR-based provisioning of clinical data for registry submission. All artifacts are open source to inform others with similar interests.
CPGPrompt: translating clinical guidelines into large language model-executable decision support.
Deng R, Martin G, Wang T … +6 more , Zhang G, Liu Y, Weng C, Wang Y, Rousseau JF, Peng Y
J Am Med Inform Assoc · 2026 Apr · PMID 41746783 · Full text
OBJECTIVE: Clinical practice guidelines (CPGs) provide evidence-based recommendations for patient care; however, integrating them into artificial intelligence (AI) remains challenging. Previous approaches, such as rule-b...
OBJECTIVE: Clinical practice guidelines (CPGs) provide evidence-based recommendations for patient care; however, integrating them into artificial intelligence (AI) remains challenging. Previous approaches, such as rule-based systems or black-box AI models, face significant limitations, including poor interpretability, inconsistent adherence to guidelines, and narrow domain applicability. To address this, we develop and validate CPGPrompt, an auto-prompting system that converts narrative clinical guidelines into large language models (LLMs). MATERIALS AND METHODS: Our framework translates CPGs into structured decision trees and utilizes an LLM to dynamically navigate them for patient case evaluation. Synthetic vignettes were generated across 3 domains-headache, lower back pain, and prostate cancer-and distributed into 4 categories to test different decision scenarios. System performance was assessed on both binary specialty referral decisions and fine-grained pathway classification tasks. RESULTS: The binary specialty referral classification achieved consistently strong performance across all domains (F1: 0.85-1.00), with high recall (1.00 ± 0.00). In contrast, multiclass pathway assignment showed reduced performance, with domain-specific variations: headache (F1: 0.47), lower back pain (F1: 0.72), and prostate cancer (F1: 0.77). DISCUSSION: Domain-specific performance differences reflected the structure of each guideline. The headache guideline highlighted challenges with negation handling. The lower back pain guideline required temporal reasoning. In contrast, prostate cancer pathways benefited from quantifiable laboratory tests, resulting in more reliable decision-making. CONCLUSION: CPGPrompt demonstrates generalizability across diverse clinical domains while maintaining high sensitivity for referral decisions. Its transparent, auditable framework enables the systematic identification of failure modes and provides advantages over black-box AI approaches. However, persistent challenges with subjective clinical assessments indicate a need for targeted improvements and greater clinical robustness.
Bridging the governance gap in health artificial intelligence: integrating nursing perspectives into the National Academy of Medicine Artificial Intelligence Code of Conduct.
Cary MP, Russell RG, Silcox C … +5 more , Lytle KS, Lehmann LS, Hightower M, Economou-Zavlanos N, Guilamo-Ramos V
J Am Med Inform Assoc · 2026 May · PMID 41739499 · Full text
BACKGROUND AND APPROACH: In 2025, the National Academy of Medicine released an Artificial Intelligence Code of Conduct (AICC). In this commentary, we examine how the AICC introduces governance mechanisms to oversee AI ap...
BACKGROUND AND APPROACH: In 2025, the National Academy of Medicine released an Artificial Intelligence Code of Conduct (AICC). In this commentary, we examine how the AICC introduces governance mechanisms to oversee AI applications and how it can support the ethical development and responsible use of AI in healthcare, paying special attention to the role of nurses. FINDINGS: One shortcoming of the AICC is its lack of explicit acknowledgment of nurses, which risks obscuring their indispensable role in the safe, equitable, and effective use of AI in healthcare. We offer practical steps for health leaders to operationalize the AICC. CONCLUSION: Implementation of the AICC can support robust AI governance in health systems, but nurse expertise must be incorporated. The AICC offers a framework into which nursing perspectives can be embedded to ensure that AI tools positively transform healthcare delivery and enhance the quality and equity of care.
Adapting machine learning models to temporal drift: oral mucositis prediction in head and neck cancer patients receiving proton and carbon ion therapy.
Wang Y, Zhang Z, Zhu Y … +4 more , Wang Z, Ning R, Zhang L, Wan H
J Am Med Inform Assoc · 2026 Apr · PMID 41729191 · Full text
AIM: Temporal drift, defined as changes over time in underlying data distributions, can degrade the performance of clinical prediction models. In head and neck cancer (HNC) radiotherapy, evolving proton and carbon ion th...
AIM: Temporal drift, defined as changes over time in underlying data distributions, can degrade the performance of clinical prediction models. In head and neck cancer (HNC) radiotherapy, evolving proton and carbon ion therapies may shift the risk of oral mucositis over time. This study aimed to compare machine learning (ML) strategies for mitigating temporal drift in predicting grade ≥2 oral mucositis among patients treated with particle therapy. METHODS: This retrospective cohort included 1751 adults with HNC treated with particle therapy between May 2015 and December 2022 at a single proton and heavy-ion center. Acute oral mucositis was graded twice weekly using Radiation Therapy Oncology Group criteria. Thirty-five demographic, clinical, and laboratory variables were extracted from electronic health records. Three complementary strategies were examined, including standard ML with inclusion of recent data, temporal modeling, and transfer learning, and each benchmarked using 14 machine-learning algorithms. Model performance was assessed using Area Under the Receiver Operating Characteristic Curve (AUC), F1-score, accuracy, precision, recall, and SHAP-based interpretability. RESULTS: The incidence of grade ≥2 oral mucositis increased from 27.3% in 2015 to 60.4% in 2022, paralleling evolving dose and modality patterns. Models trained on 2015-2020 data declined in AUC from 0.81 internally to 0.74 and 0.68 on 2021 and 2022 data. A Extras Trees transfer-learning ensemble achieved the best temporal robustness (AUC 0.87, F1 0.82) on 2022 data, demonstrating improved adaptability to drift. CONCLUSIONS: Temporal drift significantly reduced oral mucositis prediction accuracy over time. Transfer-learning ensembles improved adaptability and maintained reliable, clinically relevant performance for particle-therapy toxicity prediction.
Comparing ambient scribes: a randomized crossover clinical trial addressing ambient scribe technologies' impact on physician burnout.
Chowdhury A, Casey M, Wilson J … +4 more , Pollak KI, Goldstein BA, Bedoya A, Poon EG
J Am Med Inform Assoc · 2026 May · PMID 41729180 · Full text
OBJECTIVE: This study aims to compare the effectiveness of 2 ambient AI scribe technologies in reducing physician burnout, improving workflow satisfaction, and enhancing documentation efficiency through a randomized cros...
OBJECTIVE: This study aims to compare the effectiveness of 2 ambient AI scribe technologies in reducing physician burnout, improving workflow satisfaction, and enhancing documentation efficiency through a randomized crossover trial. MATERIALS AND METHODS: An open-label randomized crossover trial involving 160 outpatient clinicians was conducted at a tertiary academic medical center. Volunteers were randomized to 2 groups of 80 with 2 crossover periods. We assessed workflow satisfaction (1-7 scale), burnout (Copenhagen Burnout Index), and efficiency metrics (eg, electronic health record time outside scheduled hours, documentation time, etc.). Data was analyzed using Wilcoxon signed-rank tests and generalized linear mixed models. RESULTS: Surveys from 136 respondents were analyzed. Clinicians reported greater improvements in satisfaction with product B (2.51 points on a 7-point scale) compared to product A (1.91 points; mean difference: 0.60, 95% CI: 0.32-0.90). Both tools reduced personal and work burnout scores, but differences between tools were not meaningful. Product B demonstrated greater reductions in average minutes-in-notes per day compared to product A (B - A = -3.19 minutes; 95% CI -4.87 to -1.50). No meaningful differences were observed in pajama time or patient-related burnout. DISCUSSION: Both tools improved workflow satisfaction and reduced burnout, with product B showing superior performance in satisfaction and documentation time. However, efficiency metrics like pajama time were largely unaffected, potentially due to participant selection bias and the study period's timing. CONCLUSION: Product B yielded greater satisfaction and time savings compared to product A, though both tools effectively reduced physician burnout and improved workflow satisfaction.
Human-Large Language Model collaboration for systematic ontology updates: a case study in the domain of dietary lifestyle.
Jung J, Taira R, Kim H
J Am Med Inform Assoc · 2026 May · PMID 41719167 · Full text
OBJECTIVES: To develop and evaluate a human-LLM (Large Language Model) collaborative approach for systematic ontology updating, demonstrated with the Dietary Lifestyle Ontology (DILON). MATERIALS AND METHODS: One hundred...
OBJECTIVES: To develop and evaluate a human-LLM (Large Language Model) collaborative approach for systematic ontology updating, demonstrated with the Dietary Lifestyle Ontology (DILON). MATERIALS AND METHODS: One hundred dietary questionnaire items from English and Korean sources were semantically annotated by 4 state-of-the-art language models, which generated candidate concepts for inclusion into DILON. Outputs were refined through cross-model reconciliation, followed by expert review. The model curated the concept within DILON and experts reviewed and refined the outputs in Protégé to ensure accuracy and consistency. RESULTS: Claude Sonnet 4 effectively supported local tasks, including harvesting new concepts, detecting redundancies, and refining hierarchical segments. Global optimization of ontology, however, required systematic examination by human experts. DISCUSSION: These findings highlight the complementary strengths of LLMs and humans: LLMs accelerate repetitive and local updates, whereas humans maintain overall structural integrity. CONCLUSION: Human-LLM collaboration improves efficiency, scalability, and sustainability in ontology engineering, supporting the maintenance of complex biomedical ontologies.
Factors influencing the effectiveness of artificial intelligence-assisted decision-making in medicine: a scoping review.
Jackson NJ, Brown KE, Miller R … +6 more , Murrow M, Cauley MR, Collins BX, Novak LL, Benda NC, Ancker JS
J Am Med Inform Assoc · 2026 May · PMID 41719163 · Full text
OBJECTIVES: Research on artificial intelligence (AI)-based clinical decision-support (AI-CDS) systems has returned mixed results. Sometimes providing AI-CDS to a clinician will improve decision-making performance, someti...
OBJECTIVES: Research on artificial intelligence (AI)-based clinical decision-support (AI-CDS) systems has returned mixed results. Sometimes providing AI-CDS to a clinician will improve decision-making performance, sometimes it will not, and it is not always clear why. This scoping review seeks to clarify existing evidence by identifying clinician-level and technology design factors that impact the effectiveness of AI-assisted decision-making in medicine. MATERIALS AND METHODS: We searched MEDLINE, Web of Science, and Embase for peer-reviewed papers that studied factors impacting the effectiveness of AI-CDS. We identified the factors studied and their impact on 3 outcomes: clinicians' attitudes toward AI, their decisions (eg, acceptance rate of AI recommendations), and their performance when utilizing AI-CDS. RESULTS: We retrieved 5850 articles and included 45. Four clinician-level and technology design factors were commonly studied. Expert clinicians may benefit less from AI-CDS than nonexperts, with some mixed results. Explainable AI increased clinicians' trust, but could also increase trust in incorrect AI recommendations, potentially harming human-AI collaborative performance. Clinicians' baseline attitudes toward AI predict their acceptance rates of AI recommendations. Of the 3 outcomes of interest, human-AI collaborative performance was most commonly assessed. DISCUSSION AND CONCLUSION: Few factors have been studied for their impact on the effectiveness of AI-CDS. Due to conflicting outcomes between studies, we recommend future work should leverage the concept of "appropriate trust" to facilitate more robust research on AI-CDS, aiming not to increase overall trust in or acceptance of AI but to ensure that clinicians accept AI recommendations only when trust in AI is warranted.
About
- Frequency
- Sun
- Papers found
- 200
- RSS feed
- Subscribe