Desai D, Nayebi A, Ghyabi M
… +4 more, Markiewicz M, Lattanzi D, Scafide K, Wojtusiak J
AMIA Annu Symp Proc
· 2024 · PMID 41726463
Artificial Intelligence (AI)-based object detection models, like YOLO and Faster R-CNN, depend heavily on the Intersection over Union (IoU) and confidence thresholds to evaluate model performance. However, fixed threshol...Artificial Intelligence (AI)-based object detection models, like YOLO and Faster R-CNN, depend heavily on the Intersection over Union (IoU) and confidence thresholds to evaluate model performance. However, fixed thresholds can be biased and may have a disparate effect across subpopulations, even when traditional performance metrics suggest strong model performance. This paper examines how varying IoU and confidence thresholds affect standard evaluation metrics such as precision, recall, F1-score, and mean Average Precision (mAP) along with their effect on five widely used fairness metrics - Demographic Parity, Equalized Odds, Equality of Opportunity, Accuracy Equality, and Disparate Impact. This study evaluated a dataset of bruise images under natural and alternative light sources and found that fairness and performance trade-offs can be mitigated by selecting intermediate threshold values rather than fixed extremes. In addition, the results highlight the need for dynamical optimization of thresholds to achieve both, high model performance and fairness, in AI-driven bruise detection.
Migraine is a common but complex neurological disorder that doubles the lifetime risk of cryptogenic stroke (CS). However, this relationship remains poorly characterized, and few clinical guidelines exist to reduce this...Migraine is a common but complex neurological disorder that doubles the lifetime risk of cryptogenic stroke (CS). However, this relationship remains poorly characterized, and few clinical guidelines exist to reduce this associated risk. We therefore propose a data-driven approach to extract probabilistically-independent sources from electronic health record (EHR) data and create a 10-year risk-predictive model for CS in migraine patients. These sources represent external latent variables acting on the causal graph constructed from the EHR data and approximate root causes of CS in our population. A random forest model trained on patient expressions of these sources demonstrated good accuracy (ROC 0.771) and identified the top 10 most predictive sources of CS in migraine patients. These sources revealed that pharmacologic interventions were the most important factor in minimizing CS risk in our population and identified a factor related to allergic rhinitis as a potential causative source of CS in migraine patients.
Cooper LN, Vadsariya A, Varghese M
… +10 more, Nayee B, Moon J, Katterapalli C, Walker C, Gonzalez C, Sohal S, Lehmann CU, Velasco F, Basit M, Willett D
AMIA Annu Symp Proc
· 2024 · PMID 41726461
The University of Texas Southwestern Medical Center (UTSW) and Texas Health Resources (THR) implemented an Observational Medical Outcomes Partnership (OMOP) common data model (CDM) that utilizes the Epic electronic healt...The University of Texas Southwestern Medical Center (UTSW) and Texas Health Resources (THR) implemented an Observational Medical Outcomes Partnership (OMOP) common data model (CDM) that utilizes the Epic electronic health record's (EHR) extract, transform, and load (ETL) system to enable collaborative research with other health institutions within the OHDSI network. We mapped EHR data from core Epic reporting tables to 25 OMOP CDM tables and transferred the data to a shared OMOP database housed within the Caboodle infrastructure using Epic's pre-existing ETL system, minimizing the need for customization. ETL processes occur weekly at THR and daily at UTSW. OMOP CDM mapping resulted in data quality assessment values of 97% and 98% for THR and UTSW respectively. Our study established a reproduceable, collaborative pipeline using the OMOP CDM with Epic's native ETL framework, expanding the OHDSI research network resulting in better quality and more generalizable data sets available for future research.
Despite extensive literature on alert fatigue, gaps remain in understanding its impact. We examined drug alert volume and override rates across provider roles to inform future research. We retrospectively analyzed drug...Despite extensive literature on alert fatigue, gaps remain in understanding its impact. We examined drug alert volume and override rates across provider roles to inform future research. We retrospectively analyzed drug allergy alerts (DAA) and drug-drug interaction (DDI) alerts in 2023 at a large academic medical center. Alert volume and override rates were compared across providers with prescribing authority. Among 1,799 providers, 196,225 alerts were generated with an average of 0.42 alerts per clinician day. Advanced practice providers (APPs) received significantly more alerts per day than residents or attending physicians. Most providers (88%) saw fewer than one alert daily. Override rates increased with higher alert burden (98.6% for >5 alerts/day vs. 94.5% for 1-5 and 92.6% for <1; p<0.001). Alert fatigue may not be observed in all cases when analyzed at the provider level. Future research should explore other alert characteristics, measurements, and provider's perceptions.
Haji HE, Tariq A, Souadka A
… +4 more, Sbihi N, Batalini F, Ghogho M, Banerjee I
AMIA Annu Symp Proc
· 2024 · PMID 41726459
Breast cancer treatment involves surgery, radiation, chemotherapy, and endocrine therapy, with recurrence risk depending on treatment execution. We propose a weighted Cox mixtures model that integrates treatment plans an...Breast cancer treatment involves surgery, radiation, chemotherapy, and endocrine therapy, with recurrence risk depending on treatment execution. We propose a weighted Cox mixtures model that integrates treatment plans and clinical data to estimate recurrence risk. Data from Mayo Clinic (US) and the National Institute of Oncology (Morocco) inform the model. We enhance expectation maximization within the Cox mixtures model using three weighting strategies: Inverse Probability of Treatment Weighting, Adaptive Weights with focal loss, and Prioritizing Subgroups. In the Mayo Clinic cohort, Adaptive Weights improve predictive accuracy (C-index: 0.67-0.88), outperforming the standard Cox model. In the Moroccan cohort, Adaptive Weights also enhance C-index values (0.60-0.71), though with larger confidence intervals. Our findings demonstrate that weighting strategies refine recurrence risk prediction, particularly in imbalanced cohorts. Expanding datasets, especially in underrepresented populations, is crucial for improving model reliability and clinical applicability.
Osborne T, Abbasi S, Hong S
… +7 more, Sexton R, Ambut J, Patel NJ, Rosenthal RN, Ung L, Wang F, Wong R
AMIA Annu Symp Proc
· 2024 · PMID 41726458
Large language models (LLMs) have demonstrated potential to automate clinical documentation tasks that may reduce clinician burden, such as generation of hospital discharge summaries. Prior research used older LLMs and l...Large language models (LLMs) have demonstrated potential to automate clinical documentation tasks that may reduce clinician burden, such as generation of hospital discharge summaries. Prior research used older LLMs and limited data, raising concerns about fabrications and omissions. In this study, we evaluated the automatic generation of inpatient Internal Medicine discharge summaries using a HIPAA-compliant Microsoft Azure instance of OpenAI's GPT-4o. Both human-written and AI-generated discharge summaries were scored by Internal Medicine hospital faculty for quality, readability/conciseness, factuality and completeness, presence of hallucinations/omissions and their impact on safety, and compared with the actual discharge summaries. Our results showed that the AI-generated discharge summaries significantly outperformed actual human written summaries in both quality and readability/conciseness and were comparable to humans in factuality and completeness, with a minimal cost.
BACKGROUND: As Hackensack Meridian Health (HMH) experienced an influx of deceased patients during the COVID-19 pandemic three circumstances recurred: 1) the location of deceased patients were not being tracked properly,...BACKGROUND: As Hackensack Meridian Health (HMH) experienced an influx of deceased patients during the COVID-19 pandemic three circumstances recurred: 1) the location of deceased patients were not being tracked properly, 2) the belongings and bodies of the deceased were being lost throughout the system, and 3) bodies were being released before autopsies were performed. HMH took on the U.S. Surgeon General's challenge to reduce healthcare worker burnout by developing a morgue tracking process called Decedent Affairs. OBJECTIVES: To standardize the postmortem checklists for the adult, pediatric, neonatal and fetal demise populations and consolidate centralized documentation in a Decedent Affairs Navigator. This would promote transparency in the morgue tracking process, reduce redundancy in documentation, improve overall burnout, and aid in HMH's devotion to being a high reliability organization (HRO). METHOD: A multidisciplinary team customized the Decedent Affairs navigator to HMH's needs. Key features included Shared Workspaces, Reduction of Documentation Burden, and Organization of Decedent Tracking. The navigator went network-live on October 18th, 2023. Implementation approaches included the use of on-site workshops and digital tutorials. RESULTS: There was an 83 % rate decrease in the number of incidents of missing decedent items and/or bodies. Duplicative documentation reduced by 11.4%, saving documentation time by 31 seconds. CONCLUSION: The Decedent Affair navigator offers a niched and centralized hub within Epic to document information related to death procedures that can simplify workflows and reduce frequencies of lost bodies and personal properties of deceased patients in healthcare organizations.
Huang Y, Tao S, Chou WC
… +3 more, Cui L, Zhang GQ, Li X
AMIA Annu Symp Proc
· 2024 · PMID 41726456
COVID-SPHERE is a self-service web application designed to advance clinical research informatics by facilitating secondary use of Electronic Health Records (EHR) for COVID-19 research. The system employs a flexible EHR c...COVID-SPHERE is a self-service web application designed to advance clinical research informatics by facilitating secondary use of Electronic Health Records (EHR) for COVID-19 research. The system employs a flexible EHR concept framework that defines hierarchical concepts and ontologies, enabling clinical researchers to build complex temporal queries through an intuitive, single-click interface without requiring database expertise. Our method dynamically generates MongoDB queries in real-time and offers interactive, faceted visualizations to analyze longitudinal patient activities and integrated health records, supporting both individual patient analysis and population-level research. Hosted on a server managing over 5 TB of data encompassing 30 billion health records spanning 15 years from more than 8.8 million patients, this work demonstrated its generalizability by supporting multiple published research studies investigating various COVID related research topics on epidemiology, treatment outcomes, and long-term sequelae since November 2020. By simplifying the cohort discovery process, COVID-SPHERE reduces the informatics barriers between researchers and EHR data, enhancing the efficiency of clinical and translational research while promoting data-driven insights for COVID-19 surveillance and intervention. Its architecture is applicable to other large-scale clinical research data warehouses, offering a model for future public health informatics systems.
Wu Y, Ren Y, Jia H
… +4 more, Harrison T, Fan J, Liu H, Huang M
AMIA Annu Symp Proc
· 2024 · PMID 41726455
Efficient triage and response to patient portal messages (PPMs) are critical for enhancing patient-centered care. To improve the understanding of primary and secondary concerns expressed by patients, this study annotated...Efficient triage and response to patient portal messages (PPMs) are critical for enhancing patient-centered care. To improve the understanding of primary and secondary concerns expressed by patients, this study annotated and analyzed a set of 2,239 PPMs. We also automated the patient concern identification and analysis by leveraging pretrained language models with binary classification to discern all patient concerns and with multi-class classification to identify primary patient concerns. These multi-class classifications were further enhanced by integrating convolutional neural networks that utilize embeddings from the binary classification. This approach demonstrated significant potential of AI in managing the growing volume of PPMs and promptly addressing the healthcare needs of patients, thereby facilitating more effective and timely medical interventions.
Secure messaging (SM) between patients and providers has seen increasing adoption over the past decade, prompting development of computational methods to enable automation and research to enhance clinical efficiency and...Secure messaging (SM) between patients and providers has seen increasing adoption over the past decade, prompting development of computational methods to enable automation and research to enhance clinical efficiency and quality of communication. Through a systematic review, we examined the extant literature to investigate how previous studies had applied computational analyses to SM data. After screening 1,374 papers, we identified 19 relevant studies published between 2017 and 2024, most of which focused on applications for streamlining clinical workflows, facilitating early disease detection, supporting personalized decision making, and enhancing patient health literacy. Among the computational methods used, BERT was consistently shown to deliver best performance. However, all existing studies were constrained by small-size datasets and limited healthcare settings, leading to inadequate validation and poor generalizability. The results of this review highlight key research gaps, particularly the need for more robust computational approaches that ensure scalability, fairness, and clinical applicability.
Yang Q, Sharma A, Calin D
… +3 more, de Crecy C, Inampudi R, Yin R
AMIA Annu Symp Proc
· 2024 · PMID 41726453
Hepatic fibrosis poses a significant health risk for young adults with type 2 diabetes (T2D). We propose FCFNets, a novel factual and counterfactual learning framework to predict hepatic fibrosis in young adults with T2D...Hepatic fibrosis poses a significant health risk for young adults with type 2 diabetes (T2D). We propose FCFNets, a novel factual and counterfactual learning framework to predict hepatic fibrosis in young adults with T2D that can address class imbalance issue and increase interpretability leveraging electronic health records (EHRs). We designed a hybrid UNDO oversampling strategy, combining random and dissimilar oversampling that improves dataset diversity and model robustness. FCFNets also integrates SHAP-based global and instance-level explanations, alongside feature interaction analysis, providing insights into critical risk factors associated with hepatic fibrosis. The results show our proposed model outperforms various baseline methods with high sensitivity (0.846) and accuracy (0.768), while delivering counterfactual explanations. Hyperparameter tuning and dropout analysis further refine the model, ensuring optimal performance. This study demonstrates FCFNets's potential for early detection and personalized management of hepatic fibrosis, paving the way for interpretable AI applications in precision medicine.
Lee Y, Kang MJ, Baris VK
… +7 more, Lowenthal G, Rossetti SC, Cato KD, Lee RY, Kramer J, Huffam R, Dykes PC
AMIA Annu Symp Proc
· 2024 · PMID 41726452
The Communicating Narrative Concerns Entered by RNs Early Warning System (CONCERN EWS) is a machine-learning predictive model that analyzes nursing documentation patterns to detect early signs of patient deterioration, w...The Communicating Narrative Concerns Entered by RNs Early Warning System (CONCERN EWS) is a machine-learning predictive model that analyzes nursing documentation patterns to detect early signs of patient deterioration, with proven effectiveness in reducing the risk of in-hospital mortality and length of stay. This study extends the evaluation of CONCERN EWS beyond acute care settings to a rural community hospital, assessing user experience, system utilization, and accuracy in identifying patient deterioration. The study examined accuracy in a rural setting using a mixed-methods approach-qualitative interviews, quantitative data analysis, and clinical record reviews. Findingssuggestthat CONCERNEWS enhancesearly recognition of clinical deterioration andisusable in the context of busy acute care nursing workflows. These results support its adaptability to facilitate strengthening nursing surveillance, clinical decision-making, and patient safety in rural healthcare settings as well.
This study presents a Python-based Extract, Transform, and Load (ETL) pipeline that converts Medicare Limited Data Set (LDS) claims into the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM). By m...This study presents a Python-based Extract, Transform, and Load (ETL) pipeline that converts Medicare Limited Data Set (LDS) claims into the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM). By mapping Medicare LDS tables to fifteen OMOP CDM tables, we achieved minimal data loss. Rigorous validation using the OMOP Data Quality Dashboard indicated a 99% pass rate across over 1,500 checks, affirming data fidelity. A comparative analysis showed high concordance in demographic traits and clinical conditions between the original and transformed datasets. Despite structural constraints and minor syntax errors leading to some unmapped codes, our approach preserves key administrative details and standardizes healthcare data for large-scale observational research. This scalable, reproducible pipeline addresses critical gaps in Medicare-LDS-to-OMOP conversion, improving data integration for diverse applications in health services research, population health, and policy analysis. Future expansions will incorporate additional clinical details and advanced concept mappings.
Finkelstein J, Smiley A, Echeverria C
… +1 more, Mooney K
AMIA Annu Symp Proc
· 2024 · PMID 41726450
Predicting symptom escalation in chemotherapy patients is essential for proactive intervention and improved clinical outcomes. This study leverages hybrid deep learning architectures, specifically Convolutional Neural Ne...Predicting symptom escalation in chemotherapy patients is essential for proactive intervention and improved clinical outcomes. This study leverages hybrid deep learning architectures, specifically Convolutional Neural Networks with Long Short-Term Memory (CNN-LSTM), to forecast the progression of 12 self-reported symptoms, categorized into physical (e.g., nausea, fatigue, pain) and mental (e.g., anxiety, cognitive impairment, mood changes) groups. The dataset consists of daily self-reported symptom logs from individuals undergoing chemotherapy. Given the high class imbalance-where 84% of cases showed no escalation-symptom data were aggregated into intervals of 3 to 7 days to improve predictive performance and temporal resolution. The CNN-LSTM model combines convolutional layers for extracting patterns within a local time window with LSTM layers for capturing long-term temporal dependencies. The model was trained using five-fold cross-validation to ensure robustgeneralization. Results indicate that 5-day intervals yielded the highest predictive accuracy for physical symptom prediction, with the CNN-LSTM model achieving an accuracy of 83%, precision of 89%, recall of 86%, F1-score of 88%, and an AUCof 83%. These findings highlight the effectiveness of hybrid deep learning architectures in symptom monitoring and early detection, enabling AI-driven decision support for real-time clinical interventions. Integrating these models into digital health systems could facilitate continuous symptom tracking, enhance predictive accuracy, and improve the quality of care for chemotherapy patients.
Chen HY, Ostropolets A, Weng C
… +1 more, Hripcsak G
AMIA Annu Symp Proc
· 2024 · PMID 41726449
Medical vocabularies are essential tools for capturing, classifying, and analyzing healthcare data. However, the creation and maintenance of these vocabularies are often labor-intensive and costly. This preliminary study...Medical vocabularies are essential tools for capturing, classifying, and analyzing healthcare data. However, the creation and maintenance of these vocabularies are often labor-intensive and costly. This preliminary study evaluates the feasibility of using large language models (LLMs) to automate three key tasks in medical vocabulary management: term similarity, subsumption, and grouping. Using 1,533 cardiovascular terms from SNOMED CT, we applied GPT-4o and assessed the performance of 3 elementary tasks against OHDSI standardized vocabularies. While LLMs demonstrated high precision across tasks (0.78 for term similarity, 0.74 for term subsumption, 0.78 for term grouping), recall was notably lower (0.41 for term similarity, 0.08 for term subsumption, 0.52 for term grouping), indicating gaps in coverage. Overall, LLMs show promise for medical vocabulary tasks but require further refinement for clinical specificity and completeness. Future work should focus on enhancing recall, reducing hallucinations, and evaluating scalability across broader terminology sets.
Yang S, Wu Y, Liu M
… +3 more, Bian J, Liang M, Guo Y
AMIA Annu Symp Proc
· 2024 · PMID 41726448
This study developed and evaluated methods to adjust misclassification errors in electronic health record (EHR)-derived covariates using group-wise and individualized weights based on observed sensitivity and specificity...This study developed and evaluated methods to adjust misclassification errors in electronic health record (EHR)-derived covariates using group-wise and individualized weights based on observed sensitivity and specificity to reduce bias in predictive modeling. Logistic regression, XGBoost, and neural networks predicted follow-up adherence in lung cancer screening. The Lung-RADS category, extracted via natural language processing (NLP), was adjusted using group-wise weights and individualized weights from kernel and multinomial regression. Models with adjusted covariates were compared to naïve (unadjusted) and oracle (true value) models. Performance assessed by the area under the receiver operating characteristic (AUROC) curve across 10%, 20%, and 30% validation sets, showed that adjusted models outperformed naïve models, improving AUROC by 0.3%-10.4%. Compared to oracle models, adjusted models reduced the AUROC gap to 2.0%-7.5%. Individualized weights provided more precise corrections than group-wise weights. This scalable framework mitigates misclassification bias in EHR-derived covariates, enhancing predictive accuracy without resource-intensive manual review.
Li Z, Wang W, Shahani L
… +5 more, Vieira RM, Selek S, Soares J, Liu H, Huang M
AMIA Annu Symp Proc
· 2024 · PMID 41726447
Clinical phenotyping is the process of extracting patient's observable symptoms and traits to better understand their disease condition. Suicide phenotyping focuses more on behavioral and cognitive characteristics, such...Clinical phenotyping is the process of extracting patient's observable symptoms and traits to better understand their disease condition. Suicide phenotyping focuses more on behavioral and cognitive characteristics, such as suicide ideation, attempt, and self-injury, to identify suicide risks and improve interventions. In this study, we leveraged the latest reasoning models, namely 4o, o1, and o3-mini, to perform note-level multi-label classification and reasoning generation tasks using previously annotated psychiatric evaluation notes from a safety-net psychiatric inpatient hospital in Harris County, Texas. Compared with the previously finetuned GPT-3.5 model, the out-of-box reasoning models prompted with in-context learning achieved comparable and better performance, with the highest accuracy of 0.94 and F1 of 0.90. We implemented novel clinical justification generation from these models on the traditional classification tasks. This finding marked a promising direction for performing clinical phenotyping that is interpretable and actionable using smaller, efficient reasoning models.
Hu D, Guo Y, Cho HN
… +7 more, Chow E, Mukamel DB, Sorkin D, Reikes A, Perret D, Pandita D, Zheng K
AMIA Annu Symp Proc
· 2024 · PMID 41726446
The increasing burden of responding to large volumes of patient messages has become a key factor contributing to physician burnout. Generative AI (GenAI) shows great promise to alleviate this burden by automatically draf...The increasing burden of responding to large volumes of patient messages has become a key factor contributing to physician burnout. Generative AI (GenAI) shows great promise to alleviate this burden by automatically drafting patient message replies. The ethical implications of this use have however not been fully explored. To address this knowledge gap, we conducted a qualitative interview study with 21 physicians who participated in a GenAI pilot program. We found that notable ethical considerations expressed by the physician participants included oversight as ethical safeguard, transparency and patient consent of AI use, patient misunderstanding of AI's role, and patient privacy and data security as prerequisites. Additionally, our findings suggest that the physicians believe the ethical responsibility of using GenAI in this context primarily lies with users, not with the technology. These findings may provide useful insights into guiding the future implementation of GenAI in clinical practice.
Huang X, Zhou H, Hong Y
… +3 more, Zhou X, de Jong J, Wang Z
AMIA Annu Symp Proc
· 2024 · PMID 41726445
Family health history is an important component to assess risk for common chronic diseases. The integration of electronic health records and genetic data offers great potential to improve disease risk prediction by captu...Family health history is an important component to assess risk for common chronic diseases. The integration of electronic health records and genetic data offers great potential to improve disease risk prediction by capturing both clinical and genetic risk factors. We present ALIGATEHR-Gen, a graph attention network that integrates multimodal patient data including genetic information, diagnosis codes, and demographics, along with external medical ontology knowledge. ALIGATEHR-Gen constructs unified patient representations by incorporating genetically inferred first-degree relationships and disease ontology embeddings to enhance disease risk prediction. We evaluate the predictive performance of ALIGATEHR-Gen across 118 diseases in the UK Biobank and demonstrate that it outperforms state-of-the-art baseline models by an average of at least 6%. A case study on five primary fibrotic and closely related diseases reveals that ALIGATEHR-Gen effectively distinguishes patient subgroups based on clinical and genetic features. These findings illustrate the potential of ALIGATEHR-Gen to advance predictive and interpretable modeling in healthcare.
Intimate partner homicide (IPH) remains a major yet understudied cause of maternal mortality among U.S. women of childbearing age (WCBA). We leveraged the National Violent Death Reporting System (NVDRS) and county-level...Intimate partner homicide (IPH) remains a major yet understudied cause of maternal mortality among U.S. women of childbearing age (WCBA). We leveraged the National Violent Death Reporting System (NVDRS) and county-level Maternal Vulnerability Index (MVI) data from 2018-2022 to train three machine learning models-logistic regression, random forest, and XGBoost-to classify whether homicides were IPH. Among 11,498 homicides involving WCBA, 33% were IPH. XGBoost achieved the best performance (F1-score = 0.83, AUPRC = 0.87), prompting further examination of key predictors via model explainability. Results indicated that acute interpersonal conflicts (e.g., arguments, jealousy), prior IPV victimization, and structural vulnerabilities (e.g., reproductive healthcare access, physical environment) were influential predictors of IPH. By illustrating the interplay of individual, interpersonal, and broader community-level risk factors, our study shows how machine learning can inform multilevel strategies to prevent IPH and improve maternal health.