Predicting frequent emergency care use with machine learning no better than existing models

10th February 2023

Prediction of frequent emergency care use by those with chronic conditions with machine learning models is not superior to existing approaches

Predicting frequent emergency care use by patients with chronic health conditions with machine learning models does not offer any additional benefit to existing modelling approaches according to the findings of a retrospective analysis by Canadian researchers.

While, not universally accepted, those with at least three or more visits per year have been used to describe a frequent emergency care user. Such individuals often have complex health needs which are not met through primary care provision and consequently their condition deteriorates, leading to an emergency department (ED) visit. Although frequent emergency care users (FECU) represent only a small proportion of the overall population seen in an ED, they do account for a disproportionately large number of visits. Currently, logistic regression models have been used for analysing frequent users in emergency departments. However, the development of machine learning models that can incorporate large amounts of both clinical and non-clinical data, have the potential to help identify FECU individuals. Such models have already been used, for example, in predicting the need for hospitalisation at the time of triage for children with an asthma exacerbation. Nevertheless, no studies have explored the use of machine learning models – and how these compare with logistic regression – to predict frequent emergency care use in adults with chronic conditions.

In the present study the Canadian team retrospectively examined the performance of four machine learning models in comparison to a logistic regression model, for the prediction of frequent ED use in adults with a range of chronic conditions. They identified two cohorts: those who had at least 3 and 5 visits per year. The models used a number of predictor variables including age, gender and residential area and focused on chronic diseases such as coronary artery disease, mental disorders, epilepsy and chronic non-cancer pain. The models were used to predict frequent ED use as a binary outcome, i.e., frequent user or not and the outcomes were compared in terms of the area under the curve (AUC), sensitivity and specificity.

Frequent emergency care user model predictions

The analysis identified 451,775 ED users and of whom, 9.5% had at least three visits per year and 3% five visits.

The AUC for the logistic regression model for frequent users with 3 visits/year was 0.748, giving a sensitivity of 60% and a specificity of 78%. Two of the machine learning models gave a similar AUC (0.749 and 0.744) whereas the random forest model was much worse (AUC = 0.538). For prediction of frequent users with 5 visits/year, the model performance was broadly similar, i.e., machine learning-based models were no better.

Overall, the authors commented on how none of the machine learning models outperformed the logistic regression model and the most important predictor variable was the number of visits in the previous year. The authors did feel that access to more variables could have helped in refining the predictive accuracy of the machine learning models. Nevertheless, they emphasised the need for future work to consider complex non-linear interactions, since in such cases, machine learning models were likely to be superior to existing ones.

Citation
Chiu YM et al. Machine learning to improve frequent emergency department use prediction: a retrospective cohort study. Sci Rep 2023

Predictive model identifies factors linked to metformin failure in type 2 diabetics

23rd January 2023

A machine learning model using electronic health record data revealed factors associated with metformin treatment failure in type 2 diabetics

A gradient boosting (XGBoost) model, which made use of both clinical and demographic factors, was able predict the most important factors associated with metformin failure in type 2 diabetics according to work by a team of US researchers.

It has been estimated that globally, some 415 million people are currently living with diabetes and the World Health Organization suggests that more than 95% of those with diabetes have type 2 disease. One of the most widely used anti-diabetic agents is metformin and the drug is suggested as a first-line treatment either alone or in combination for those with type 2 disease. Nevertheless, some evidence highlights that monotherapy with metformin is associated with treatment failure. In one study, for instance it was found that, the proportion of patients able to a achieve an HbA1c of below 7% in the first year, ranged from 19 to 86% of those started on metformin. Understanding the factors linked to a respond to metformin can therefore help to personalise medicine and allow for an early adjustment of therapy. However, determining which specific factors are relevant to metformin treatment failure from an examination of a patient’s electronic health record (EHR) is challenging. In an attempt to identify relevant predictors held within the patient’s EHR, in the current study, the US researchers made use of a machine learning model and turned to a patient cohort with at least one abnormal diabetic result (e.g., elevated fasting glucose or HbA1c) that lead to the initiation of metformin treatment. For the purposes of they study, the team defined treatment failure as either an inability to achieve a target HbA1c or < 7% within 18 months of initiation or the addition of other pharmacological agents during the same time frame. Many of the EHR factors were assimilated into the model and included demographics (age, gender, ethnicity), lifestyle factors (smoking status), body mass index as well as laboratory findings such as lipid profiles, blood pressure and liver function tests. The predictive value of the model was assessed using the C-index and individual predictors using Shapley Additive Explanations (SHAP), for which higher values indicated a more important contribution to the model.

Model predictors and metformin failure

The study included 22,047 patients with a mean age of 57 years (48% female) who were started on metformin, Using the target of an HbA1c of below 7%, the overall metformin failure rate was 33% and the median time to failure was 3.9 months.

When the XGBoost model included baseline values of HBA1c, age, gender and ethnicity, it had a high discrimination performance for the risk of metformin failure (C-index = 0.73, 95% CI 0.72 – 0.74). There were a total of 15 different influential factors identified that impacted on metformin failure, the most important of which, was the baseline HBA1c value (SHAP value = 0.76). In contrast, factors such as age (SHAP = 0.016) or body mass index (SHAP = 0.041) were less important. Nonetheless, incorporation of each of these 15 factors did improve the model’s performance (C-index = 0.745, 95% CI 0.73 – 0.75, p < 0.0001).

The authors concluded that although baseline HBA1c was the most important factor in metformin failure, adding other important and readily available variables to the model, improved its performance. They suggested that is was therefore possible to easily identify patients most at risk of metformin failure and who would benefit from closer monitoring and earlier treatment modification.

Citation
Bielinski SJ et al. Predictors of Metformin Failure: Repurposing Electronic Health Record Data to Identify High-Risk Patients. J Clin Endrocrinol Met 2023

Machine learning diffusion tensor imaging models diagnose sleep apnoea

24th October 2022

Machine learning diffusion tensor imaging models have the potential to screen for brain changes associated with the presence of sleep apnoea

Two machine learning diffusion tensor imaging models were able to successfully distinguish between healthy controls and patients with obstructive sleep apnoea, according to a study by US researchers from California.

Obstructive sleep apnoea (OSA) is an extremely common condition with a 2019 study estimating that globally, 936 million adults aged 30-69 years have mild to severe disease. OSA leads to a significant impairment in quality of life related to physical functioning as well as causing daytime sleepiness, decreased learning skills and neuro-cognitive deficits that include impaired episodic memory, executive function, attention and visuospatial cognitive functions. The gold standard diagnostic test for OSA is overnight polysomnography which requires a dedicated sleep laboratory and trained staff to interpret the results. In contrast, magnetic resonance imaging (MRI) and in particular, diffusion tensor magnetic resonance imaging, has already been used to reveal how global brain mean diffusivity values are significantly reduced in OSA compared with controls.

The increased use of machine learning models with various imaging modalities, led the US researchers to wonder if a machine learning diffusion tensor imaging model might be able to detect the brain changes associated with OSA. After all, this approach had already been successfully used to identify major depressive disorder. For the present study, the team focused to two types of machine learning models; a support vector machine (SVM) and a random forest (RF) to assess mean diffusivity maps from brain MRI scans. Both models were trained and compared for their ability to accurately identify OSA and cross-validated within the training dataset.

Machine learning diffusion model’s accuracy for detecting OSA

The researchers recruited 59 patients (mean age 50.2 years, 61% male) with OSA and who had been recently diagnosed through overnight polysomnography with at least moderate disease severity and who were also treatment naive. In addition, 18 OSA and 29 control patients who were not included in the training set and to assess the predictive accuracy of the models.

The cross-validation process showed that the accuracy of the SVM model was 0.73 whereas the RF model was 0.77, i.e., both models showed similar fitting accuracy for OSA and control data. Similarly, the area under the receiver-operator curve was 0.85 for the RF model and 0.84 for the SVM model.

The authors concluded that both the RF and SVM models were comparable for the diagnosis of OSA and suggested that either could be used as a screening tool for OSA in patients where diffusion tensor imaging data was available.

Citation
Pang Bo et al. Machine learning approach for obstructive sleep apnea screening using brain diffusion tensor imaging J Sleep Res 2022