Predicting frequent emergency care use with machine learning no better than existing models

10th February 2023

Prediction of frequent emergency care use by those with chronic conditions with machine learning models is not superior to existing approaches

Predicting frequent emergency care use by patients with chronic health conditions with machine learning models does not offer any additional benefit to existing modelling approaches according to the findings of a retrospective analysis by Canadian researchers.

While, not universally accepted, those with at least three or more visits per year have been used to describe a frequent emergency care user. Such individuals often have complex health needs which are not met through primary care provision and consequently their condition deteriorates, leading to an emergency department (ED) visit. Although frequent emergency care users (FECU) represent only a small proportion of the overall population seen in an ED, they do account for a disproportionately large number of visits. Currently, logistic regression models have been used for analysing frequent users in emergency departments. However, the development of machine learning models that can incorporate large amounts of both clinical and non-clinical data, have the potential to help identify FECU individuals. Such models have already been used, for example, in predicting the need for hospitalisation at the time of triage for children with an asthma exacerbation. Nevertheless, no studies have explored the use of machine learning models – and how these compare with logistic regression – to predict frequent emergency care use in adults with chronic conditions.

In the present study the Canadian team retrospectively examined the performance of four machine learning models in comparison to a logistic regression model, for the prediction of frequent ED use in adults with a range of chronic conditions. They identified two cohorts: those who had at least 3 and 5 visits per year. The models used a number of predictor variables including age, gender and residential area and focused on chronic diseases such as coronary artery disease, mental disorders, epilepsy and chronic non-cancer pain. The models were used to predict frequent ED use as a binary outcome, i.e., frequent user or not and the outcomes were compared in terms of the area under the curve (AUC), sensitivity and specificity.

Frequent emergency care user model predictions

The analysis identified 451,775 ED users and of whom, 9.5% had at least three visits per year and 3% five visits.

The AUC for the logistic regression model for frequent users with 3 visits/year was 0.748, giving a sensitivity of 60% and a specificity of 78%. Two of the machine learning models gave a similar AUC (0.749 and 0.744) whereas the random forest model was much worse (AUC = 0.538). For prediction of frequent users with 5 visits/year, the model performance was broadly similar, i.e., machine learning-based models were no better.

Overall, the authors commented on how none of the machine learning models outperformed the logistic regression model and the most important predictor variable was the number of visits in the previous year. The authors did feel that access to more variables could have helped in refining the predictive accuracy of the machine learning models. Nevertheless, they emphasised the need for future work to consider complex non-linear interactions, since in such cases, machine learning models were likely to be superior to existing ones.

Citation
Chiu YM et al. Machine learning to improve frequent emergency department use prediction: a retrospective cohort study. Sci Rep 2023

MRI radiomics model predicts disease recurrence in triple negative breast cancer after neoadjuvant chemotherapy

13th December 2021

An MRI radiomics model shows good predictive accuracy for the 3-year triple negative breast cancer recurrence after neoadjuvant chemotherapy

An MRI radiomics model based on deep learning has shown good predictive accuracy at identifying whether patients with triple negative breast cancer (TNBC) would have systemic recurrence within three years of neoadjuvant chemotherapy, according to researchers from the Department of Radiology, Peking University, Beijing, China.

A triple negative breast cancer describes an aggressive tumour which lacks expression of oestrogen receptor, progesterone receptor and HER2 and accounts for around 15% of all breast cancers. Moreover, TNBC has been found to have the lowest 4-year survival rate at 77%.

Neoadjuvant chemotherapy is seen as the mainstay of treatment for early stage TNBC followed by definitive surgery although relapse within 3 years has been found to occur in up to 97% of patients. Magnetic resonance imaging (MRI) has been used to enable prediction of disease recurrence although these MRI radiomics models have required a radiologist to manually draw regions of interest (ROIs) as the input which is very time-consuming.

In the present study, the Chinese team used radiomics based on deep learning models and which utilised automated segmented ROIs of breast tumours from MRI images to predict whether patients with TNBC who had received neoadjuvant chemotherapy, would experience disease recurrence within three years of treatment. The team undertook a retrospective analysis of consecutive female patients with unilateral primary TNBC and for whom pre- and post-neoadjuvant MRI imaging was available. The researchers also collected clinico-pathologic factors from the patient medical records, e.g., menopausal status, the histological type of cancer, the clinical T and N stages. They developed a MRI radiomics model based on three separate features; pre-treatment features (model 1), post-treatment features (model 2) and a combination of pre and post-treatment MRI features (model 3). Multivariate analysis was used to assess associations between clinical factors and disease recurrence and the predictive performance of the models was assessed using the receiver operating characteristic curves and the area under the curves (AUC).

Findings

A total of 147 women with a median age of 49.5 years were included in the analysis, 104 (22 with disease recurrence) were used for the training cohort and 43 for the testing cohort (9 with disease recurrence) and the median time to disease recurrence was 17 months.

In both the univariate and multivariate analysis the clinical T stage and pathological T stage were significantly associated with systemic disease recurrence within three years of treatment (p < 0.05). Using these two variables in a clinical model yielded AUCs of 0.75 for the training cohort and 0.74 in the testing cohort. In comparison, the AUCs for each of the MRI radiomics model was 0.88 (model 1), 0.91 (model 2) and 0.96 (model 3).

Model 3 had a perfect sensitivity (i.e., 1) and a specificity of 0.85 compared to a sensitivity of 0.67 and specificity of 0.82 for the clinical model and this difference was statistically significant (p < 0.05). However, neither model 1 or 2 were significantly better than the clinical model.

Commenting on their findings, the authors speculated that the predictive ability of the third MRI radiomics model was likely to have a greater prognostic value by inclusion of post-neoadjuvant chemotherapy features that were reflective of treatment-induced changes.

They concluded that the value of their non-invasive model was in being able to more easily identify those at risk of disease recurrence and to therefore strengthen the treatment of these patients after surgery to improve their prognosis.

Citation

Ma M et al. Radiomics features based on automatic segmented MRI images: Prognostic biomarkers for triple-negative breast cancer treated with neoadjuvant chemotherapy Eur J Radiol 2021

Delayed transfer of care predictive model 70% accurate

16th November 2021

A delayed transfer of care can be reasonably well predicted based on eight routinely collected pieces of information upon admission.

A model which accurately predicts delayed transfer of care (DTOC) has been developed with only eight pieces of data routinely collected from patients upon admission to hospital. This was the finding of a retrospective study by a team from University Hospitals of North Midlands NHS Trust, Stoke-on-Trent, UK. Within a hospital setting it is necessary to ensure not only that patients receive appropriate clinical care but that they can be discharged either back home or to another setting, in a timely manner. A DTOC has become associated with the term ‘bed blocking” and a symbol of inefficiency within the national health service, occurring when a medically fit person is unable to go home or to another clinical setting and thus still occupies a hospital bed. In fact, national data for England shows that in February 2020 there were 155,700 total delayed days, of which 103,000 were in acute care, amounting to 5,370 people delayed per day. Furthermore, a report from the Department of Health in the UK estimated that in 2014-15 the cost due to discharge delay among patients over 65 years of age was £820 million.

With such enormous costs associated with DTOC, the Stoke-on-Trent team, sought to explore whether it was possible to identify the specific risk factors associated with DTOC among those patients admitted to hospital following attendance at an emergency department. They hypothesised that the capacity to predict which patients were more likely to experience a delayed transfer could enable earlier discharge planning.

The team turned to routinely collected data within the hospital including age, gender, ethnicity, national early warning score (NEWS), arrival by ambulance, the Glasgow admission prediction (GAP) score and an index of multiple deprivation (IMD) for their DTOC analysis. Using data on all adult patients admitted through the emergency department between January 2018 and December 2020, the team randomised these patients into a training and a validation dataset. Using the above and other variables, the team created a predictive model that included only statistically significant variables. The final model was assessed using the area under the receiver operating curve (AUC).

Findings

There were a total of 132,311 admissions over the 3-year period which were available for the delayed transfer of care analysis. The cohort had an overall mean age of 63 years (52% female) and over 90% were Caucasian. Initially, 10 variables were included in the predictive model, of which eight remained statistically significant: age, gender, ethnicity, GAP score, IMD, NEWS, arrival by ambulance, admitted in the last 12 months. Using all eight variables, the predictive DTOC model achieved a sensitivity of 0.77 (95% CI 0.75 – 0.78) and a specificity of 0.70 (95% CI 0.69 – 0.70) with an overall accuracy of 70%.

In their discussion, the authors noted that for example, patients arriving by ambulance were 13 times more likely to experience a DTOC. From a policy perspective, they suggested that use of the model would enable targeting of potential delayed patients for more proactive support.

They concluded that future studies should examine the potential effect of other factors and which together with machine learning, could improve the accuracy of prediction.

Citation

Davy A et al. A predictive model for identifying patients at risk of delayed transfer of care: a retrospective, cross-sectional study of routinely collected data. Int J Qual Health Care 2021

Three amino acids predict 90-day mortality in patients admitted to ED with dyspnoea

8th November 2021

Three amino acids, glycine, phenylalanine and valine were associated with the risk of 90-day mortality in ED patients admitted with dyspnoea.

Levels of three amino acids, glycine, phenylalanine and valine measured upon admission to an ED in patients admitted with dyspnoea are strongly predictive of 90-day mortality. This was the conclusion of a study by a team from the Department of Clinical Sciences, Lund University, Malmo, Sweden. Dyspnoea is a common presentation in an ED with one study of over 3,000 patients, finding that 5.2% of ED presentations, 11.4% of ward admissions and 19.9% of intensive care unit admissions were due to dyspnoea. There are a number of underlying conditions which can cause dyspnoea which presents as either an impaired ventilation or increased ventilatory demand, or some cases, both. Irrespective of the underlying cause, in patients with dyspnoea there is the release of stress hormones and metabolic changes, one of which is the induction of a catabolic state and insulin resistance.

Interestingly, some previous work has shown that the elevation of a combination of three amino acids could be used to successfully predict future diabetes. Based on these observations, the Lund University team hypothesised that the insulin resistance induced by stress in those with acute dyspnoea, would also alter levels of certain amino acids and that these alterations might be of valve in the assessment of dyspnoea severity and possibly even predictive of dyspnoea mortality.

In an effort to examine their hypothesis, the researchers retrospectively analysed patient data for those admitted to an ED with acute dyspnoea between 2013 and 2015. Plasma levels of nine amino acids were measured and Cox proportional hazard models used to explore the relationship between the level of these amino acids and the risk of 90-day mortality, which served as the primary endpoint for the study.

Findings

Data were analysed for a total of 663 patients with a mean age of 71.5 years (53.4% female), of whom 61% were admitted to a ward and 20.1% required intensive care treatment. Overall, 12% of patients died during the 90-day follow-up period. Only three amino acids of the original nine measured, demonstrated a significant association with 90-day mortality. These were glycine (hazard ratio, HR = 1.32, 95% CI 1.08 – 1.62, p < 0.001), phenylalanine (HR = 1.53) and valine (HR = 0.61).

Next, the researchers created an amino acid mortality risk score (AMRS) which was divided into quartiles and they found that in quartile 1, the 90-day mortality was 2.4% whereas it increased massively to 26.5% in quartile 4.

Commenting on these findings, the authors suggested that changes in the levels of these three amino acids, measured during presentation at the ED, were able to strongly predict 90-mortality in patients with acute dyspnoea, irrespective of the underlying cause. They concluded that a score using just these three amino acids could be used as a guide in risk assessment and to support decision-making to establish an appropriate level of care for patients presenting to an ED with acute dyspnoea.

Citation

Wiklund K et al. Amino acids predict prognosis in patients with acute dyspnea. BMC Emerg Med 2021