Prognostic performance of the Rapid Emergency Medicine Score (REMS) and Worthing Physiological Scoring system (WPS) in emergency department
© Ha et al. 2015
Received: 28 March 2015
Accepted: 20 May 2015
Published: 4 June 2015
The Rapid Emergency Medicine Score (REMS) and Worthing Physiological Scoring system (WPS) have been developed for predicting in-hospital mortality in nonsurgical emergency department (ED) patients. The prognostic performance of the scoring systems in independent populations has not been clear. The aim of the study is to evaluate the prognostic accuracy of REMS and WPS systems in the estimation of 30-day mortality risk among medical patients in ED.
The study was designed as a prospective investigation, with the setting being the ED of the National Hospital of Can Tho, Vietnam. We enrolled medical patients aged 16+ years who met the study entry criteria. Clinical data were obtained as required for each scoring system. The primary outcome was mortality within 30 days since hospitalization. The association between each scoring system and mortality was assessed by the hazard ratio (HR) of the Cox’s proportional hazard model.
The study involved 1746 patients, average age 65.9 years (SD 17). During the period of follow-up, 172 patients (9.9 %) died. The risk of 30-day mortality was increased by 30 % for each additional REMS unit (HR: 1.28; 95 % confidence interval (CI): 1.23–1.34) and by 60 % for each additional WPS unit (HR: 1.6; 95 % CI: 1.5–1.7). The AUC of the REMS was 0.71 (95 % CI: 0.67–0.76) which was significantly lower than that of the WPS (0.80; 95 % CI: 0.76–0.83).
Both REMS and WPS have good prognostic value in the prediction of death in ED patients. The WPS appeared to have a better prognostic performance than the REMS system.
The assessment of outcomes among patients admitted to a non-surgical medical emergency department (ED) is a challenging task. This is true because ED patients are characterized by a broad spectrum of illnesses and disease severity. Although clinical judgment by experienced clinicians is highly valuable, the judgment is often unreliable, particularly in patients who are in between the two extremes of very seriously ill and ill . Therefore, a number of algorithms have been developed to identify high-risk patients who might need intensive intervention. These algorithms have also been used as a triage tool for allocating appropriate resources on emergency and post-emergency period. Among the algorithms, the Rapid Emergency Medicine Score (REMS) , Rapid Acute Physiology Score (RAPS) , Worthing Physiological Scoring system (WPS) , Routine Laboratory Data (RLD) , and Admission Laboratory Tests (ALT)  have been widely used in practice .
The REMS (Additional file 1) and WPS (Additional file 2) are relatively simple and highly applicable to ED patients, because the input variables are readily available in most ED patients. Both REMS and WPS scoring systems have been validated on European patients in modern hospital environment [8, 4]. In a validation study, REMS was found to be better than RAPS in terms of predicting in-hospital mortality in ED patients, with the area under the receiver operating characteristic curve (AUC) being 0.74 for REMS and 0.64 for RAPS . Furthermore, WPS was found to have good discriminatory power in mortality prediction, with AUC being 0.74 . Nevertheless, it is not clear whether the prognostic performance of the two scoring systems is similar in ED patients, since there were no head-to-head comparison studies. Moreover, both REMS and WPS have not been studied in developing countries, and as a result, their prognostic performance in Asian patients remains largely unknown.
We hypothesized that in Asian patients the prognostic performance of the REMS and WPS in predicting mortality is comparable. The present study was designed to test the hypothesis by pursuing the following specific aims: (1) to determine the prognostic significance of REMS and WPS in predicting the risk of 30-day mortality in Asian patients admitted to ED and (2) to compare the predictive values of REMS and WPS in the patients.
Setting and patients
The study procedures and protocol were approved by the Scientific Research and Ethics Board of the National Hospital of Can Tho (Vietnam). This was a prospective observational investigation, conducted in the ED of the National Hospital of Can Tho, in the Mekong Delta (Vietnam). The hospital is a tertiary teaching hospital that serves 17 million residents in the Mekong Delta region. On average, the hospital’s ED admits 75 non-surgical patients per day.
We enrolled all medical patients aged 16 years and older. “Medical patients” were defined as those were admitted from non-trauma causes with no surgical indication, except surgery for stroke, upper gastrointestinal bleeding, and primary spontaneous pneumothorax. Patients were excluded from the study if they had at least one of the following conditions: acute coronary syndrome, burns, cardiac arrest before admitting to the hospital or which occurred in the ED with failure of cardiopulmonary resuscitation, snakebite, insect bite or sting, poisoning (drugs, alcohol intoxication, paraquat, insecticides, rodenticides, corrosive substances). These patients were transferred to more specialized hospitals for treatment. Patients under 16 years of age, women in labor, and those declared dead on arrival were also excluded from the study.
Due to ethical requirements, patients could withdraw from study at any time without giving reasons. However, in reality, reasons for withdrawal included non-compliance with therapy, transfer to another hospital for more specialized care, loss of follow-up, and family-initiated discharge.
All patients who met the inclusion criteria were invited to participate in the study. Written informed consent was obtained from all patients. If patients were unconscious or aged less than 18 years, their close relatives or guardians signed the written informed consent. Unconscious patients who regained the cognitive ability were re-consented within 30 days of hospitalization.
Upon the consent was obtained, data collection was conducted by a trained research worker using a structured data form. Patient characteristics, medical history, and physiological data were obtained as required for each scoring system and inclusion criteria (Additional files 1, 2, and 3). Physiological data were recorded at baseline (i.e., first measurement) by medical or nursing staff. After 30 days from the day of admission, the research worker made a call to a relative or delegated person to obtain information on vital status. Patients who had stayed in hospital for more than 30 days were considered “censored”.
The primary outcome of the study was mortality within 30 days since the time of admission to the hospital. Mortality was defined as (a) patients who died in hospital from any cause; (b) family-initiated discharged patients who died either on the way home or within 24 h after the hospital discharge; (c) doctor-initiated discharged patients who died at home. It should be noted that in Vietnamese culture, patients prefer to die at home rather than in hospital. As a result, when a patient is in the terminal stage, the patient or family usually requests for discharge from hospital. The secondary outcome was length of hospital stay, which was determined from the time of hospital admission to the time of discharge.
The research staff recorded data on pulse, body temperature, blood pressure (automatic blood pressure monitor, OMRON HEALTHCARE Co, Vietnam), breathing rate, peripheral oxygen saturation (NONIN Co, USA), days of therapy from other hospital(s), doses of vasopressor agents, cardiopulmonary resuscitation, mechanical ventilation, comorbidity, and functional status (Additional file 3). In some cases (approximately 5 %) where blood pressure was too low or too high, the automatic blood pressure monitor could not measure, and we manually took the blood pressure measurements.
Data management and analysis
We used the double data entry approach to enter data. The first data entry was undertaken within a day after hospital admission. The second data entry was done at the end of the study. Data from the two entries were then independently verified to check for potential discrepancies. Any discrepancy was adjudicated with the original patient record.
Data analysis was conducted according to a plan that was set out prior to the data collection. In the first stage, we calculated descriptive statistics (i.e., mean, standard deviation, median with interquartile range, proportion) for each physiological variable for the REMS and WPS systems with stratification by survival status. Categorical variables will be tested by Fisher’s exact test. The distribution of continuous variables was tested by Shapiro-Wilk normality test. The relationship between continuous variables and 30-day mortality will be tested by the Student’s t test for normally distributed variables or permutation test for non-normally distributed variables. In the next stage, we used the Cox’s proportional hazards model to evaluate the association between REMS and WPS and mortality risk. The discrimination of each scoring system was assessed by AUC value. The difference between two AUCs was tested by the DeLong method . All statistical analyses were performed with the R Statistical Environment version 3.1.0 .
Between March 13, 2013 and June 1, 2013, we had enrolled 2179 patients into the study. However, 71 patients were excluded from the analysis because they did not meet inclusion/exclusion criteria; 271 patients were discontinued from the study. The reasons for discontinuation were as follows: loss to follow-up (28 patients), family-initiated discharge or transfer to another hospital (243 patients). Among the remaining 1837 patients, there were 91 patients with missing values. Therefore, the data were analyzed from 1746 patients.
The median length of stay was 7 days (IQR: 4 to 10). During the follow-up period, 172 patients died within 30-day after admission, and this represents a mortality rate of 9.9 % (95 % confidence interval (CI): 8.5–11.3 %). The 5-day mortality rate was 5.7 % (95 % CI: 4.6–6.9), and this rate increased progressively with time of hospitalization.
Baseline characteristics of 1746 ED patients stratified by survival status
Alive after 30 days
Deceased within 30 days
n (%) or median (Q1, Q3)
n (%) or median (Q1, Q3)
Number of patients
Number of female patients (%)
68 (55, 80)
71 (59, 81)
Pulse (per min)
90 (78, 103)
100 (83, 115)
Body temperature (°C)
37.1 (37.0, 37.7)
37.3 (37.0, 38.1)
Systolic blood pressure (mmHg)
139 (120, 159)
131 (110, 158)
Diastolic blood pressure (mmHg)
80 (70, 90)
78 (63, 90)
Mean arterial pressure (mmHg)
99 (87, 112)
96 (80, 110)
Breathing rate (per min)
22 (20, 26)
26 (22, 30)
Peripheral oxygen saturation (%)
97 (94, 99)
93.5 (87, 97)
Days of therapy from other hospital (days)
0 (0, 29)a
0 (0, 63)a
Days from illness onset (days)
1 (0, 3)
1 (0, 2)
0 (0.0, 6.8)a
0 (0.0, 10)a
0 (0.0, 2.0)a
0 (0.0, 2.8)a
0 (0.0, 9.8)a
0 (0.0, 11.0)a
Length of stay (days)
7 (4, 10)
3 (1, 8)
Glasgow coma score
15 (15, 15)
15 (10, 15)
Unresponsive to all stimuli
Responds to painful stimuli
Responds to verbal stimuli
Admitted intensive care unit
Immunocompromised by agent
Leukemia or myeloma
Chronic renal failure
Chronic respiratory failure
Cirrhosis with ascites
6 (4, 8)
9 (6, 12)
2 (1, 4)
5 (3, 6)
Association between REMS factors and 30-day mortality risk in ED patients
Hazard ratio (95 % confidence interval)
Body temperature (°C)a
36.0 to 38.4
34.0 to 35.9/38.5 to 38.9
30.0 to 31.9/39.0 to 40.9
Mean arterial pressure (mmHg)a
70 to 109
50 to 70/110 to 130
130 to 159
Pulse (per minute)a
70 to 109
55 to 69/110 to 139
40 to 54/140 to 179
Breathing rate (per minute)a
12 to 24
10 to 11/25 to 34
35 to 49
Peripheral oxygen saturation (%)a
86 to 89
75 to 85
Glasgow coma scorea
11 to 13
8 to 10
5 to 7
45 to 54
55 to 64
65 to 74
REMS score (+1)
Association between WPS factors and 30-day mortality risk in ED patients
Hazard ratio (95 % confidence interval)
Breathing rate (per minute)a
Pulse (per minute)a
Systolic blood pressure (mmHg)a
Peripheral oxygen saturation (%)a
96 to 100
94 to 95
92 to 93
0 to 2
WPS score (+1)
The assessment of outcome in medical patients is a challenging task, because the diversity and variation in severity of the patient population. The difficulty in the assessment of ED patients is also related to the problem of assigning appropriate weight to each symptom and comorbidity. In that setting, it appears that clinical algorithms with statistical weighting can offer useful prognostic information. In this study, we compared the prognostic performance of two popular algorithms, REMS and WPS, in the prediction of mortality in ED patients. We found that both algorithms have good discrimination and calibration, but the WPS appeared to have better discriminatory power than the REMS system. These findings deserve more elaboration.
It is difficult to compare the present results with previous studies, because there have been no external validation studies for the WPS algorithm. In the original study where REMS was developed, the AUC was 0.852 , which was higher than our observed AUC (0.712). We note that in the original study, the mortality rate was 2.4 %, which is much lower than our study’s (9.9 %); the original population was from ED (~56 % hospitalized patients) which is also different from our target population. We also note that the REMS and WPS were designed to predict the risk of in-patient mortality, whereas in our study the outcome was 30-day mortality, and this difference could partly explain the difference in AUC values. It is unlikely that any single prognostic model will perform well for a whole spectrum of seriously ill in ED . However, our study demonstrated that the calibration (e.g., agreement between observed and predicted probability) of the REMS system was very good.
Our result shows that the WPS system had better mortality discrimination than the REMS system. In the initial study , the AUC was 0.740, which was lower than the present study’s (0.797). The calibration of WPS was also reasonably good. We note that the relationship between mortality risk and WPS scores was approximately linear, such that patients in the top scores (10–11) had a risk of death over 10 times greater than those in the lowest range (0–1), suggesting a good discrimination.
However, it is interesting to observe that there was a modest correlation between the predicted risk of death by REMS and WPS systems. For a given WPS-predicted risk, there is a high variability in the REMS-predicted risk. Nevertheless, a combination of both REMS and WPS systems did not improve the AUC value over and above the AUC of the WPS alone, suggesting that REMS contributes little to the prediction of mortality by WPS.
The use of REMS or WPS is highly feasible in the ED setting, because the two systems use data that are either readily available from patients at the time of admission or routinely collected in the ED. However, both systems categorize continuous data with distinct groups, and this categorization could be a potential weakness. It is well known that any categorization of continuous data can result in loss of information and loss of statistical power. Thus, we consider that there is room for improvement of the predictive value of the REMS and WPS system.
To our knowledge, this is the first validation of REMS and WPS systems in a head-to-head way and in a developing country hospital. However, the AUC is still not confidently high, suggesting that false positives and false negatives are still a problem. Nevertheless, the use of WPS in conjunction of experienced clinical judgment can help better assess outcome for ED patients.
The present findings have to be considered within the context of strength and limitation of the study. The study was based on a large sample of patients, which allows the delineation of modest effect sizes otherwise not detected by smaller studies. The assessment of outcome and the data collection were rigorously done to ensure the accuracy and reliability of data. However, patients in this study were drawn from a population whose socioeconomic status and cultural values are not necessarily the same with the Caucasian populations whose REMS and WPS were based on. In terms of cultural values, it should be noted that Vietnamese patients prefer to pass away in their home rather than in hospital, and this is the reason for including death occurred at home after 24 h of hospital discharge as an outcome. Although the National Hospital of Can Tho is a major and teaching hospital in Vietnam, its resources and infrastructure are not necessarily as good as in most Western teaching hospitals, and the difference in mortality rates may be expected. In addition, the rate of loss to follow-up (~13 %) could potentially reduce the sensitivity of the scoring systems.
In summary, we have demonstrated in this study that both REMS and WPS scoring systems have good prognostic value in the prediction of mortality in ED patients in a tertiary teaching hospital in Vietnam. We also found that the WPS has a better discrimination than the REMS system. The application of WPS together with clinical judgment may provide useful information to clinicians in the allocation of resources and treatment of ED patients.
We acknowledge the assistance of the nursing and medical staff of the Emergency Department, National Hospital of Can Tho in the data collection. We also thank medical students of the Can Tho University of Medicine for helping in the collection of data.
- McClish DK, Powell SH. How well can physicians estimate mortality in a medical intensive care unit? Int J Soc Med Dec Making. 1989;9(2):125–32.View ArticleGoogle Scholar
- Olsson T, Terent A, Lind L. Rapid Emergency Medicine score: a new prognostic tool for in-hospital mortality in nonsurgical emergency department patients. J Intern Med. 2004;255(5):579–87. doi:10.1111/j.1365-2796.2004.01321.x.PubMedView ArticleGoogle Scholar
- Rhee KJ, Fisher Jr CJ, Willitis NH. The rapid acute physiology score. Am J Emerg Med. 1987;5(4):278–82.PubMedView ArticleGoogle Scholar
- Duckitt RW, Buxton-Thomas R, Walker J, Cheek E, Bewick V, Venn R, et al. Worthing physiological scoring system: derivation and validation of a physiological early-warning system for medical admissions. An observational, population-based single-centre study. Br J Anaesth. 2007;98(6):769–74. doi:10.1093/bja/aem097.PubMedView ArticleGoogle Scholar
- Prytherch DR, Sirl JS, Schmidt P, Featherstone PI, Weaver PC, Smith GB. The use of routine laboratory data to predict in-hospital death in medical admissions. Resuscitation. 2005;66(2):203–7. doi:10.1016/j.resuscitation.2005.02.011.PubMedView ArticleGoogle Scholar
- Froom P, Shimoni Z. Prediction of hospital mortality rates by admission laboratory tests. Clin Chem. 2006;52(2):325–8. doi:10.1373/clinchem.2005.059030.PubMedView ArticleGoogle Scholar
- Brabrand M, Folkestad L, Clausen NG, Knudsen T, Hallas J. Risk scoring systems for adults admitted to the emergency department: a systematic review. Scand J Trauma Resusc Emerg Med. 2010;18:8. doi:10.1186/1757-7241-18-8.PubMed CentralPubMedView ArticleGoogle Scholar
- Goodacre S, Turner J, Nicholl J. Prediction of mortality among emergency medical admissions. Emerg Med J. 2006;23(5):372–5. doi:10.1136/emj.2005.028522.PubMed CentralPubMedView ArticleGoogle Scholar
- DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837–45.PubMedView ArticleGoogle Scholar
- Team RC. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. 2014. http://www.R-project.org/. 2014.
- Challen K, Goodacre SW. Predictive scoring in non-trauma emergency patients: a scoping review. Emerg Med J. 2011;28(10):827–37. doi:10.1136/emj.2010.099291.PubMedView ArticleGoogle Scholar
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.