A comparative analysis of risk stratification tools for emergency department patients with chest pain
© Burkett et al.; licensee Springer. 2014
Received: 18 October 2013
Accepted: 29 January 2014
Published: 7 February 2014
Appropriate disposition of emergency department (ED) patients with chest pain is dependent on clinical evaluation of risk. A number of chest pain risk stratification tools have been proposed. The aim of this study was to compare the predictive performance for major adverse cardiac events (MACE) using risk assessment tools from the National Heart Foundation of Australia (HFA), the Goldman risk score and the Thrombolysis in Myocardial Infarction risk score (TIMI RS).
This prospective observational study evaluated ED patients aged ≥30 years with non-traumatic chest pain for which no definitive non-ischemic cause was found. Data collected included demographic and clinical information, investigation findings and occurrence of MACE by 30 days. The outcome of interest was the comparative predictive performance of the risk tools for MACE at 30 days, as analyzed by receiver operator curves (ROC).
Two hundred eighty-one patients were studied; the rate of MACE was 14.1%. Area under the curve (AUC) of the HFA, TIMI RS and Goldman tools for the endpoint of MACE was 0.54, 0.71 and 0.67, respectively, with the difference between the tools in predictive ability for MACE being highly significant [chi2 (3) = 67.21, N = 276, p < 0.0001].
The TIMI RS and Goldman tools performed better than the HFA in this undifferentiated ED chest pain population, but selection of cutoffs balancing sensitivity and specificity was problematic. There is an urgent need for validated risk stratification tools specific for the ED chest pain population.
Chest pain remains one of the most common presenting complaints in patients presenting to emergency departments (ED), with >100,000 patients being hospitalized each year in Australia with acute coronary syndromes (ACS) . The morbidity, mortality and economic costs associated with this constitute a significant burden on the Australian health system . In these patients, use of risk stratification tools to predict risk of death, cardiac complications and the pre-test probability of ACS has been demonstrated to aid clinicians in appropriate prioritizing of patients for investigations [3, 4] and assists in identification of those at higher risk who might benefit most from potent drug therapies or an early invasive therapeutic approach [5–10]. Risk stratification tools have further been shown to allow patients to be better informed of their prognosis , improve cost-effectiveness while minimizing unnecessary treatment complications  and reduce unnecessary admissions to inpatient monitored beds, without increasing complications, thereby potentially having a positive impact on access block [3, 6, 12, 13].
Recently, in Queensland, a suite of clinical pathways for the management of patients presenting to public hospitals with chest pain was developed , with application incentivized by practice improvement payments . These pathways rely on risk stratification utilizing the National Heart Foundation of Australia risk stratification tool (HFA), perhaps the most prominent risk stratification utilized in Australia . Although the individual components of this tool are evidence based and the tool was developed by consensus of an expert panel, it was designed for risk stratification of patients with ACS rather than for the undifferentiated ED chest pain population. There are conflicting data regarding its performance in ED chest pain populations [17, 18].
The aim of this prospective observational study was to compare the performance of three methods of risk stratification, namely, the National Heart Foundation of Australia risk stratification tool (HFA) , the Goldman score  and the Thrombolysis in Myocardial infarction (TIMI) risk score  for prediction of a composite outcome of major adverse cardiac events (MACEs) within 30 days of ED attendance.
This was a prospective, cohort study undertaken at a single tertiary referral ED. Patients presenting to the ED with non-traumatic chest pain during the preceding 48 h and aged >30 years were eligible for inclusion in the study. Exclusion criteria were the presence of a definitive non-ischemic cause for chest pain, isolated angina-equivalent symptoms, trauma-related chest pain, cardiac arrest on arrival to the ED, patients with ECG criteria for ST-elevation myocardial infarction (MI) on arrival to the ED and inability to provide informed consent.
Data collection occurred during the ED presentation on weekdays that a trained research nurse was available. Follow-up was undertaken by both review of medical records utilizing a standardized data collection tool and a phone call to the patient (or proxy if the patient was not contactable) employing a structured interview at 72 h and 30 days. A single emergency physician, blinded at the time to the patient outcomes, retrospectively undertook the risk stratification process using the prospectively collected data items and initial ED-acquired electrocardiogram.
Risk stratification tools
Presentation with clinical features consistent with ACS and any of:
• Repetitive or prolonged (>10 min) ongoing chest pain/discomfort
• Elevation of at least one cardiac biomarker (troponin or CK-MB)
• Persistent of dynamic ST depression ≥0.5 mm or new T wave inversion ≥2 mm
• Transient ST segment elevation (≥0.5 mm) in more than two contiguous leads
• Hemodynamic compromise: systolic BP <90 mmHg, cool peripheries, diaphoresis, Killip class >1 and/or new onset mitral regurgitation
• Sustained ventricular tachycardia or syncope
• Left ventricular systolic dysfunction (LVEF <40%)
• Prior PCI within 6 months or prior CABG
• Presence of known diabetes or chronic kidney disease (eGFR <60 ml/min) with typical symptoms of ACS
Presentation with clinical features consistent with ACS and any of:
• Chest pain or discomfort within the past 48 h that occurred at rest or was repetitive or prolonged (but currently resolved)
• Age >65 years
• Known coronary artery disease: prior MI with LVEF ≥40% or known coronary lesion >50% stenosis
• No high-risk ECG changes
• Two or more of: known hypertension, family history, active smoking and hyperlipidemia
• Presence of known diabetes or chronic kidney disease (eGFR <60 ml/min) with atypical symptoms of ACS
• Prior aspirin use
AND NOT meeting the criteria for high-risk NSTEACS
Presentation with clinical features consistent with ACS without intermediate- or high-risk features
• Onset of angina symptoms within the last month
• Worsening in severity or frequency of angina
• Lowering in angina threshold
TIMI RS 19
1 point for each positive factor
• Age >65 years
• Documented prior coronary artery stenosis >50% or prior cardiac catheterization with known disease or PCI or prior CABG or documented prior myocardial infarction
• 3 or more conventional cardiac risk factors (hypertension, diabetes, cholesterol elevation, family history of coronary artery disease/MI, history of tobacco use)
• Use of aspirin in the preceding 7 days
• 2 or more angina events in the past 24 h
• ST-segment elevation or depression >1 mm
• Elevated cardiac biomarkers
Very low risk
• No ECG evidence of acute ischemia/MI and none of the following urgent factors:
▪ Rales above both lung bases
▪ Systolic BP <100 mmHg
▪ Unstable IHD (worsening of previously stable angina, new onset of post-infarction angina or angina after a coronary revascularization procedure or pain that was the same as associated with a prior MI)
No ECG evidence of acute ischemia/MI and 1 of above urgent factors
No ECG evidence of acute ischemia/MI and 2 or 3 of above urgent factors
OR ECG evidence of acute ischemia AND 0 or 1 of above urgent factors
ECG evidence of AMI alone OR ECG evidence of acute ischemia with 2 or 3 of above urgent factors
The primary outcome of interest was MACE within 30 days of ED presentation. MACE components were defined utilizing the American College of Cardiology Clinical Data Standards definitions  and included acute myocardial infarction (prevalent and incident), recurrent ischemia requiring urgent revascularization, cardiogenic shock, ventricular arrhythmia requiring emergent intervention or high-grade atrioventricular block requiring treatment, cardiac arrest and all-cause mortality.
For analysis, continuous variables with normal distribution were expressed as medians and interquartile ranges; categorical data were presented as percentages. Group differences in continuous and categorical variables are compared with Kruskal-Wallis and chi-square tests respectively. For each of the risk stratification methods, ROC curves were used to evaluate its predictive performance. Area under the curve (AUC) was utilized as a summary measure for diagnostic accuracy of the prediction tools across the gamut of risk groups , with 95% confidence intervals (CI). For the comparison of clinical performance of the risk scores, we chose to include AHA high risk, Goldman high risk and two cutoffs of the TIMI score (≥1 and ≥2). The latter were chosen pragmatically a priori and attempted to balance case discrimination and sensitivity. Inclusion of patients with a TIMI score of zero provides no case discrimination, while using a cutoff of ≥3 has been shown to have a sensitivity <60% .
For all comparisons a p-value of <0.05 was considered statistically significant. Statistical analysis was performed with Stata, version 10 (College Station, TX, USA). Power calculations were generated as follows: the AUC of the TIMI score has previously been demonstrated to be 0.6 , while the Goldman score has been shown to have an AUC of 0.9 [3, 6]. The performance of the HFA score was expected to be similar to that of the Goldman. Assuming a correlation between positive and negative groups of 0.4, a sample size of 400 patients was required to distinguish an AUC of 0.85 from one of 0.9, at a p value of 0.05 with an 80% power. An interim analysis was performed as approved study duration and funding were nearing their end. The study was terminated early as this interim analysis revealed a clearly statistically significant result. The institutional ethics committee approved the study.
Age (years; median, interquartile range)
Risk factors (%)
Medications on arrival to hospital (%)
Aspirin within last 7 days
Angiotensin-converting enzyme inhibitors/angiotensin receptor blockers
Clinical findings on admission
Heart rate (median, interquartile range)
Systolic blood pressure (median, interquartile range)
Signs of heart failure (%)
New ST segment depression or T wave inversion (%)
Troponin I > 0.04 ng/ml (%)
Results of previous investigations
Left ventricular ejection fraction < 40%
Previous coronary artery disease with known > 50% stenosis (%)
Prior history of revascularization (%)
Percutaneous coronary intervention (%)
Coronary artery bypass graft (%)
Comparative performance of risk stratification tools
Sensitivity% (95% confidence interval)
Specificity% (95% confidence interval)
AUC (95% confidence interval)
All high-risk patients
All patients with a risk category of low or higher
TIMI RS of ≥1
TIMI RS ≥2
Utilizing the chi-square test to compare the AUC for the TIMI, Goldman and HFA tools, the difference between the tools in terms of predictive ability for MACE was found to be highly significant (chi2, N = 276, p <0.0001). Further analysis showed that the HFA tool performance was different from that of each of the other tools, which were similar in performance.
In this study of ED patients with chest pain, where no alternative non-ACS cause was apparent, none of the tools under investigation were ideal. Characteristics of an ideal tool for stratifying risk of chest pain in the ED population would be that it has high sensitivity for risk of MACE, specificity sufficient to enable feasible application and, from a patient perspective, limit exposure to unnecessary investigations or interventions. It would also have data elements that were non-subjective or, at least, had high interobserver reliability. The TIMI risk score had the best performance in stratifying risk for MACE. However, utilizing a cutoff of ≥1 to achieve a sensitivity of 97% resulted in a specificity of only 13%. A higher cutoff of >3 would be required to achieve a more acceptable specificity of 60%; however, the resultant sensitivity of 74% is unacceptable. The sensitivity of the Goldman score (69%) was insufficient to be useful in the ED population, even with the lowest possible cutoff, namely including all patients with a risk higher than ‘very low’ risk. The HFA risk stratification tool had an AUC of 0.54, with 95% confidence intervals (0.45-0.63) disturbingly encompassing an AUC of <0.5. Its sensitivity for MACE, using a cutoff to include all high-risk patients, was 100%, but clearly with a specificity of 8%, its feasibility in ED clinical practice is limited.
The use of an unstructured or individualized approach to ED assessment of chest pain has been shown to be associated with high resource utilization for patients with no coronary artery disease, while concurrently resulting in significant proportions of patients with ACS being missed . Subsequently, emphasis has been placed on utilization of risk stratification tools. Unfortunately, studies employing risk stratification tools for chest pain in the ED setting often have two significant limitations. First, the tools employed have largely been designed for risk stratification of those with known ACS, usually patients admitted to the hospital. Utilizing these tools in the undifferentiated ED patient with chest pain may have a significant impact on both safety and efficiency. Second, many trials of chest pain risk stratification have tested the tools on ED populations where their chest pain is “thought to be of ischaemic origin.” However, as physician discrimination of ACS from other causes of chest pain has previously been shown to be poor, this may result in potentially flawed inferences .
The TIMI tool is widely reported as being utilized in the undifferentiated chest pain population in the ED, including as part of rapid diagnostic protocols . In this study, TIMI had comparable sensitivity and specificity for MACE to that found in previously published studies , but again highlights that the selection of an appropriate cutoff, balancing sensitivity, specificity and clinical feasibility, is problematic.
Earlier studies identified a higher sensitivity for the Goldman tool in the ED setting than our results [3, 26]. This disparity may be accounted for by the fact that these studies differed in their inclusion criteria, with a focus on patients admitted to the hospital, including patients with ST elevation MI, and/or had much earlier follow-up for the endpoint of MACE. Additionally, both the definition of MI and the available cardiac biomarkers have changed considerably since these earlier studies.
The HFA risk stratification tool had an AUC of 0.54. Its high sensitivity for MACE (100%) came at the cost of a specificity of only 8.4%, reflecting classification of 93% of patients as being at “high risk.” This finding is concordant with a previously published study, which similarly questioned the HFA risk stratification tool’s suitability for use in this patient cohort . If the HFA decision support tool  were applied to the patient population in this study, it would lead to 93% of ED chest pain patients being admitted to hospital wards and receiving treatments such as heparin. The weight of evidence suggests that the HFA risk stratification tool is inappropriate for use in chest pain pathways in the unselected ED chest pain population.
Other approaches to risk stratification of ED chest pain patients have recently been reported. The HEART score [4, 27] was developed in The Netherlands based on clinical experience and literature review rather than database methods. It has five components – history, ECG, age, risk factors and troponin level – which are each rated 0, 1 or 2 based on criteria. In a validation study, it was shown to have better discriminative performance than the GRACE and TIMI risk scores . In multicentre validation, a low HEART score (≤3) had a 1.7% rate of MACE . Of concern is that this score relies in part on subjective assessment of likelihood of ACS for which inter-rater data are scarce.
Another approach was taken by the GRACE investigators with the development of a score aimed at predicting the absence of MACE. The GRACE freedom from events score  was developed in an admitted chest pain cohort with likely ACS and has undergone limited external validation on admitted chest pain cohorts [30, 31]. No validation in an ED chest pain cohort has yet been published.
A similar approach has been taken in the development of the North American Chest Pain Rule (NACPR) . It aims to identify low-risk ED chest pain patients suitable for early discharge. The rule consists of the absence of five predictors—ischemic ECG changes not known to be old, history of coronary artery disease, pain typical for ACS, initial or 6-h troponin level greater than the 99th percentile and age greater than 50 years. In internal validation, it was 100% sensitive (95% confidence interval 97.2% to 100.0%) and 20.9% specific (95% confidence interval 16.9% to 24.9%) for a cardiac event within 30 days, with 11% of patients being defined as low risk . Its utility has been challenged in an external validation study and comparison to the HEART score . The NACPR identified 4.4% (95% CI 3-6%) for early discharge with 100% (95% CI 98-100%) sensitivity for ACS, while the HEART score identified 20% (95% CI 18-23%) for early discharge with 99% (95% CI 97-100%) sensitivity for ACS. The low proportion of patients identified as low risk in a population with MACE of 22% is a serious threat to this score’s clinical utility. That said, the approach of trying to decide who is safe for discharge rather than identification of high risk is worthy of further exploration.
This study has some limitations that should be considered in interpreting the results. Bias in the verification of events is possible if the inability to contact patients in follow-up coincided with those having events. The potential for verification bias has been minimized by follow-up not only via chart review but also via admissions registry review and with the patient or proxy. While most data were collected prospectively, some were collected retrospectively and so are subject to potential data omission. Some of the data were reliant on patient self-report (e.g., of past history and risk factors). If available in the chart, this was verified; however, otherwise, no attempt was made to verify these data, reflecting the real-world ED clinical interaction. The study was conducted at a single site, and this may have impacted the external validity.
The TIMI risk stratification score appeared most suitable for use in an undifferentiated ED chest pain population, but selection of an appropriate cutoff is problematic. This study highlights the need for validated risk stratification tools for the ED chest pain cohort that examine not only their safety in terms of their sensitivity, but also their flow and efficiency impacts, as these factors have significant implications for safety for all ED patients.
- Hacker S, Madden R: Australia’s Health. Australian Institute of Health and Welfare. Canberra: AIHW Cat. No. AUS 44; 2004.Google Scholar
- Begg S, Vos T, Barker B, et al.: The burden of disease & injury in Australia 2003. In Australian Institute of Health and Welfare. Canberra: AIHW; 2007.Google Scholar
- Reilly BM, Evans AT, Schaider JJ: Impact of a clinical decision rule on hospital triage of patients with suspected acute cardiac ischemia in the emergency department. JAMA 2002, 288: 342–350. 10.1001/jama.288.3.342View ArticlePubMedGoogle Scholar
- Six AJ, Backus BE, Kelder JC: Chest pain in the emergency room: value of the HEART score. Netherlands Heart J 2008, 16: 191–196. 10.1007/BF03086144View ArticleGoogle Scholar
- Lagerqvist B, Diderholm E, Lindahl B, et al.: FRISC score for selection of patients for an early invasive treatment strategy in unstable coronary artery disease. Heart 2005, 91: 1047–1052. 10.1136/hrt.2003.031369PubMed CentralView ArticlePubMedGoogle Scholar
- Goldman L, Cook F, Johnson PA, et al.: The need for intensive care in patients who come to emergency departments with acute chest pain. New Eng J Med 1996, 334: 1498–1504. 10.1056/NEJM199606063342303View ArticlePubMedGoogle Scholar
- Sabatine M, Antman E: The thrombolysis in myocardial infarction risk score in unstable angina/non ST-segment elevation myocardial infarction. J Am Coll Cardiol 2003, 41: S89-S95. 10.1016/S0735-1097(02)03019-XView ArticleGoogle Scholar
- Sabatine MS, McCabe CH, Morrow DA, et al.: Identification of patients at high risk for death and cardiac ischemic events after hospital discharge. Am Heart J 2002, 143: 966–970. 10.1067/mhj.2002.122870View ArticlePubMedGoogle Scholar
- Pollack CV, Sites FD, Shofer FS, et al.: Application of the TIMI risk score for unstable angina and non-ST elevation acute coronary syndrome to an unselected emergency department chest pain population. Acad Emerg Med 2006, 13: 13–18. 10.1111/j.1553-2712.2006.tb00978.xView ArticlePubMedGoogle Scholar
- Yan AT, Yan RT, Tan M, et al.: Risk scores for risk stratification in acute coronary syndromes: useful but simpler is not necessarily better. Eur Heart J 2007, 28: 1072–1078. 10.1093/eurheartj/ehm004View ArticlePubMedGoogle Scholar
- Cakir B, Blue K: How to improve the management of chest pain: hospitalists and use of prediction rules. Southern Med J 2007, 100: 242–247. 10.1097/SMJ.0b013e31802f7f94View ArticlePubMedGoogle Scholar
- Durairaj L, Reilly B, Das K, et al.: Emergency Department admissions to inpatient cardiac telemetry beds: a prospective cohort study of risk stratification and outcomes. Am J Med 2001, 110: 7–11.View ArticlePubMedGoogle Scholar
- Hollander JE, Sites FD, Pollack CV, et al.: Identification of cardiac death and life-threatening ventricular dysrhythmias in low-risk patients with chest pain. Ann Emerg Med 2004, 43: 71–76. 10.1016/S0196-0644(03)00719-4View ArticlePubMedGoogle Scholar
- Clinical Practice Improvement Centre statewide cardiac pathways. 2009. [cited 2009 Jan 15]; Available from: . Accessed Febuary 11, 2014 http://www.health.qld.gov.au/caru/pathways/pathways.asp
- Centre CPI (Ed): Clinical Practice Improvement Payment - user guide In version 2.0 ed: Queensland Health 2009, 1–18.Google Scholar
- Aroney C, Aylward P, Kelly A-M, et al.: Guidelines for the management of acute coronary syndromes 2006. Med J Aust 2006,184(Suppl):S1-S29.Google Scholar
- Cullen L, Greenslade J, Hammett CJ, et al.: Comparison of three risk stratification rules for predicting patients with acute coronary syndrome presenting to an australian emergency department. Heart Lung Circ 2013. epub ahead of printGoogle Scholar
- Kelly AM: How useful are the heart foundation risk criteria for assessment of emergency department patients with chest pain? Emerg Med Australas 2012, 24: 260–265. 10.1111/j.1742-6723.2012.01536.xView ArticlePubMedGoogle Scholar
- Antman EM, Cohen M, Bernink P, et al.: The TIMI risk score for unstable Angina/Non-ST elevation MI: a method for prognostication and therapeutic decision making. JAMA 2000, 284: 835–842. 10.1001/jama.284.7.835View ArticlePubMedGoogle Scholar
- Cannon CP, Battler A, Brindis RG, et al.: American College of Cardiology key data elements and definitions for measuring the clinical management and outcomes of patients with acute coronary syndromes. A report of the American College of Cardiology Task Force on Clinical Data Standards (Acute Coronary Syndromes Writing Committee). J Am Coll Cardiol 2001, 38: 2114–2130. 10.1016/S0735-1097(01)01702-8View ArticlePubMedGoogle Scholar
- Zou KH, O’Malley AJ, Mauri L: Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models. Circulation 2007, 115: 654–657. 10.1161/CIRCULATIONAHA.105.594929View ArticlePubMedGoogle Scholar
- Hess EP, Agarwal D, Chandra S, et al.: Diagnostic accuracy of the TIMI risk score in patients with chest pain in the emergency department: a meta-analysis. CMAJ 2010, 182: 1039–1044. 10.1503/cmaj.092119PubMed CentralView ArticlePubMedGoogle Scholar
- de Araujo GP, Ferreira J, Aguiar C, et al.: TIMI, PURSUIT, and GRACE risk scores: sustained prognostic value and interaction with revascularization in NSTE-ACS. Eur Heart J 2005, 26: 865–872. 10.1093/eurheartj/ehi187View ArticleGoogle Scholar
- Christenson J, Innes G, McKnight D, et al.: Safety and efficiency of emergency department assessment of chest discomfort. CMAJ 2004, 170: 1803–1807. 10.1503/cmaj.1031315PubMed CentralView ArticlePubMedGoogle Scholar
- Than M, Cullen L, Reid CM, et al.: A 2-h diagnostic protocol to assess patients with chest pain symptoms in the Asia-Pacific region (ASPECT): a prospective observational validation study. Lancet 2011, 377: 1077–1084. 10.1016/S0140-6736(11)60310-3View ArticlePubMedGoogle Scholar
- Goldman L, Cook EF, Johnson PA, et al.: Prediction of the need for intensive care in patients who come to the emergency departments with acute chest pain. New Eng J Med 1996, 334: 1498–1504. 10.1056/NEJM199606063342303View ArticlePubMedGoogle Scholar
- Backus BE, Six AJ, Kelder JC, et al.: A prospective validation of the HEART score for chest pain patients at the emergency department. Int J Cardiol 2013. doi: 10.1016/j.ijcard.2013.01.255. [Epub ahead of print]Google Scholar
- Six AJ, Cullen L, Backus BE, et al.: The HEART score for the assessment of patients with chest pain in the emergency department: a multinational validation study. Crit Pathw Cardiol 2013, 3: 121–126.View ArticleGoogle Scholar
- Brieger D, Fox KA, Fitzgerald G, et al.: Predicting freedom from clinical events in non-STelevation acute coronary syndromes: the Global Registry of Acute Coronary Events. Heart 2009, 95: 888–894. 10.1136/hrt.2008.153387View ArticlePubMedGoogle Scholar
- Kelly AM, Dabee P, Klim S, et al.: External validation of the GRACE Freedom from Events score. Heart Lung Circ 2012, 21: 582–585. 10.1016/j.hlc.2012.05.005View ArticlePubMedGoogle Scholar
- Soderholm M, Deligani MM, Choudhary M, et al.: Ability of risk scores to predict a low complication risk in patients admitted for suspected acute coronary syndrome. Emerg Med J 2012, 29: 644–649. 10.1136/emermed-2011-200328View ArticlePubMedGoogle Scholar
- Hess EP, Brison RJ, Perry JJ, et al.: Development of a clinical prediction rule for 30-day cardiac events in emergency department patients with chest pain and possible acute coronary syndrome. Ann Emerg Med 2012, 59: 115–125. 10.1016/j.annemergmed.2011.07.026View ArticlePubMedGoogle Scholar
- Mahler SA, Miller CD, Hollander JE, et al.: Identifying patients for early discharge: Performance of decision rules among patients with acute chest pain. Int J Cardiol 2012. doi: 10.1016/j.ijcard.2012.10.010. [Epub ahead of print]Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.