Prompting with electronic checklist improves clinician performance in medical emergencies: a high-fidelity simulation study
- Ronaldo Sevilla-Berrios†1, 2,
- John C. O’Horo†1, 2,
- Christopher N. Schmickl1, 2,
- Aysen Erdogan2, 3, 4,
- Xiaomei Chen1, 2, 5,
- Lisbeth Y. Garcia Arguello1, 2,
- Yue Dong1, 2,
- Oguz Kilickaya2, 3, 6,
- Brain Pickering2, 3,
- Rahul Kashyap2, 3Email authorView ORCID ID profile and
- Ognjen Gajic1, 2
© The Author(s). 2018
Received: 18 January 2018
Accepted: 9 April 2018
Published: 27 April 2018
Inefficient processes of care delivery during acute resuscitation can compromise the “Golden Hour,” the time when quick interventions can rapidly determine the course of the patient’s outcome. Checklists have been shown to be an effective tool for standardizing care models. We developed a novel electronic tool, the Checklist for Early Recognition and Treatment of Acute Illness (CERTAIN) to facilitate standardized evaluation and treatment approach for acutely decompensating patients. The checklist was enforced by the use of a “prompter,” a team member separate from the leader who records and reviews pertinent CERTAIN algorithms and verbalizes these to the team. Our hypothesis was that the CERTAIN model, with the use of the tool and a prompter, can improve clinician performance and satisfaction in the evaluation of acute decompensating patients in a simulated environment.
Volunteer clinicians with valid adult cardiac life support (ACLS) certification were invited to test the CERTAIN model in a high-fidelity simulation center. The first session was used to establish a baseline evaluation in a standard clinical resuscitation scenario. Each subject then underwent online training before returning to a simulation center for a live didactic lecture, software knowledge assessment, and practice scenarios. Each subject was then evaluated on a scenario with a similar content to the baseline. All subjects took a post-experience satisfaction survey. Video recordings of the pre-and post-test sessions were evaluated using a validated method by two blinded reviewers.
Eighteen clinicians completed baseline and post-education sessions. CERTAIN prompting was associated with reduced omissions of critical tasks (46 to 32%, p < 0.01) and 12 out of 14 general assessment tasks were completed in a more timely manner. The post-test survey indicated that 72% subjects felt better prepared during an emergency scenario using the CERTAIN model and 85% would want to be treated with the CERTAIN if they were critically ill.
Prompting with electronic checklist improves clinicians’ performance and satisfaction when dealing with medical emergencies in high-fidelity simulation environment.
Acute critical illness is routinely treated by highly trained staff in specialized care units. However, the initial resuscitation, the “Golden Hour” (after the initial hour following physiological insult or trauma is most crucial for successful resuscitation) can be impeded by inefficient processes of care delivery . A key factor contributing to this is that critically ill patients continually generate vast quantities of clinical data . This information can overwhelm providers, especially those not specifically trained to work in fast-paced, high-stress environments . It is crucial to develop strategies that streamline the processes of care to minimize clinical misjudgment .
Checklists are a validated tool for dealing with such challenges; long adopted by aviation and nuclear industries, they have recently been demonstrated invaluable in standardizing and improving clinical care . Haynes et al. showed a decreased in complications and 30-day mortality after non-cardiac surgery by implementing a surgical safety checklist .
However, checklist implementation has been slow in many settings, often due to a perception that the tool disturbs existing workflows. One approach to reduce the disruption is the presence of an additional team member, a checklist “prompter” specifically tasked with ensuring checklist completion. This has been tested during ICU  rounds, and it was associated with decreased mortality and shorter length of stay .
Proper care and timely interventions are critical in initial resuscitation [1, 9–12] especially in situations with diverse provider background and training . Algorithms like advance cardiac life support (ACLS) and advance trauma life supports (ATLS) have been created to structure the care provided in emergent cases. However, none of these algorithms address the more frequently encountered clinical problems like altered mental state, respiratory distress, syncope, and sepsis with a prescribed structured approach. Furthermore, these packages are based on memorizing and recall rather than providing easy access to informational cues to guide the resuscitation workflow.
It seems evident that providing an interface capable of summarizing this information and providing decision support and validated management algorithms would address a critical need . We have recently designed and developed a novel electronic tool, the Checklist for Early Recognition and Treatment of Acute Illness (CERTAIN), to apply a standard approach to evaluation and management of the acutely decompensating patient which included process of care workflows and a designated prompter . We hypothesized that a standardized approach to the evaluation and management of acutely decompensating patients using the CERTAIN model would improve clinicians’ performance and satisfaction in a simulated acute care environment.
Participants were recruited from the trainees and staff of a tertiary care teaching medical center in Rochester Minnesota. Subjects included medical students, medical and surgical residents and fellows, nurse practitioners, and physicians. To ensure a common minimum level of competence and training, all volunteers were required to have ACLS/BLS certification to be able to participate. All subjects consented to being recorded and having their performance analyzed as part of this study and the Institutional Review Board approved the study protocol. Each participant attended two sessions, a baseline evaluation without the use of the CERTAIN and a second session with the CERTAIN after receiving training.
The study was performed at a high-fidelity multidisciplinary simulation center equipped with technologically advanced mannequins programmed to show complex findings and react just as a patient would to treatment decisions and with video/audio recording capabilities .
Participants were asked to come back to the simulation center for a final session in groups of two to three clinicians in a minimum of 2 weeks for “washout” time. These 90-min sessions consisted of a brief 20-min didactic session, where we reviewed the most important components of the CERTAIN methodology and use, followed by a practical examination. The practical examination evaluated the participant’s ability to navigate the tool before allowing hands on the use of the CERTAIN in the simulator. A minimum passing grade of 80% was required before moving on. The participants were then allowed to practice with two to three nongraded scenarios to experience the team leader and prompter roles. A debrief period of 5 to 10 min was done after each practice session in order to give positive or corrective feedback as indicated. A final videotaped testing scenario was then evaluated, where the scenario was designed to reflect the same critical tasks as the baseline evaluation for that subject. Each scenario was designed to have two variations with similar clinical progression and scorable points but disguised with a different clinical vignette to minimize recall bias (see Additional files 1 and 2). Even though the clinicians were given feedback for usability, the actual testing components were not revealed to them at any point, prior to the final test. Ultimately, the participants were asked to complete an online survey regarding their opinion of the CERTAIN method on clinical practice  (see Additional files 3 and 4).
All video recordings were evaluated by two independent reviewers. In cases of disagreement, the reviewers had the opportunity to replay the video and clarify based on discussion. However, if disagreement persisted, a senior critical care physician, using the definition on the SOP, would review the video and adjudicate disagreements. Each scorable item was graded as either “done” or “not done,” and the time from simulation start to item completion was recorded. The data were analyzed with the use of JMP statistical software (JMP Version 7. SAS institute Inc. Cary, NC, 1989–2012). All reported P values are two-sided, and P values of less than 0.05 were considered to indicate statistical significance. Each subject served as its own control, so paired comparisons using McNemar and Wilcoxon signed rank test were used as appropriate. Pooling data analysis of total task, specific, and generic data was also presented as indication of group performance.
From 24 participants enrolled in the simulation study, 18 completed both baseline and follow-up assessments; 11 critical care fellows, two residents, two visiting clinicians, and three medical students. Two thirds of the study participants were clinicians with at least 6 months of formal critical care training and were considered with high level of expertise.
Proportion of individual task completion with and without CERTAIN prompting
Without CERTAIN (N = 13)
With CERTAIN (N = 18)
Code status discussion
Evaluation of vital sign
Evaluation of temperature
Review of past medical history
Review of home medication
Review of allergies
Order initial basic lab test
Start oxygen supplementation
Review of differential diagnosis
Total omissions (mean ± SD)
13 ± 4.3 (71%)
15 ± 2.7 (85%)
Time to task completion on those cases that have task completion on the pre- and post-intervention test
Number of cases available for assessment
Mean time change in seconds
Code status discussion
Evaluation of vital sign
Evaluation of temperature
Review of past medical history
Review of home medication
Review of allergies
Order initial basic lab test
Start oxygen supplementation
Review of differential diagnosis
Total time to task completion
11.7 ± 5.2
− 62.4 ± 68
Response score (N = 18)
Percentage of score with four points or higher (%)
The CERTAIN approach helps you feel better prepared during the emergency scenario
4.2 ± 1.0
The CERTAIN software was easy to use
2.7 ± 1.3
I would want to be treated by CERTAIN approach if I were critically ill or injured
4.2 ± 0.9
I think that checklist are useful in a medical emergency
4 ± 0.9
In a high-fidelity simulation environment prompting with the CERTAIN improved clinical task completion and decreased omission rates of critical tasks. Clinician satisfaction was high, with majority of participants wanting to incorporate this method to their own clinical practice.
Traditionally, resuscitation teams have been formed and trained to perform under stressful situations where they have to rapidly coordinate evaluation and treatment efforts [17, 18]. However, they have been structured around specific types of illness like trauma or cardiac arrest and largely rely on memory recall [18, 19]. Our experience demonstrated the performance of clinicians under stress is suboptimal at best, with an overall task completion rate of 50% at baseline. This is consistent with the findings of Smith et al. showing a decline in skill retention and loss of ability to perform ACLS and BLS skills to standard level when re-tested at 12 months .
As a potential solution, other studies have evaluated memory aids to improve health team performance in other situations. Haynes et al. tested the use of a checklist applied to regular operating room workflow on elective surgeries. His work showed a decrease on preventable surgical-related complications in the operating room . Two studies designed for emergency teams dealing with late-phase resuscitation, one with a smart phone application  and the other a traditional checklist , had promising results in simulated environments. However, their designs did not include a prompter, which likely reduced their team’s compliance. Prompting with the CERTAIN is aimed to approach critical illness resuscitation earlier in the natural course of the disease, targeting to standardize the care on the so-called Golden hour. Early structured treatment has been shown to give better outcomes in simulated operating room crises , sepsis , myocardial infarction , and other critical illness states and providing a unified approach to decompensation may prevent the need of cardiopulmonary resuscitation efforts.
In a different setting, Weiss et al. tested the usefulness of prompting in critical care practice by implementing the use of checklist with a prompter versus checklist alone during daily ICU rounds. This study showed improvement in compliance with process of care, decreased length of stay, and a decrease in mortality . These findings clearly demonstrate that important role prompting can play in facilitating complex process of care. The present study differs, however, in using a prompter in a higher stress environment with simulated acute medical emergencies. This stress imposes an extra burden on providers which could increase the risk of task omission. In this sense, the choreography of the CERTAIN with a prompter combines the lessons of prior studies on checklists and prompting with leadership “best practices” , such as egalitarian leadership  and closed-loop communication .
Ideally, one of the existing team members should be able to play a role of prompter, without any added cost. However, in resource constraint situations, the team lead should focus on resuscitation and as soon as the time allows review the checklist to see if anything has been missed.
The CERTAIN approach was well received by the participants. However, it is worth noting that software usability limitations were evident in the post-intervention survey. With only 33% of participants feeling the software is easy to use, a combination of improved training and interface may be necessary to make this practical in real high-stress environments. In this simulation study, due to volunteers’ time and schedule constraints, the training was limited to 90 min. Most initial training in clinical resuscitation models (ACLS, BLS, or ATLS) are 12 h courses (usually two full training days) .
Another potential limitation could be that, even though the didactic sessions and practice scenarios were focused on tool usability rather than the scenario performance, having these done just prior to final testing could have influenced the performance.
The inferences from our results are further limited due to the simulation nature of this study, as well as the small sample size. Simulation training has been increasingly suggested as a valid research and training tool paired with good outcomes which makes it the ideal scenario to test a new method and clinical software . The spectrum of scenarios encountered by our test subjects was limited to three common types: respiratory distress and hypoxia, hypotension due to severe sepsis/septic shock, and chest pain secondary to acute coronary syndrome. These cases were chosen as they are the most common clinical presentations in hospitalized medical patients [30–33]. Each clinician serving as his/her own control minimized the effect of variability in general medical knowledge. However, the absence of the control group of clinicians who were simply re-tested may limit the ability to discern the effectiveness of CERTAIN prompting vs training.
Prompting with an electronic checklist (CERTAIN) improves clinical performance, subjective perceptions, and confidence of bedside clinicians confronted with typical emergency medical scenarios in high-fidelity simulation environment. Enhanced design/usability and better training are needed to leverage potential benefits of electronic checklist at the bedside of critically ill patients.
Special thanks to Denise Foy and Penny Pries and Multidisciplinary Simulation Center for their support.
This publication was supported by CTSA grant number UL1 TR000135 from the National Center for Advancing Translational Science (NCATS). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH.
This publication was also supported by grant number 1C1CMS330964 from the Department of Health and Human Services, Centers for Medicare and Medicaid Services. The contents of this publication are solely the responsibility of the authors and do not necessarily represent the official views of the US Department of Health and Human Services or any of its agencies.
These funding bodies have no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.
Availability of data and materials
Please contact author for data requests.
RSB, JCO, YD, OK, PB, RK, and OG conceived the idea of the study. RSB, JCO, SNC, AE, XC, and LGA participated in conducting and data gathering for the study. RSB and JCO wrote the first draft, and all others helped in editing the manuscript. All authors read and approved the final manuscript.
Ethics approval and consent to participate
This study was approved by the Mayo Clinic IRB, and participant consent was waived as no identifying data is being used for reporting. No animal, human, or tissue data are presented in this manuscript.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
- Sebat F, Musthafa AA, Johnson D, et al. Effect of a rapid response system for patients in shock on time to treatment and mortality during 5 years*. Crit Care Med. 2007;35:2568–75.View ArticlePubMedGoogle Scholar
- Manor-Shulman O, Beyene J, Frndova H, Parshuram CS. Quantifying the volume of documented clinical information in critical illness. J Crit Care. 2008;23:245–50.View ArticlePubMedGoogle Scholar
- Woods DD, Patterson ES, Roth EM. Can we ever escape from data overload? A cognitive systems diagnosis. Cogn Tech Work. 2002;4:22–36.View ArticleGoogle Scholar
- Pickering BW, Hurley K, Marsh B. Identification of patient information corruption in the intensive care unit: using a scoring tool to direct quality improvements in handover*. Crit Care Med. 2009;37:2905.View ArticlePubMedGoogle Scholar
- Gawande A. The checklist manifesto: how to get things right. In: Vol 200: metropolitan books New York; 2010.Google Scholar
- Haynes AB, Weiser TG, Berry WR, et al. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med. 2009;360:491–9.View ArticlePubMedGoogle Scholar
- Weiss CH, Moazed F, McEvoy CA, et al. Prompting physicians to address a daily checklist and process of care and clinical outcomes: a single-site study. American Journal of Respiratory and CriticalCare Medicine. 2011;184:680.View ArticleGoogle Scholar
- Kilickaya O, Bonneton B, Gajic O. Structured Approach to Early Recognition and Treatment of Acute Critical Illness In Annual Update in Intensive Care and Emergency Medicine. 2014;2014(2014):689–703. https://doi.org/10.1007/978-3-319-03746-2_51 edited by Jean-Louis Vincent.
- Rivers EP, Nguyen HB, Sepsis AD. A landscape from the emergency department to the intensive care unit*. Crit Care Med. 2003;31:968–9.View ArticlePubMedGoogle Scholar
- Kumar A, Zarychanski R, Light B, et al. Early combination antibiotic therapy yields improved survival compared with monotherapy in septic shock: a propensity-matched analysis*. Crit Care Med. 2010;38:1773–85.View ArticlePubMedGoogle Scholar
- De Luca G, Suryapranata H, Ottervanger JP, Antman EM. Time delay to treatment and mortality in primary angioplasty for acute myocardial infarction. Circulation. 2004;109:1223–5.View ArticlePubMedGoogle Scholar
- Hacke W, Kaste M, Bluhmki E, et al. Thrombolysis with alteplase 3 to 4.5 hours after acute ischemic stroke. N Engl J Med. 2008;359:1317–29.View ArticlePubMedGoogle Scholar
- McAdams DJ. Acute hospitalist medicine and the rapid response system. Textbook of Rapid Response Systems 2011:47–53.Google Scholar
- Hales BM, Pronovost PJ. The checklist––a tool for error management and performance improvement. J Crit Care. 2006;21:231.View ArticlePubMedGoogle Scholar
- Eagle DM, Coltvet G, Farley D. The Mayo Clinic, multidisciplinary simulation center. J Surg Educ. 2010;67:470–2.View ArticlePubMedGoogle Scholar
- Arriaga AF, Bader AM, Wong JM, et al. Simulation-based trial of surgical-crisis checklists. N Engl J Med. 2013;368:246–53.View ArticlePubMedGoogle Scholar
- Chan PS, Khalid A, Longmore LS, Berg RA, Kosiborod M, Spertus JA. Hospital-wide code rates and mortality before and after implementation of a rapid response team. JAMA. 2008;300:2506–13.View ArticlePubMedGoogle Scholar
- Moretti MA, Cesar LAM, Nusbacher A, Kern KB, Timerman S, Ramires JAF. Advanced cardiac life support training improves long-term survival from in-hospital cardiac arrest. Resuscitation. 2007;72:458–65.View ArticlePubMedGoogle Scholar
- ATLS Subcommittee; American College of Surgeons’ Committee on Trauma; International ATLS working group. Advanced trauma life support (ATLS®): the ninth edition. J Trauma Acute Care Surg. 2013;74(5):1363-6. https://doi.org/10.1097/TA.0b013e31828b82f5. PubMed PMID: 23609291.
- Smith KK, Gilcreast D, Pierce K. Evaluation of staff’s retention of ACLS and BLS skills. Resuscitation. 2008;78:59–65.View ArticlePubMedGoogle Scholar
- Low D, Clark N, Soar J, et al. A randomised control trial to determine if use of the iResus© application on a smart phone improves the performance of an advanced life support provider in a simulated medical emergency. Anaesthesia. 2011;66:255–62.View ArticlePubMedGoogle Scholar
- Cooper S, Cant R, Porter J, et al. Rating medical emergency teamwork performance: development of the TEAM emergency assessment measure (TEAM). Resuscitation. 2010;81:446–52.View ArticlePubMedGoogle Scholar
- Rivers EP, Nguyen HB, Huang DT, Donnino M. Early goal-directed therapy. Crit Care Med. 2004;32:314–5. author reply 315View ArticlePubMedGoogle Scholar
- Cannon CP, Gibson CM, Lambrew CT, et al. Relationship of symptom-onset-to-balloon time and door-to-balloon time with mortality in patients undergoing angioplasty for acute myocardial infarction. JAMA. 2000;283:2941–7.View ArticlePubMedGoogle Scholar
- Havyer RD, Wingo MT, Comfere NI, et al. Teamwork assessment in internal medicine: a systematic review of validity evidence and outcomes. J Gen Intern Med. 2014;29:894–910.View ArticlePubMedGoogle Scholar
- Calhoun AW, Boone MC, Porter MB, Miller KH. Using simulation to address hierarchy-related errors in medical practice. Perm J. 2014;18:14.View ArticlePubMedPubMed CentralGoogle Scholar
- Härgestam M, Lindkvist M, Brulin C, Jacobsson M, Hultin M. Communication in interdisciplinary teams: exploring closed-loop communication during in situ trauma team training. BMJ Open. 2013;3:e003525.View ArticlePubMedPubMed CentralGoogle Scholar
- Association AH. http://cpr.heart.org/AHAECC/CPRAndECC/Training/HealthcareProfessional/AdvancedCardiovascularLifeSupportACLS/UCM_473186_Advanced-Cardiovascular-Life-Support-ACLS.jsp.
- McLaughlin SA, Doezema D, Sklar DP. Human simulation in emergency medicine training: a model curriculum. Acad Emerg Med. 2002;9:1310–8.View ArticlePubMedGoogle Scholar
- Seferian EG, Afessa B. Demographic and clinical variation of adult intensive care unit utilization from a geographically defined population. Crit Care Med. 2006;34:2113–9.View ArticlePubMedGoogle Scholar
- Simpson H, Clancy M, Goldfrad C, Rowan K. Admissions to intensive care units from emergency departments: a descriptive study. Emerg Med J. 2005;22:423–8.View ArticlePubMedPubMed CentralGoogle Scholar
- Staudinger T, Stoiser B, Müllner M, et al. Outcome and prognostic factors in critically ill cancer patients admitted to the intensive care unit. Crit Care Med. 2000;28:1322–8.View ArticlePubMedGoogle Scholar
- Knaus WA, Wagner DP, Zimmerman JE, Draper EA. Variations in mortality and length of stay in intensive care units. Ann Intern Med. 1993;118:753–61.View ArticlePubMedGoogle Scholar