Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Exploring intensive care nurses’ team performance in a simulation-based emergency situation, − expert raters’ assessments versus self-assessments: an explorative study

  • Randi Ballangrud1, 2Email author,
  • Mona Persenius1,
  • Birgitta Hedelin2 and
  • Marie Louise Hall-Lord1, 2
BMC Nursing201413:47

https://doi.org/10.1186/s12912-014-0047-5

Received: 9 May 2013

Accepted: 9 December 2014

Published: 17 December 2014

Abstract

Background

Effective teamwork has proven to be crucial for providing safe care. The performance of emergencies in general and cardiac arrest situations in particular, has been criticized for primarily focusing on the individual’s technical skills and too little on the teams’ performance of non-technical skills. The aim of the study was to explore intensive care nurses’ team performance in a simulation-based emergency situation by using expert raters’ assessments and nurses’ self-assessments in relation to different intensive care specialties.

Methods

The study used an explorative design based on laboratory high-fidelity simulation. Fifty-three registered nurses, who were allocated into 11 teams representing two intensive care specialties, participated in a videotaped simulation-based cardiac arrest setting. The expert raters used the Ottawa Crisis Resource Management Global Rating Scale and the first part of the Mayo High Performance Teamwork Scale to assess the teams’ performance. The registered nurses used the first part of the Mayo High Performance Teamwork Scale for their self-assessments, and the analyses used were Chi-square tests, Mann–Whitney U tests, Spearman’s rho and Intraclass Correlation Coefficient Type III.

Results

The expert raters assessed the teams’ performance as either advanced novice or competent, with significant differences being found between the teams from different specialties. Significant differences were found between the expert raters’ assessments and the registered nurses’ self-assessments.

Conclusions

Teams of registered nurses representing specialties with coronary patients exhibit a higher competence in non-technical skills compared to team performance regarding a simulated cardiac arrest. The use of expert raters’ assessments and registered nurses’ self-assessments are useful in raising awareness of team performance with regard to patient safety.

Keywords

Assessment Emergency Intensive care Non-technical skills Nursing Patient safety Simulation-based training Team performance

Background

Structuring work in health-care teams has been found to largely account for quality and safety improvements in patient care [1],[2], and is shown as being crucial for patient safety in the intensive care unit (ICU). Failures in team processes such as communication, coordination or leadership have frequently been associated with patient safety incidents [1],[3]. Effective teamwork in emergency situations is expected, although teamwork is not often evaluated or discussed on a regular basis in clinical practice [4]. Effective team training is recommended for those who are expected to work in teams [5], and nurses and physicians should conduct both disciplinary and interdisciplinary team training to increase patient safety. Simulation-based training is recommended today as a method to make health-care professionals aware of and understand the importance of teamwork and the aspects of team performance [6],[7]. Moreover, simulation is seen as an effective method for assessing the ongoing competence of critical care nurses [8],[9].

According to Flin et al. [10], teamwork consists of a number of elements such as supporting others, solving conflicts, exchanging information and coordinating activities, with one of the approaches used to minimize the effect of human error being to assist health-care teams in team performance training [11] by increasing individuals’ and teams’ competencies in non-technical skills (NTS) [12],[13]. NTS are described as the “cognitive, social and personal resource skills that complement technical skills and contribute to safe and efficient task performance” [10]. According to Flin et al. [14], NTS generally include situational awareness, decision-making, communication, teamwork, leadership and the management of stress and fatigue. Team performance evaluation can be defined as, “the application of standard diagnostic measurement tools to assess the behaviors, cognitions and attitudes enacted by team members in relation to clearly operationalized criteria” [15]. NTS underpin the domain-specific competencies for the profession and team training programmes designed to increase the use of NTS, often referred to as crisis resource management (CRM) [16]. CRM aims to coordinate, utilize and apply all available resources to help optimize patient safety and outcome, as well as preventing errors and minimizing the negative consequences of errors that have already occurred. In addition to equipment, resources include all people involved with their abilities, attitudes, skills and limitations [17]. The principles for training teams to cope with stressful situations and error management were developed by the airline industry [18] and transferred to health care by Gaba et al. [19], who adapted the use of human patient simulators into the CRM programme. With regard to CRM programmes, NTS are often termed as CRM skills [20],[21].

Reader et al. [22] identified that a large proportion of the contributory factors underlying critical incidents in the ICU could be attributed to poor NTS in terms of communication, leadership, coordination and decision making. Furthermore, data from an ICU incident-reporting system showed that team communication failure led to patient harm [23],[24], while critical incident studies have indicated the importance of team leadership for guiding the way in which ICU team members interact and coordinate [24],[25]. With regard to cardiopulmonary resuscitation (CPR) situations, simulation-based research has demonstrated that an absence of leadership and task distribution was associated with poor interdisciplinary team performance [26]. However, hierarchies have been identified as barriers to a team’s action [27],[28], while interdisciplinary teamwork is seen as one of the key processes in the safe delivery of patient care [5]. Nonetheless, by observing the simulation-based interdisciplinary interaction of nurses in obstetric and neonatal emergencies, Miller et al. [29] found a need for improvement with regard to nurses’ NTS necessary for their contribution to high-reliability teams.

Team performance measurement is important in order to verify team effectiveness and to identify strengths and weaknesses as a basis for further training and continuing education to help achieve a safe and effective patient care [30],[31]. To identify team performance, the evaluation may include both observational expert assessments and team self-assessment [15], though research has documented an incongruity between self-assessment compared with the observed measures of NTS competence [32],[33]. Nurses who work in different intensive care specialties with somewhat varied categories of critically ill patients are involved in emergency situations such as cardiac arrests in both real- and training situations. However, the CPR performing situations have been criticized for mainly focusing on the individual’s technical skills and too little on the team’s NTS performance [26]-[28]. There are few available studies that focus on ICU nurses’ NTS competence in emergencies team performance. Therefore, the aim of this study was to explore intensive care nurses’ team performance in a simulation-based emergency situation by using expert raters’ assessments and nurses’ self-assessments in relation to different intensive care specialties.

Methods

Design

The study used an explorative design based on laboratory high-fidelity simulation.

Setting

The study was conducted in a simulation centre at a university college in Norway, in a laboratory created as an ICU environment and with the use of a high-fidelity human patient simulator.

Participants

A convenience sample of 53 registered nurses (RNs) was recruited from seven ICUs in four hospitals within one hospital trust (Table 1).
Table 1

Characteristics of the sample (11 teams, n = 53 RNs)

Demographic information

Subcategory

Subgroups

  
   

Mean (SD)

n (%)

Specialties (n = 2)

 

G-ICU1 (5 teams)

 

26 (49.1)

  

M-ICU2 (6 teams)

 

27 (50.9)

Age

    
  

G-ICU

46.69 (6.98)

 
  

M-ICU

43.59 (8.04)

 

Gender

Female

   
  

G-ICU

 

24 (92.3)

  

M-ICU

 

24 (88.9)

 

Male

   
  

G-ICU

 

2 (7.7)

  

M-ICU

 

3 (11.1)

Education level

Postgraduate

   
  

G-ICU

 

26 (100.0)

  

M-ICU

 

24 (88.9)

Years as PG3- RN (n = 50)

    
  

G-ICU

14.38 (8.90)

 
  

M-ICU

7.28 (8.62)

 

Years at unit

    
  

G-ICU

14.73 (8.90

 
  

M-ICU

8.52 (6.86)

 

1G-ICU = General intensive care unit, 2M-ICU = Medical intensive care unit, 3PG = Postgraduate.

The RNs worked in two specialties: (i) general intensive care units (G-ICU) (n = 26 RNs) with surgical and medical patients, usually without coronary patients, and (ii) medical intensive care units (M-ICU) (n = 27 RNs) with coronary, medical and in one unit also surgical patients. Participation in the study occurred during the RNs’ scheduled work time and the RNs who wanted to participate signed up on a list. Each team consisted of four to six RNs from same units who knew each other and had worked together. The unit nurse/manager allocated the RNs into teams with regard to their work schedules and the staffing resources to ensure safe care at the unit. A total of 11 teams from the G-ICU (five teams) and M-ICU (six teams) participated.

Simulation-based team training programme and scenario

The cardiac arrest scenario chosen for this study, was a part of a simulation-based team training programme which comprised two half days for each team, and was developed by the research group.

On the first half day of the programme, the RNs received information about the topic of the simulation scenario cases, the CRM learning objectives and recommendations about relevant literature to read. Furthermore, the RNs participated in theory inputs concerning patient safety in the ICU, NTS with a focus on the key CRM points by Rall and Gaba [34] and the use of simulation for training.

The other half day of the programme that referred to conducting two simulation scenarios, was carried out for one team at a time two to four weeks after the first day. In addition to the scenario which involved a patient undergoing cardiac arrest (reported in this study), a scenario of a receipt of a trauma patient was conducted. The two simulation scenarios were carried out in varying order for each team within each specialty. Two trained simulator instructors were responsible for the implementation of the scenario, one for facilitating the simulation and the other for performing the operation of the mannequin software. Three different scenario roles were randomly assigned among the team members by the facilitator: one as a patient charge nurse (leader), one as the charge nurse’s assistant (assistant) and the third with the responsibility for the patient in the neighboring bed (helper). The RNs also had the opportunity to call a physician (acted out by the facilitator), who gave the RNs medical directives for the handling of the emergency over the phone. The rest of the team members were placed in another room and observed the team performance on a screen, with the videotaped scenario lasting approximately 12–15 minutes.

Cardiac Arrest simulation scenario

A patient (simulator), 55-year-old man arrived at the ICU three days ago with a diagnosis of cardiac arrest. He has been through an effective implementation of 24 hours of therapeutic hypothermia treatment and has been extubated for some few hours. He is calling for help and the RNs that had just arrived on duty enter the patient room. The patient is suffering from chest pain and is restless and anxious. A while after the RNs entered the patient’s room, and the patient suffered cardiac arrest due to ventricular tachycardia displayed on the monitor. With the onset of cardiac arrest, the patient closed his eyes, ceased to speak and to breathe, and his pulse was no longer palpable. Cardiopulmonary resuscitation was expected to be initiated by the RNs. Regardless of the measures taken, the patient stayed in cardiac arrest for many minutes. Thereafter, ROSC could be achieved by defibrillation.

Team training objectives

  • Practicing an organized and effective problem solving

  • Adapts leadership skills based on the team composition and situation

  • Communicates clearly and concisely

  • Utilize available resources

  • Constantly reassesses and reevaluates situation

Data collection

Procedure

The data collection took place from April 2009 to May 2010, and immediately after completing the scenario all the RNs in the team conducted a team performance self–assessment using Mayo High Performance Teamwork Scale (MHPTS) (items 1–8) [20]. After all teams had completed the simulation-based team training programme, two trained expert raters, who had no contact with each other, individually viewed each videotaped scenario for all 11 teams and assessed their team performance by using Ottawa Crisis Resource Management Global Rating Scale (Ottawa GRS) [21] and MHPTS (items 1–8). The raters were postgraduate (PG) ICU RNs chosen for their expertise as CRM simulation instructors. One-day rater training organized by the researcher and the simulation instructors was conducted in order to standardize the evaluation of team performance and to become familiar with the instruments. Video files from a pilot-testing of the scenarios, with teams not involved in the study performing the scenario, were used in the training of the raters. After watching the video, each rater individually rated the video files of different teams by using the Ottawa GRS and MHPTS, and then together they reconsidered their rating in order to develop a common understanding of the team’s performance.

Instruments

MHPTS [20] provides a brief, reliable and practical measure of NTS that can be used by participants as a self-assessment of CRM training, in which they reflect on and evaluate their team performance. The scale is also recommended for comparisons between self-assessment and expert assessment [20]. Each item is eligible for a 0-, 1- or 2 point assessment: 0 = never or rarely, 1 = inconsistently and 2 = consistently. The first part of MHPTS (items 1–8) was used because the items were judged as being consistent with the simulation scenario.

MHPTS assessment criteria

  1. 1.

    A leader is clearly recognized by all members.

     
  2. 2.

    The team leader assures maintenance of an appropriate balance between command authority and team member participation.

     
  3. 3.

    Each team member demonstrates a clear understanding of his or her role.

     
  4. 4.

    The team prompts each other to attend all significant clinical indicators throughout the procedure/intervention.

     
  5. 5.

    When team members are actively involved with the patient, they verbalize their activities aloud.

     
  6. 6.

    Team members repeat back or paraphrase instructions and clarifications to indicate that they heard them correctly.

     
  7. 7.

    Team members refer to established protocols and checklists for the procedure/intervention.

     
  8. 8.

    All members of the team are appropriately involved and participate in the activity.

     

Ottawa GRS [21] provides an observational expert rating scale which consists of six categories: one overall CRM performance category (Overall-CRM) and five subsets of CRM skills (NTS) categories, including assessment criteria regarding leadership, problem solving, situational awareness, resource utilization and communication. All categories used a seven-point adjective scale, using a rating from 1-2 = novice (all CRM skills requiring a significant improvement), 3-4 = advanced novice (many CRM skills requiring a moderate improvement), 5-6 = competent (most CRM skills requiring minor improvement) and 7 = clearly superior (few, if any, CRM skills which only require a minor improvement).

Ottawa CRM skills categories with assessment criteria

Leader ship:

Stay calm and in control during crisis

Prompt and firm decision-making

Maintains global perspective (“Big picture”)

Problem solving:

Organized and efficient problem solving approach (ABC’s)

Quick in implementation (Concurrent management)

Considers alternatives during crisis

Situation awareness:

Avoid fixation error

Reassess and re-evaluates situation constantly.

Anticipates likely events

Resource utilization:

Calls for help appropriately

Utilizes resources at hand appropriately

Prioritizes tasks appropriately

Communication:

Communicate clearly and concisely

Uses directed verbal/non-verbal communication

Listen to team input

In addition, seven background questions on age, gender, education level, type of unit, years as RN and postgraduate RN and years on present unit were included.

The MHPTS and Ottawa GRS were translated into Norwegian using a “back translation” [35], and the instruments were tested for both face- and content validity by an expert group of simulation instructors, with only minor linguistic adjustments being required.

Ethical considerations

This study was approved by the Norwegian Social Data Services as well as being approved by the hospital’s administrative heads, and the study was conducted according to the Ethical Guidelines for Nursing Research in the Nordic Countries [36]. Information, and an invitation to participate in the study were given to the RNs in both written and oral form, and referred to the principle of autonomy addressed by voluntariness, informed consent and the right to withdraw from the research project at any time, as well as confidentiality in terms of making data anonymous and assurance that the videotapes would been deleted at the end of the project. The RNs written consent was obtained.

Data analyses

IBM SPSS Statistics 19 was used to analyse the data, with descriptive statistics presenting the frequencies, percentages, means and standard deviations. The units of analyses of team performance were the team score for the expert raters’ assessments and the RNs’ self-assessments, respectively. Chi-square tests were used to compare the two specialties with regard to the RNs’ background in general and with regard to the three different scenario roles. The expert raters’ assessments were calculated by adding each rater’s score for each team, and then dividing by two. Mann–Whitney U tests were conducted to compare the differences between the teams from two types of specialties (G-ICU, M-ICU) with regard to the Ottawa GRS CRM categories and the MHPTS items. The RNs’ self-assessments were calculated by adding each team member’s score for the team, and then dividing the score by the number of team members. Mann–Whitney U tests were conducted to compare the differences between the teams from two types of specialties (G-ICU, M-ICU) concerning the MHPTS items. Additionally, Mann–Whitney U tests were used to compare the differences between the expert raters’ assessments and the RNs’ self-assessments of the MHPTS items. Correlational analyses with Spearman’s rho were used to explore the relationship between the expert raters’ assessments and the RNs’ self-assessments of the MHPTS items. A two-tailed significance level of p < .05 was used for all tests.

Intraclass Correlation Coefficient (ICC) Type III was used to assess inter-rater reliability regarding Ottawa GRS CRM categories. ICC coefficients > .75 indicate a good reliability, with coefficients from .50 -.75 suggesting moderate reliability and values < .50 representing poor reliability [37]. This study showed an ICC ranging from .667-.854 (moderate - good reliability) (Table 2).
Table 2

Ottawa GRS interrater reliability intraclass correlation (ICC)

Ottawa GRS categories

Type III ICC 1

Overall

.729

Leadership

.854

Problem solving

.742

Situation awareness

.667

Resource utilization

.735

Communication

.788

1Two-way mixed effects model.

Percentage agreements were conducted in relation to MHPTS, and in this study the percentage of agreements between raters was 60%.

Results

Most of the 53 participants were women (91%) and were post-graduates (94%). Comparisons between the two specialties’ background showed significant differences with regard to “years as PG-RNs” (χ2 (2) =12.53, p = .002), and “years at unit” (χ2 (2) =7.32, p = .026), with the RNs from the G-ICU with the longest experience as PG-RNs and most years at unit. In relation to scenario roles, significant differences were found with regard to the helper role in “age” (χ2 (2) = 7.71, p = .013) and in “years as PG-RNs” (χ2 (2) =7.81, p = .008), with helpers in teams representing G-ICU with a higher age and longer experience as PG-RNs.

The expert raters’ assessments of the 11 teams’ performance in relation to the different specialties of unit based on Ottawa GRS are shown in Table 3.
Table 3

Expert raters’ assessments in relation to types of specialties

Ottawa GRS CRM categories

G-ICU 1(n = 5)

M-ICU 2(n = 6)

Mann–Whitney U tests

 

mean (SD)

mean (SD)

Z

p

Overall performance

3.70 (1.04)

5.92 (0.58)

−2.666

.008

Leadership skills

3.80 (0.67)

5.83 (0.52)

−2.777

.005

Problem solving skills

3.30 (0.67)

5.83 (0.61)

−2.764

.006

Situational awareness skills

3.70 (0.84)

5.83 (0.61)

−2.764

.006

Resource utilization skills

3.60 (0.82)

5.83 (0.26)

−2.844

.004

Communication skills

3.50 (0.71)

5.58 (0.38)

−2.796

.005

Rating: 1-2 = novice, 3-4 = advanced novice, 5-6 = competent and 7 = clearly superior.

1G-ICU = General intensive care unit. 2M-ICU = Medical intensive care unit.

The teams’ effectiveness ranged from “advanced novice” to “competent”, and there were significant differences between teams from different specialties in all six of the Ottawa GRS categories. Moreover, the M-ICU teams achieved higher scores than the G-ICU teams.

The expert raters’ assessments and the RNs’ self-assessments of team performance in relation to the different specialties, as well as the comparisons between expert raters’ assessments and RNs’ self-assessments based on MHPTS, are shown in Table 4.
Table 4

Expert raters’ assessments and RNs’ self-assessments in relation to types of specialties

  

Expert raters’ assessments

RNs’ self-assessments

Comparisons of expert raters’ and RNs’ self-assessments

MHPTS Item

Specialties

 

Mann–Whitney U tests

 

Mann–Whitney U tests

Mann–Whitney U tests

   

Mean (SD)

Z

p

Mean (SD)

Z

p

Z

p

1

A leader is clearly recognized by all members

G-ICU1

0.90 (0.55)

−2.191

.028

1.45 (0.19)

−1.287

.198

−1.954

.051

M-ICU2

1.67 (0.41)

1.22 (0.31)

−2.021

.044

2

The team leader assures maintenance of an appropriate balance between command authority and team member participation

G-ICU

0.80 (0.57)

−2.175

.030

1.45 (0.19)

-.463

.644

−1.972

.049

M-ICU

1.58 (0.38)

1.38 (0.22)

−1.366

.172

3

Each team member demonstrates a clear understanding of his or her role

G-ICU

1.00 (0.35)

−2.063

.039

1.46 (0.89)

-.835

.404

−2.010

.044

M-ICU

1.50 (0.32)

1.49 (0.39)

-.337

.736

4

The team prompts each other to attend all significant clinical indicators throughout the procedure/intervention

G-ICU

0.70 (0.45)

−2.837

.005

1.26 (0.30)

−1.851

.064

−1.954

.051

M-ICU

1.83 (0.26)

1.63 (0.18)

−2.037

.042

5

When team members are actively involved with the patient, they verbalize their activities aloud

G-ICU

0.80 (0.27)

−2.844

.004

1.33 (0.21)

−1.038

.299

−1.496

.135

M-ICU

1.83 (0.26)

1.42 (0.20)

−2.024

.043

6

Team members repeat back or paraphrase instructions and clarifications to indicate that they heard them correctly

G-ICU

0.50 (0.35)

−2.837

.005

1.07 (0.29)

−1.116

.264

−1.775

.076

M-ICU

1.83 (0.26)

1.33 (0.31)

−1.677

.094

7

Team members refer to established protocols and checklists for the procedure/intervention

G-ICU

0.40 (0.22)

−2.837

.005

0.74 (0.19)

−2.114

.035

-.804

.421

M-ICU

1.58 (0.49)

1.27( 0.40)

−1.181

.238

8

All members of the team are appropriately involved and participate in the activity

G-ICU

1.20 (0.27)

−2.515

.017

1.53 (0.29)

−1.864

.062

−1.206

.381

M-ICU

1.83 (0.26)

1.84 (0.12)

-.692

.489

Rating: 0 = never or rarely, 1 = inconsistently, 2 = consistently. 1G-ICU = General intensive care unit, 2M-ICU = Medical intensive care unit.

Bold numbers reflect significance level of p < .05.

The expert raters’ assessments demonstrated significant differences between the ICU teams for all eight MHPTS items, with the G-ICU teams having lower scores than the teams from the M-ICU. One significant difference was found between different intensive care specialties with regard to the RNs’ self-assessments. RNs’ scores for each team are shown in Additional file 1. Additionally, there were significant differences between the expert raters’ assessments and the RNs’ self-assessments on five items, with the teams from G-ICU having higher scores than the expert raters on two items and the teams from M-ICU having lower score than expert raters on three items. Correlations between expert raters’ assessments and the RNs’ self-assessments of the eight items demonstrated for G-ICU a variation from r = .00 to .92 and for M-ICU from r = −.11 to .50. Of a total of 16 analyses 13 showed positive correlations, with one significant correlation for G-ICU with regard to item 1 (r = .92, p = .028). No correlation was found on item 5 for both G-ICU and M-ICU, and the negative correlation was found on item 8 for G-ICU.

Discussion

The main result of the study showed that the expert raters’ assessment exhibited a variation in the intensive care nurses’ team performance, as the teams from the M-ICU were given higher scores by the expert raters in regard to both the Ottawa GRS six categories and the MHPTS items. According to the Ottawa GRS criteria, the M-ICU was assessed as competent, which means that most CRM skills required some minor improvements. However, the G-ICU teams were assessed as being “advanced novice”, meaning that many CRM skills required some moderate improvement. Even so, none of the teams were assessed as being clearly superior. However, significant differences were found between the expert raters and the RNs’ self-assessment of their team’s performance according to MHPTS. To help ensure quality and patient safety, all RNs working in ICUs should possess the knowledge, skills and attitudes necessary to be an effective team member in an emergency situation, and the quality of care is no better than the quality created on the sharp end of care [38]. Team training has been found to lead to safety culture changes [39] and the M-ICUs with coronary patients have a long tradition of training CPR with a focus on technical skills [40], which may also have an influence on the team performance concerning NTS.

RNs team performance

Team leadership refers to how to influence team performance by facilitating team problem solving through cognitive processes, coordination processes and the team’s collective motivation and behaviour [41]. In emergencies generally, and in cardiac arrest situations in particular, the behaviour of the first responders is crucial, and the first-responding RN should take the leadership role [42]. RNs from the M-ICU representing units with coronary patients may have experienced cardiac arrest situations in real- and training situations to a greater extent, and were therefore more aware of the leader’s role. Conversely, the RNs from G-ICU had the longest experience as PG-RNs and most years at unit. This may indicate the importance of regular training. A lack of knowledge of procedure or guidelines may result in low leadership performance [43], and research has shown that well-trained, first-reacting nurses have successfully taken leadership roles in CPR [44],[45]. It is important that RNs stay in the leadership role until others with more leadership experience appear. Team effectiveness regarding CPR is found to be most optimal with a leader who demonstrates immediate directive leadership behaviour [46],[47]. Teamwork may be affected by the institutional culture, and RNs from G-ICU may be less comfortable being a leader in a cardiac arrest situation. Still, Andersen et al. [27] found that the RNs’ cultural and professional role inhibited them from participating in decision making in the CPR despite being certified as Advanced Life Support instructors, even though nurses are responsible for decision making in emergencies [48]. According to Künzle et al. [43], an inadequate leadership behaviour in critical care teams could be due to a lack of leadership training; it is therefore important that all RNs in ICUs receive this training to help improve patient safety. RNs often embrace CRM training because it results in a greater respect and an improved quality of work life by flattening hierarchies of authority through team members’ shared responsibility for patient safety [49].

The teams from the M-ICU with the most experience with coronary patients achieved higher scores compared to the G-ICU with regard to problem solving in both the experts’-and the RNs’ self-assessments. However, research demonstrates that a failure to transform theoretical knowledge into effective team activity during CPR appears to be a major problem [26],[50], which is a result that highlights the importance of training team performance. Thomas et al. [51] found that nurses who received team training as part of their neonatal resuscitation training were generally shown to display more teamwork-related behaviours than those who did not receive such training. Skills identified as being the most improved during patient emergencies after followed up simulation-based team training were found to be the ability to respond in a systematic way, handover to the emergency team and airway management [52].

All teams demonstrated positive results regarding the MHPTS’s resource utilization skills as assessed by both the expert raters and the RNs. The RNs from the G-ICU assessed themselves significantly higher than the experts in the items “each team demonstrated a clear understanding of his or her role”. One reason for this more positive assessment may be that the RNs in the scenario were assigned different roles by the facilitator, which helped them to clarify their function in the team. However, patient safety climate studies confirm that RNs in ICUs have positive perceptions of the teamwork within the unit which includes, e.g. supporting others and coordinating activities [53]-[55], which may also influence the team’s resource utilization skills.

A similar picture emerged with the M-ICU teams, with higher scores than the G-ICU teams with regard to communication skills, which may be a result of RN teams from the M-ICU being more familiar with the CPR guidelines. This resonates with “shared mental model” literature, in which teams communicate and coordinate more effectively when members form a shared mental model for goals, tasks and team member roles and responsibilities [56]. As one of the central components of health-care teamwork, communication has been identified for several years as a central root cause of sentinel events [57]. Tools such as SBAR (situation, background, assessment, recommendation), which provide a common and predictable structure to the communication, as well as a “critical language” that gets everyone to stop and listen, have been found to be effective and essential for safe care in complex environments [58]. Furthermore, “closed-loop communication”, with the verbal confirmation of orders and messages for ensuring that sent communications are heard and accurately understood, is recommended [56]. The implementation of standardized communication tools and techniques may improve team performance in an emergency, thereby exerting a positive impact on patient safety, which should be recommended to be adopted for team training.

Expert rater assessment vs. self-assessment

The present study included both self-assessments of team performance and video-based analyses by expert raters to study RNs’ team performance. Together with direct observation, these are in accordance with the most up-to-date measures of team performance in health care [31]. Video-based analyses offer advantages since they allow for a precise analysis and repeated review by multiple experts [31], and the observational measures may capture the behaviour that teams actually engage in, although observers cannot assess all of the implicit components of teamwork [15]. Situational awareness skills represent implicit components of teamwork, and self-assessments are of great importance. The RNs from G-ICU assessed themselves as generally higher than the expert raters, and significant differences were shown on the two items referring to leadership and resource utilization. On the other hand, the RNs from M-ICU assessed themselves generally lower than the experts, with significant differences on the three items referring to leadership, situational awareness and communication skills. An incongruity between RNs’ self-assessment and the experts’ assessment may indicate that RNs both overestimated and underestimated their NTS competence. Although the correlations between self-assessment and the experts’ assessment had positive directions in 13 of 16 analyses, only one significant relationship was found. Moreover, no correlations (item 5) and a negative correlation (item 8) may also show an incongruity assessment among the RNs. Arora et al. [33] found a strong correlation between surgeons’ self-assessments and expert assessments with regard to technical skills, though no significant correlations were found with regard to NTS. Davis et al.’s [32] systematic review of the accuracy of physicians’ self-assessments compared with observed expert measures of competence documented that in a majority of the relevant studies, physicians did not appear to accurately self-assess. Our results indicate a tendency of poor performers to overestimate their performance while high performers underestimate theirs. Kruger and Dunning [59] studied the calibration of subjects carrying out various tasks such as logical reasoning, grammar and humour. They demonstrated that participants who were in the lowest-scoring quartile greatly overestimated their abilities; those in the middle quartiles were generally accurate in their self-assessment; whereas participants who had the highest scores underestimated their abilities. These tendencies were confirmed by Hodges et al. [60] studying family medicine residents who interviewed a standardized patient in a difficult “telling bad news” scenario.

While self-report measures have their limitations, they offer a means for assessing unobservable components of team performance, which are no less important than observable components pertaining to training [15]. Different approaches in evaluating team performance have both their strengths and weaknesses [15],[31], and a combined approach can help to obtain a more complete picture of the complexity of team performance, while also providing a method to guide learning through systematic feedback [15].

Limitations

A limitation of this study was the small number of ICU teams; therefore, the generalizability of the results must be interpreted with caution. Additionally, the result of the RNs’ team performance may be influenced by participants who were initially positive to the use of simulation-based training, and on this basis agreed to participate. Even though simulation studies have many opportunities and advantages, there are also limitations, and although behaviour patterns in simulation are similar to those in real-life situations, the transfer of knowledge from the simulator to real-patient situations has not been clearly confirmed [61]. Although rater training was conducted in order to standardize the evaluation of teamwork performance, some moderate ICC scores were demonstrated regarding the Ottawa GRS. Moreover, a 60% agreement was found for the MHPTS rating of the teams’ performance. The complexities of team performance pose a challenge, and measurement and evaluation have weaknesses due to responsible biases of individual raters [15],[30], which may influence the achievement of a high reliability.

Conclusion

Simulation-based team performance measurement among ICU nurses revealed a variation in team performance in a simulated cardiac arrest emergency situation. RNs’ team performance required minor to moderate improvements, and the RNs with the most experience from coronary patients showed the best team performance. The use of expert raters’ assessments and RNs’ self-assessment is useful in raising awareness of team performance with regard to patient safety.

One important implication for practice is that the results contribute to a focus on RNs’ knowledge, skills and attitudes regarding team performance in order to ensure quality and patient safety in emergency patient situations in the ICU. Further research in terms of the RNs’ contribution of NTS is needed, while it will also be of great interest to investigate the ICU RNs’ perceptions on the use of simulations for team training in promoting patient safety in the ICU.

Authors’ information

RB, PhD, RNT, RNCCN, Department of Health Sciences, Faculty of Health, Science and Technology, Karlstad University, Sweden and Department of Nursing, Faculty of Health, Care and Nursing, Gjøvik University College, Norway.

MP, Senior Lecturer, PhD, RNANIC, Department of Health Sciences, Faculty of Health, Science and Technology, Karlstad University, Sweden.

BH, Professor, PhD, RNT, Department of Nursing, Faculty of Health, Care and Nursing, Gjøvik University College, Norway.

MLHL, Professor, PhD, RNT, Department of Health Sciences, Faculty of Health, Science and Technology, Karlstad University, Sweden and Professor, Department of Nursing, Faculty of Health, Care and Nursing, Gjøvik University College, Norway.

Additional file

Abbreviations

CPR: 

Cardiopulmonary resuscitation

CRM: 

Crisis recourse management

G-ICU: 

General intensive care units

ICU: 

Intensive care unit

ICC: 

Intraclass correlation coefficient

M-ICU: 

Medical intensive care units

MHPTS: 

Mayo high performance teamwork scale

NTS: 

Non-technical skills

Ottawa GRS: 

Ottawa crisis resource management global rating scale

RN: 

Registered nurse

SBAR: 

Situation, background, assessment, recommendation

Declarations

Acknowledgements

The authors would like to thank the registered nurses for their participation. In addition, we want to express our deepest gratitude to the simulation instructors, Viktor Haugom, Tore Karlsen, Christer Eriksson and Terje Ødegården, moreover Jari Appelgren for statistical support. This study was financially supported by the Laerdal Foundation for Acute Medicine.

Authors’ Affiliations

(1)
Department of Health Sciences, Faculty of Health, Science and Technology, Karlstad University
(2)
Department of Nursing, Faculty of Health, Care and Nursing, Gjøvik University College

References

  1. Manser T: Teamwork and patient safety in dynamic domains of healthcare: a review of the literature. Acta Anaesthesiol Scand. 2009, 53 (2): 143-151. 10.1111/j.1399-6576.2008.01717.x.View ArticlePubMedGoogle Scholar
  2. Buljac-Samardzic M, Dekker-van Doorn CM, Van Wijngaarden JDH, Van Wijk KP: Interventions to improve team effectiveness: a systematic review. Health Policy. 2010, 94 (3): 183-195. 10.1016/j.healthpol.2009.09.015.View ArticlePubMedGoogle Scholar
  3. Manojlovich M, DeCicco B: Healthy work environments, nurse-physician communication, and patients’ outcomes. Am J Crit Care. 2007, 16 (6): 536-543.PubMedGoogle Scholar
  4. Raemer DB: Team-Oriented Medical Simulation. Simulators in Critical Care and Beyond. Edited by: Dunn WF. 2004, Society of Critical Care Medicine, Rochester, Minnesota, USA, 42-46.Google Scholar
  5. Kohn LT, Corrigan JM, Donaldson MS: To Err is Human: Building a Safer Health System. 2000, National Academies press, Washington DCGoogle Scholar
  6. Gaba DM: The future vision of simulation in health care. Qual Saf Health Care. 2004, 13: i2-i10. 10.1136/qshc.2004.009878.View ArticlePubMedPubMed CentralGoogle Scholar
  7. IOM: Crossing the Quality Chasm: a New Health System for the 21st Century. Institute of Medicine. 2001, National Academy Press, Washington, D.CGoogle Scholar
  8. Cato D, Murray M: Use of simulation training in the intensive care unit. Crit Care Nurs Q. 2010, 33 (1): 44-51. 10.1097/CNQ.0b013e3181c8dfd4.View ArticlePubMedGoogle Scholar
  9. Nagle BM, McHale JM, Alexander GA, French BM: Incorporating scenario-based simulation into a hospital nursing education program. J Contin Educ Nurs. 2009, 40 (1): 18-25. 10.3928/00220124-20090101-02.View ArticlePubMedGoogle Scholar
  10. Flin R, O’Connor P, Crichton M: Safety at the Sharp End: A Guide to Non-Technical Skills. 2008, Ashgate Publishing Company, Hamphshire, EnglandGoogle Scholar
  11. Salas E, DiazGranados D, Weaver SJ, King H: Does team training work? principles for health care. Acad Emerg Med. 2008, 15 (11): 1002-1009. 10.1111/j.1553-2712.2008.00254.x.View ArticlePubMedGoogle Scholar
  12. Flin R, Maran N: Identifying and training non-technical skills for teams in acute medicine. Qual Saf Health Care. 2004, 13 (suppl 1): i80-i84. 10.1136/qshc.2004.009993.View ArticlePubMedPubMed CentralGoogle Scholar
  13. Gaba DM, Howard SK, Fish KJ, Smith BE, Sowb YA: Simulation-based training in Anesthesia Crisis Resource Management (ACRM): a decade of experience. Simulat Gaming. 2001, 32 (2): 175-193. 10.1177/104687810103200206.View ArticleGoogle Scholar
  14. Flin R, Patey R, Glavin R, Maran N: Anaesthetists’ non-technical skills. Br J Anaesth. 2010, 105 (1): 38-44. 10.1093/bja/aeq134.View ArticlePubMedGoogle Scholar
  15. Rosen MA, Weaver SJ, Lazzara EH, Salas E, Wu T, Silvestri S, Schiebel N, Almeida S, King HB: Tools for evaluating team performance in simulation-based training. J Emerg Trauma Shock. 2010, 3 (4): 353-359. 10.4103/0974-2700.70746.View ArticlePubMedPubMed CentralGoogle Scholar
  16. Glavin RJ, Maran NJ: Intergration human factors into the medical curricum. Med Educ. 2003, 37 (s1): 59-64. 10.1046/j.1365-2923.37.s1.5.x.View ArticlePubMedGoogle Scholar
  17. Rall M, Dieckmann P: Crisis resoruce management to improve patient safety. Euroanesthesia 2005. Retrieved 20.04.2013, from http://goo.gl/ZH6qcg,
  18. Helmreich RL: On error management: lessons from aviation. Br Med J. 2000, 320 (7237): 781-785. 10.1136/bmj.320.7237.781.View ArticleGoogle Scholar
  19. Gaba DM, Fish KJ, Howard SK: Crisis Management in Anesthesiology. 1994, Churchill Livingstone, New YorkGoogle Scholar
  20. Malec JF, Torsher LC, Dunn WF, Wiegmann DA, Arnold JJ, Brown DA, Phatak V: The mayo high performance teamwork scale: reliability and validity for evaluating key crew resource management skills. Simul Healthc. 2007, 2 (1): 4-10. 10.1097/SIH.0b013e31802b68ee.View ArticlePubMedGoogle Scholar
  21. Kim J, Neilipovitz D, Cardinal P, Chiu M, Clinch J: A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: the University of Ottawa critical care medicine, high-fidelity simulation, and crisis resource management I study. Crit Care Med. 2006, 34 (8): 2167-2174. 10.1097/01.CCM.0000229877.45125.CC.View ArticlePubMedGoogle Scholar
  22. Reader T, Flin R, Lauche K, Cuthbertson BH: Non-technical skills in the intensive care unit. Br J Anaesth. 2006, 96 (5): 551-559. 10.1093/bja/ael067.View ArticlePubMedGoogle Scholar
  23. Pronovost P, Wu AW, Dorman T, Morlock L: Building safety into ICU care. J Crit Care. 2002, 17 (2): 78-85. 10.1053/jcrc.2002.34363.View ArticlePubMedGoogle Scholar
  24. Pronovost PJ, Thompson DA, Holzmueller CG, Lubomski LH, Dorman T, Dickman F, Fahey M, Steinwachs DM: Toward learning from patient safety reporting systems. J Crit Care. 2006, 21 (4): 305-315. 10.1016/j.jcrc.2006.07.001.View ArticlePubMedGoogle Scholar
  25. Reader TW, Flin R, Mearns K, Cuthbertson BH: Developing a team performance framework for the intensive care unit. Crit Care Med. 2009, 37 (5): 1787-1793. 10.1097/CCM.0b013e31819f0451.View ArticlePubMedGoogle Scholar
  26. Marsch SCU, Müller C, Marquardt K, Conrad G, Tschan F, Hunziker PR: Human factors affect the quality of cardiopulmonary resuscitation in simulated cardiac arrests. Resuscitation. 2004, 60 (1): 51-56. 10.1016/j.resuscitation.2003.08.004.View ArticlePubMedGoogle Scholar
  27. Andersen PO, Jensen MK, Lippert A, Østergaard D: Identifying non-technical skills and barriers for improvement of teamwork in cardiac arrest teams. Resuscitation. 2010, 81 (6): 695-702. 10.1016/j.resuscitation.2010.01.024.View ArticlePubMedGoogle Scholar
  28. Hunziker S, Johansson AC, Tschan F, Semmer NK, Rock L, Howell MD, Marsch S: Teamwork and leadership in cardiopulmonary resuscitation. J Am Coll Cardiol. 2011, 57 (24): 2381-2388. 10.1016/j.jacc.2011.03.017.View ArticlePubMedGoogle Scholar
  29. Miller K, Riley W, Davis S: Identifying key nursing and team behaviours to achieve high reliability. J Nurs Manag. 2009, 17 (2): 247-255. 10.1111/j.1365-2834.2009.00978.x.View ArticlePubMedGoogle Scholar
  30. Rosen MA, Salas E, Wilson KA, King HB, Salisbury M, Augenstein JS, Robinson DW, Birnbach DJ: Measuring team performance in simulation-based training: adopting best practices for healthcare. Simul Healthc. 2008, 3 (1): 33-41. 10.1097/SIH.0b013e3181626276.View ArticlePubMedGoogle Scholar
  31. Jeffcott SA, Mackenzie CF: Measuring team performance in healthcare: review of research and implications for patient safety. J Crit Care. 2008, 23 (2): 188-196. 10.1016/j.jcrc.2007.12.005.View ArticlePubMedGoogle Scholar
  32. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L: Accuracy of physician self-assessment compared with observed measures of competence. JAMA. 2006, 296 (9): 1094-1102. 10.1001/jama.296.9.1094.View ArticlePubMedGoogle Scholar
  33. Arora S, Miskovic D, Hull L, Moorthy K, Aggarwal R, Johannsson H, Gautama S, Kneebone R, Sevdalis N: Self vs expert assessment of technical and non-technical skills in high fidelity simulation. Am J Surg. 2011, 202 (4): 500-506. 10.1016/j.amjsurg.2011.01.024.View ArticlePubMedGoogle Scholar
  34. Rall M, Gaba D: Human performance and patient safety. Millers’s Anesthesia. Edited by: Miller RG. 2005, Elsevier Churchill Livingstone, Philadephia, 3021-3072. 6Google Scholar
  35. Brislin RW: Back-translation for cross-cultural research. J Cross Cult Psychol. 1970, 1 (3): 185-216. 10.1177/135910457000100301.View ArticleGoogle Scholar
  36. Northern Nurses Federation: Ethical guidelines for nursing research in the Nordic countries. Retrieved 24.04.2013 from http://www.sykepleien.no/ikbViewer/Content/337889/SSNs%20etiske%20retningslinjer.pdf,
  37. Portney LG, Watkins MP: Foundations of Clinical Research: Applications to Practice. 2009, Pearson Prentice Hall, Upper Saddle RiverGoogle Scholar
  38. Nelson EC, Batalden PB, Godfrey MM: Quality by Design: a Clinical Microsystems Approach. 2007, Jossey-Bass, San FranciscoGoogle Scholar
  39. Jones KJ, Skinner AM, High R, Reiter-Palmon R: A theory-driven, longitudinal evaluation of the impact of team training on safety culture in 24 hospitals. BMJ Qual Saf. 2013, 22 (5): 394-404. 10.1136/bmjqs-2012-000939.View ArticlePubMedGoogle Scholar
  40. Quinn T, Thompson D: History and development of coronary care. Intensive Crit Care Nurs. 1999, 15 (3): 131-141. 10.1016/S0964-3397(99)80043-3.View ArticlePubMedGoogle Scholar
  41. Salas E, Burke CS, Stagel KC: Developing teams and team leaders: Strategies and principles. Leader Development for Transforming Organizations: Growing Leaders for Tomorrow. Edited by: Day DV, Zaccaro SJ, Halpin SM. 2004, Lawrence Erlbaum, Mahwah, N.J, 325-358.Google Scholar
  42. Hunziker S, Tschan F, Semmer N, Howell M, Marsch S: Human factors in resuscitation: lessons learned from simulator studies. J Emerg Trauma Shock. 2010, 3 (4): 389-394. 10.4103/0974-2700.70764.View ArticlePubMedPubMed CentralGoogle Scholar
  43. Künzle B, Kolbe M, Grote G: Ensuring patient safety through effective leadership behaviour: a literature review. Safety Sci. 2010, 48 (1): 1-17. 10.1016/j.ssci.2009.06.004.View ArticleGoogle Scholar
  44. Gilligan P, Bhatarcharjee C, Knight G, Smith M, Hegarty D, Shenton A, Todd F, Bradley P: To lead or not to lead? Prospective controlled study of emergency nurses’ provision of advanced life support team leadership. Emerg Med J. 2005, 22 (9): 628-632. 10.1136/emj.2004.015321.View ArticlePubMedPubMed CentralGoogle Scholar
  45. Marsch SCU, Tschan F, Semmer N, Spychiger M, Breuer M, Hunziker PR: Performance of first responders in simulated cardiac arrests. Crit Care Med. 2005, 33 (5): 963-967. 10.1097/01.CCM.0000157750.43459.07.View ArticlePubMedGoogle Scholar
  46. Tschan F, Semmer NK, Gautschi D, Hunziker P, Spychiger M, Marsch SU: Leading to recovery: group performance and coordinative activities in medical emergency driven groups. Hum Perform. 2006, 19 (3): 277-304. 10.1207/s15327043hup1903_5.View ArticleGoogle Scholar
  47. Hunziker S, Tschan F, Semmer N, Zobrist R, Spychiger M, Breuer M, Hunziker P, Marsch S: Hands-on time during cardiopulmonary resuscitation is affected by the process of teambuilding: a prospective randomised simulator-based trial. Emerg Med J. 2009, 9 (1): 3-10.1111/j.1442-2026.1997.tb00547.x.View ArticleGoogle Scholar
  48. Hancock HC, Easen PR: The decision-making processes of nurses when extubating patients following cardiac surgery: an ethnographic study. Int J Nurs Stud. 2006, 43 (6): 693-706. 10.1016/j.ijnurstu.2005.09.003.View ArticlePubMedGoogle Scholar
  49. Powell SM, Kimberly Hill R: My copilot is a nurse—using crew resource management in the OR. AORN J. 2006, 83 (1): 178-202. 10.1016/S0001-2092(06)60239-1.View ArticleGoogle Scholar
  50. Abella BS, Alvarado JP, Myklebust H, Edelson DP, Barry A, O’Hearn N, Vanden Hoek TL, Becker LB: Quality of cardiopulmonary resuscitation during in-hospital cardiac arrest. JAMA. 2005, 293 (3): 305-310. 10.1001/jama.293.3.305.View ArticlePubMedGoogle Scholar
  51. Thomas E, Sexton J, Lasky R, Helmreich R, Crandell D, Tyson J: Teamwork and quality during neonatal care in the delivery room. J Perinatol. 2006, 26 (3): 163-169. 10.1038/sj.jp.7211451.View ArticlePubMedGoogle Scholar
  52. Buckley T, Gordon C: The effectiveness of high fidelity simulation on medical–surgical registered nurses’ ability to recognise and respond to clinical emergencies. Nurse Educ Today. 2011, 31 (7): 716-721. 10.1016/j.nedt.2010.04.004.View ArticlePubMedGoogle Scholar
  53. France DJ, Greevy RA, Liu X, Burgess H, Dittus RS, Weinger MB, Speroff T: Measuring and comparing safety climate in intensive care units. Med Care. 2010, 48 (3): 279-284. 10.1097/MLR.0b013e3181c162d6.View ArticlePubMedGoogle Scholar
  54. Snijders C, Kollen BJ, van Lingen RA, Fetter WPF, Molendijk H: Which aspects of safety culture predict incident reporting behavior in neonatal intensive care units? a multilevel analysis. Crit Care Med. 2009, 37 (1): 61-67. 10.1097/CCM.0b013e31819300e4.View ArticlePubMedGoogle Scholar
  55. Ballangrud R, Hedelin B, Hall-Lord ML: Nurses’ perceptions of patient safety climate in intensive care units: a cross-sectional study. Intensive Crit Care Nurs. 2012, 28 (6): 344-354. 10.1016/j.iccn.2012.01.001.View ArticlePubMedGoogle Scholar
  56. Salas E, Sims DE, Burke CS: Is there a “big five” in teamwork?. Small Group Res. 2005, 36 (5): 555-599. 10.1177/1046496405277134.View ArticleGoogle Scholar
  57. JCAHO: Sentinel event data root causes by event type 2004-third quarter 2011. 12-01-2012, from http://www.jointcommission.org/Sentinel_Event_Statistics/,
  58. Leonard M, Graham S, Bonacum D: The human factor: the critical importance of effective teamwork and communication in providing safe care. Qual Saf Health Care. 2004, 13 (suppl 1): i85-i90. 10.1136/qshc.2004.010033.View ArticlePubMedPubMed CentralGoogle Scholar
  59. Kruger J, Dunning D: Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Pers Soc Psychol. 1999, 77 (6): 1121-10.1037/0022-3514.77.6.1121.View ArticlePubMedGoogle Scholar
  60. Hodges B, Regehr G, Martin D: Difficulties in recognizing one’s own incompetence: novice physicians who are unskilled and unaware of it. Acad Med. 2001, 76 (10): S87-S89. 10.1097/00001888-200110001-00029.View ArticlePubMedGoogle Scholar
  61. Doumouras AG, Keshet I, Nathens AB, Ahmed N, Hicks CM: A crisis of faith? A review of simulation in teaching team-based, crisis management skills to surgical trainees. J Surg Educ. 2012, 69 (3): 274-281. 10.1016/j.jsurg.2011.11.004.View ArticlePubMedGoogle Scholar

Copyright

© Ballangrud et al.; licensee BioMed Central Ltd. 2014

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.