Skip to main content
Download PDF

Face-to-face versus 360° VR video: a comparative study of two teaching methods in nursing education

BMC Nursing volume 23, Article number: 199 (2024) Cite this article

Abstract

Background

The practical sessions during skills laboratory simulation or clinical simulation are cores of nursing education. For this, different modalities have been devised to facilitate psychomotor skills learning. One of the commonly used educational material or instructional method to supplement skills learning across various disciplines is video-based teaching method. The opportunities of traditional two-dimensional video might be limitless and maximized with 360º virtual reality (VR) video, which offers immersive experience. This study incorporates 360º VR video into skills laboratory training as an alternative approach to face-to-face procedure demonstration.

Methods

An open-label, parallel (1:1), randomized controlled trial study was conducted among third-year undergraduate nursing students at Hiroshima University, Japan. The nursing students were block-randomized into 360º VR video and face-to-face demonstration group. After a 3-hour theoretical class of patient management on ventilator and closed-suction principles of mechanically ventilated patients in an Intensive Care Unit focused class, the 360º VR group watched the 360º VR video of closed tracheal suction (including oral) using the head-mounted display of Meta Quest 2 individually, while the face-to-face group attended the instructor’s demonstration. A week after the skills laboratory, the students’ psychomotor skills, knowledge, satisfaction, confidence were evaluated; the 360º VR video group’s perception was explored; Wilcoxon rank-sum test was used to compare the two groups.

Results

A total of 57 students were analyzed; 27 students in the 360º VR video group and 30 students in face-to-face group. There were no statistically significant differences between both groups in skills, knowledge, and confidence. However, the face-to-face group had higher satisfaction level than the 360º VR group; this difference was statistically significant. In the 360º VR video group, 62% agreed that VR makes learning more interesting; more than half of students (62.5%) experienced VR sickness symptoms, and “feeling of drunk” is the highest. The students appreciated the ready to use, immersiveness, and realism; however, symptoms and discomfort, burdensomeness, and production limitations were improvements recommended.

Conclusion

Although face-to-face demonstration is the established method of teaching psychomotor skills to nursing students, the use of 360º VR video could achieve similar learning effect as an alternative approach.

Peer Review reports

Background

Clinical training is an indispensable foundation in nursing education; in principle, it is the pathway to prepare nursing students, ensure competence, and achieve patient`s safety in the healthcare system. The pathway to ensuring competent nurse is the translation of theoretical knowledge to practical knowledge [1, 2]; this is enabled through skills laboratory. The practical sessions during skills laboratory simulation or clinical simulation are cores of nursing education. For this, different modalities have been devised to facilitate psychomotor skills learning. However, despite the adoption of strategies to ensure the application of theory into practice, nursing education still experiences nursing students and newly graduates with deficient practical skills [3,4,5].

One of the commonly used educational material or instructional method to supplement skills learning across various disciplines is video-based teaching method [6,7,8,9,10]. Medical students source educational videos to learn clinical skills [11], and 90% of medical students reported using videos to learn procedures [12]. Moreover, the use of educational technology is part of nursing education and nurses are forerunners [13] It is argued that learning through image is relatively experiencing the real situation or an experiential process [14]. The use of educational videos in teaching positively affects the learning process [13, 15], and has shown to enhance performance [13, 16], significantly reduce study time compared to text-based material [16], and improve confidence in performing some procedures [17].

The opportunities of traditional two-dimensional (2D) video might be limitless and maximized with 360º virtual reality (VR) video, which offers immersive experience. 360º VR video employ real-world images captured with an omnidirectional camera or multiple cameras simultaneously to create an immersive environment [18]. The term VR and 360º VR video are used interchangeably; although VR is generated by using computer graphics, 360º VR video is created from real-world images [18, 19]. It is noteworthy that the defining factor of a VR system in research reviews is the VR technology rather than the level of interactivity. The undivided attention offered by 360º positively influences conceptual and spatial learning [20]. The 360º video with head-mounted display (HMD) might provide an edge over 2D videos where environmental distractions are in view.

The potential benefits of 360º VR video on learning outcomes [21], and suitability for action-oriented activities requiring visual details, which is infeasible in a traditional 2D Video [22] has been demonstrated in research. This immersive and involvement opportunity in 360º VR video has raised a debate on its use in retention of information and enhancement of learning over traditional 2D video. Harrington et. al [23] reported 65% of students preferred 360º Video over 2D; the 360º VR video group demonstrated significant higher engagement and no difference in information retention. Contrarily, Rupp et al [24] found the overwhelming feeling of presence contributed to less information recall. It is well established that 360º VR video improves student learning performance [25,26,27]. A systematic review on 360º VR video technology by Baysan et al [19], which included majority of non-interactive 360º video systems, concluded that the use is convenient and effective for nursing education. For this, robust research is essential as disparities exist between studies.

In Japan, research interest in VR and using VR in nursing education is increasing. However, to the best of our knowledge, the adoption of VR in nursing education is not widespread in any country. In nursing skills laboratory, procedures are demonstrated to students by nursing instructors of the intended procedure; a web video is provided to complement for future reference. The instructors deliver the procedure to the total number of students at once; this crowding could hamper the ease of understanding and better visualization. It has been reported in research that a video-based group perceived the teaching method to facilitate ease and better understanding than live demonstration [28, 29]. Closed tracheal suction is one of the important nursing procedures in Intensive Care Unit (ICU) and involves action-oriented activities requiring visual details. If this demonstration is captured in immersive 360º VR, it could offer an individualized experience, be reused by students without the need for web video as supplement and reduce faculty dependence in future demonstration of the procedure. Moreover, video-based teaching is a self-directed learning approach and could reduce the number of instructors needed to conduct hands-on practice in nursing skills laboratory; teachers’ dependence of students is one of the negatives of live demonstration [30]. Therefore, this study incorporates 360º VR video into skills laboratory training as an alternative approach to face-to-face procedure demonstration. The purpose of this study was to examine the effects of 360º VR video and face-to-face teaching method in learning closed tracheal suction (including oral suction). We hypothesized that (1) nursing students who learned the procedure with 360º VR video would have higher skill performance scores than students who received the face-to-face demonstration, (2) the 360º VR video group would have better theoretical knowledge than the face-to-face group, (3) the nursing students who watched the 360º VR video would self-report higher satisfaction and confidence level than the students who received face-to-face demonstration, and (4) the 360º VR video would have considerably good perception of the use of the technology.

Method

Study design and participants

An open-label, parallel (1:1), randomized controlled trial design was conducted among undergraduate nursing students at Hiroshima University, Japan. Participants were third-year nursing students enrolled in the Practicum in Adult Nursing in 2023.

In the third year, students study each nursing science area after completing basic nursing subjects; they study theory and skills in parallel, and after completion, they go on to clinical practice.

Data collection procedure

Since this study was implemented in the regular class, the data for this analysis was obtained after the class completion as an opt-in basis. Prior to commencement of the specified class explained below, students were informed of the purpose of this experiment, procedure, voluntarily of participation, no disadvantages of withdrawal and/or no participation, and the secondary use of the data. Then, students submitted the written consent form for providing their data submitted in the class to the researcher who was not involved in the course.

After a 3-hour theoretical class of patient management on ventilator and closed-suction principles of mechanically ventilated patients in an ICU focused class, a seven-question knowledge pretest was conducted for all the available nursing students enrolled in the course; a total of 62 students completed the pretest for randomization.

Randomization and allocation

To assure equal distribution in terms of academic achievement or intelligence, the pretest score was used as a factor to block randomize the students into the 360º VR video training group (360º VR group: an intervention group) and the face-to-face traditional training group (face-to-face group: a control group). A block size of 2 resulted in 31 blocks, and students assigned from each block into the face-to-face and 360º VR groups. Figure 1 shows the study procedure during the course.

Fig. 1
figure 1

Study procedure

Full size image

Development of the 360º VR video

A video of a certified nurse in critical care performing closed-suction procedure in a high-fidelity mannequin-based simulation was recorded with Insta360 (ONE X2). The procedure of suctioning was conducted in a step-by-step manner following a checklist developed by the research team. The video involved a voice over of the instructor explaining the procedure, and the nurse performing the procedure. This was edited using Adobe Premiere Pro ver (23.2.0); the final product of the video was 18 min divided into three phases for better understanding of the procedure: Preparation and assessment phase; instrument identification and oral suction phase; tracheal suction and patient report phase (Fig. 2).

Fig. 2
figure 2

A shot from the 360º VR video. A shot of nursing students watching the procedure with HMDs

Full size image

360º VR group

After the theoretical class, the 360º VR group watched (see Fig. 2a) the 360º VR video using the HMD of Meta Quest 2 individually; after then, the students answered the VR safety questionnaire developed by the researcher. For hands-on practice, the students engaged in self-directed practice with group feedback in 5 subgroups for 90 min; each subgroup containing 6 students except one with 7 students.

Face-to-face group

The face-to-face group attended the face-to-face demonstration of the same nurse that carried out the procedure in the video using the researcher developed checklist. After then, the students engaged in hands-on practice in 5 subgroups for 90 min; each subgroup containing 6 students except one having 7 students; clinical instructors (experienced registered nurses) were present in each subgroup during the practice.

For both groups, the critical care nurse captured in the video was available between the intervention and control groups to address the students’ concerns and questions. After the hands-on practice, the control group was provided the usual supplemental procedural video for closed suctioning attached to their study material, and the intervention group could either rewatch the 360º VR video with VideoLAN Client (VLC) player or request for use of the HMD at their convenience.

A week after the skill demonstration and hands-on practice, a total of 9 instructors (nursing faculties and registered nurses) assessed and evaluated the students’ closed suction skills (including oral suction) using the procedure checklist. To ensure consistent evaluation, a session was held to communicate the grading criteria. The students were requested to perform the procedure in an Objective Structured Clinical Exam (OSCE) and evaluated by clinical instructors. At the end of the skills evaluation, both groups answered the Microsoft Forms on knowledge, satisfaction, and confidence. In addition, the 360º VR group answered the VR perception questionnaire. According to the study procedure, it was planned to explore the perception of the face-to-face group on VR by watching the 360º VR video after skills evaluation, but none of the students watched the 360º VR video.

Evaluation outcomes and instrument

Evaluation was conducted under the framework of psychomotor skills, knowledge, confidence, and satisfaction of the closed tracheal suction technique. For the 360º VR group, perception and VR sickness symptoms were also explored.

Closed tracheal suction checklist (including oral suction)

A Closed tracheal suction (including oral suction) checklist was developed from available literature review of evidence-based practice [31,32,33,34,35,36,37] to evaluate the skills of the nursing students (supplementary file 1). To ensure the validity of the checklist, the developed checklist was submitted to certified critical care nurses of Hiroshima University Hospital, and a version of procedure checklist available at the unit was received by the researchers as a guide. The checklist was further modified resulting in 38-steps procedure (items) checklist. To assign grades to the steps, each step was dichotomized to critical and non-critical. For a critical step, a score of 4, 2, 0 was assigned to satisfactory, unsatisfactory, and not performed, respectively; a score of 2, 1, 0 was assigned to a non-critical item as aforementioned for the level of performance. The criterion for the three level of performance was outlined for each item for consistent rating. To establish the content validity, using Lynn’s (1986) technique [38], the checklist was submitted to four certified critical care nurses; the relevance, accuracy of terminology, and grading of the steps were evaluated. The checklist was reviewed and modified based on the experts’ opinion, and the final checklist score ranges from 0 to 64. The item-level content validity index (I-CVI) was computed for each item; the scale-level content validity index of universal agreement (S-CVI/UA) was 0.97.

Knowledge test scores for suction in ventilated patients

The researchers developed practical knowledge questions on tracheal suctioning (supplementary file 2). A total of 24 questions were outlined, and after the researchers’ group discussion, it was reduced to 17 questions. This was pretested with two certified critical care nurses for an expert-driven pretest to assess the face and construct validity of the questionnaire. The nurses answered the questionnaire, and suggested modifications or discard of some questions were addressed accordingly. After then, two questions were added, and 19 questions were pretested with two different certified critical care nurses. In order to achieve a 20-questions questionnaire, one question was included to the final expert-driven pretest. From the 20 questions developed, 7 questions, which were identified to address the basics and overview of tracheal suctioning, were used for pretest. For the post test, the total of 20 questions was administered; the correct answer is given 1 point, and the incorrect answer is given 0 points.

Degree of satisfaction and confidence in learning

To assess the students’ satisfaction and confidence, the Japanese version [39] of the Students Satisfaction and Self-Confidence in Learning by the National League for Nursing (NLN) was adopted. It consists of 13 questions in two different questionnaire; five questions for satisfaction and eight questions for self-confidence. The questionnaire is on a 5-point Likert scale form 1 = strongly disagree, 2 = disagree, 3 = undecided, 4 = agree, and 5 = strongly agree; the higher the score, the higher the satisfaction and confidence. The satisfaction score ranges from 5 points to 25 points and confidence from 8 points to 40 points. As reported by NLN, the Cronbach alpha for the satisfaction and self-confidence are 0.94 and 0.87, respectively. In this study, the Cronbach alpha for satisfaction and self-confidence is 0.93 and 0.92, respectively.

Perception of 360º VR use (including open-ended questions)

The perception of 360º VR video group was assessed with an adapted tool from Peart et al [40] study. The tool was developed based on the Technology Acceptance Model (TAM) and included 6 items on a 7-point Likert scale (strongly disagree, disagree, somewhat disagree, cannot decide, somewhat agree, agree, and strongly agree) and 2 open ended questions. However, to fit in our study, only one of the open-ended questions was retained (is there a way that the use of X could be improved). The tool was forward and back translated by the researchers, and an additional two questions (1. How was the comfort and ease of understanding of VR; 2. If you notice anything else or have any impressions, please write it down) were added to the open-ended questions. In Peart et al [40], the Cronbach alpha was > 0.7. As the tool was translated and adapted, the Cronbach alpha in this study is 0.61.

The safety questionnaire was developed to explore the side effects of using the VR. The VR sickness symptoms explored in the questionnaire were based on the Meta Quest 2 health and safety manual and other VR studies [19, 25, 41, 42]. It consisted of two questions (1) did you have any symptoms (2) please, pick all that applies. Ten symptoms were provided as option with an “other” option to allow for free answer.

Ethical consideration

The study was conducted according to the Declaration of Helsinki and the Ethical Guidelines on Clinical Studies of the Ministry of Health, Labour and Welfare of Japan. This study was reviewed and approved by the Hiroshima University Epidemiological Ethics Review Committee (E2023-0054). One of the researchers who was not part of the adult health nursing course explained the study purpose and data collection procedure, and consent was received from students agreeing to secondary use of the data. It was explained that not consenting to the provision of data obtained in class would not affect the class grade in any way, and there would be no disadvantage on the part of the students; a written informed consent was obtained from all the students. Therefore, to ensure the class instructors would not be able to identify which students had consented, consent procedure and data extraction were done by the research coordinator.

Data analysis

Skills checklist and questionnaires

Data analysis was performed with JMP, Pro 17 (SAS Institute Inc., Cary, NC, 1989–2023). Due to the non-normal distribution, descriptive data are presented in median, quartile, frequency, and percentage. Wilcoxon rank-sum test was used to compare the two groups in skills, knowledge, satisfaction, and confidence. The perception is presented as frequency, and percentages based on the level of agreement on the Likert scale; the open-ended questions were analyzed following the conceptual content analysis method to describe the attitudinal and behavioral responses of the students toward the 360º VR video. The VR side effects are presented as frequency and percentage. The level of significance was considered at 0.05.

Open-ended questions for feedback

The open-ended questions were analyzed following the conceptual content analysis method to describe the attitudinal and behavioral responses [43] of the students toward the 360º VR video. The coding unit of analysis was defined as the individual theme; according to Minichiello et al. as cited in Zhang and Wildemuth [44], this strategy is to capture the expressions of an idea. In the initial stage of the open coding, the phrases used by the students were singled out to enable in vivo codes; this prevents contamination of the data and allows valid representation of the students’ idea [45]. Furthermore, the in vivo codes of similar ideas were grouped together; the codes were organized to derive categories. The process of the analysis was examined by the research team and disagreements in the process were addressed accordingly.

Result

Out of the 62 students randomized for the study, data of 57 students were used for analysis (Fig. 3). They were all females and aged between 20 and 22 years.

Table 1 shows the median score of both groups based on the knowledge pretest for randomization. Both groups were equally randomized, and there is no significant difference between the two groups.

Fig. 3
figure 3

CONSORT flow chart

Full size image
Table 1 Pretest score for students allocated into face-to-face and 360º VR Video groups
Full size table

Psychomotor skills of the students on closed endotracheal suctioning

The median scores for face-to-face and 360º VR groups were 56.5 and 56.0, respectively; there is no significant difference between the two teaching methods (Z= -0.385, P = 0.700) (Table 2; Fig. 4).

Table 2 Psychomotor skills score of closed tracheal suction (including oral suction)
Full size table
Fig. 4
figure 4

Psychomotor skill scores of closed tracheal suction (including oral suction)

Full size image

Knowledge of the nursing students

Among the 57 students, 55 students answered the knowledge test (Face-to-Face = 29; 360º VR = 26). The median scores for face-to-face and 360º VR groups were 16 and 15, respectively. No statistically significant difference was observed between the groups (Z = 0.059, P = 0.952). Table 3 presents the median scores of the respective teaching methods.

Table 3 Knowledge scores of closed tracheal suction (including oral suction)
Full size table

The levels of satisfaction and confidence

Tables 4, 5 and 6; Fig. 5 presents the comparison between the level of satisfaction and confidence of the face-to-face and 360º VR groups. While there was statistically significant difference in the level of satisfaction between the two groups, the difference in the confidence level was not significant. On the item-level satisfaction, all the statements are statistically significant except, “the teaching material used in this simulation were motivating and helped me to learn” (P = 0.063).

Table 4 Satisfaction and Confidence scores of students using the face-to-face and 360º VR Video teaching method
Full size table
Table 5 Satisfaction scores of students using the face-to-face and 360º VR Video teaching methods
Full size table
Table 6 Confidence scores of students using the face-to-face and 360º VR Video teaching methods
Full size table
Fig. 5
figure 5

Satisfaction scores of students using the face-to-face and 360º VR Video teaching method. Confidence scores of students using the face-to-face and 360º VR Video teaching method

Full size image

The perception of the 360º VR use

Table 7 shows the perception of the 360º VR group regarding the VR use. It was intended to introduce the 360º VR to the face-to-face group and explore their perception, but no student in the face-to-face group volunteered to watch 360º VR video after the class. Therefore, the endpoint of comparing how both groups perceived the 360º VR video use was not achieved. Out of the 27 students in the VR video group, a total of 26 students answered the questionnaire. Among them, 81% of the students disagreed that VR is a bad idea, and 42.3% perceived the technology useful for learning. About 62% agreed that VR makes learning more interesting; however, 35% of the student cannot decide if they would like to use VR video in future clinical skills, and 42% disagreed.

Table 7 Perception of students using the 360º VR Video
Full size table

VR sickness symptoms

Table 8 presents the VR sickness symptoms reported by the students using 360º VR video. More than half of students (62.5%) experienced VR sickness symptoms, and “feeling of drunk” is the highest.

Table 8 VR sickness symptoms reported by the students using 360º VR Video
Full size table

Qualitative analysis of the open-ended questions

In order to supplement the quantitative data, the open-ended questions were analyzed, and yielded 5 categories (see Table 9). The students appreciated the ready to use, immersiveness and realism; however, symptoms and discomfort, burdensomeness and production limitations were improvements recommended. Moreover, the impression (see Table 10) of the students concerning the 360º VR video was preference for face-to-face teaching because it offers the opportunity to engage with the instructor and ask practical questions. Another aspect is some students believed regular videos are easier to watch. One of the students noted “Difficult, because I have to use equipment (headset) to review videos”.

Table 9 Qualitative analysis of students’ comments on 360º VR video teaching method (N = 26)
Full size table
Table 10 Nursing students’ impression of 360º VR video teaching method (N = 26)
Full size table

Discussion

This study assesses the effectiveness of 360º VR video in teaching nursing procedure over the traditional face-to-face teaching method. Our hypotheses were that the 360º VR video group would demonstrate better skill, knowledge, confidence, and satisfaction level than the face-to-face group; these hypotheses were not supported as there was no statistically significant difference between the groups in skill, knowledge, and confidence, and the face-to-face group had higher satisfaction level than the 360º VR video group. On the other hand, the qualitative result suggested that VR symptoms, burdensomeness, and production limitation, were negative experiences often cited; the feeling of immersion and the opportunity to reuse the video were positive aspects of the 360º VR video perceived by the students.

The research on 360º VR video is a relatively new area in medical education, and teaching method for comparison are not usually the same, which makes the evidence disparate. A complementary study by Arents et al [41] compared a group of students learning medical obstetrics and cesarean section in face-to-face combined with 360º VR video to the face-to-face group only; there was no statistically significant difference between the groups in knowledge retention. Similar to our findings, in Sweden, Ulrich et al [46] in a three-arm study (i.e., 360º VR video group, Regular video group, Traditional teaching group) compared physiotherapy students on academic performance and students’ learning satisfaction. The three methods have the same effect on enhancement of academic performance; however, the traditional teaching was more effective than both 360º VR video and regular video in students’ learning satisfaction. In a similar approach to our study in Saudi Arabia, Sultana et al [47] compared 360º VR group of medical students learning communication skills with a conventional group that received interactive lecture on the same skills. Contrary to our findings, the 360º VR video group scored significantly higher than the conventional group in Multiple Choice Questions (MCQs) and OSCE. In other studies comparing 360º VR video to 2D, Yoganathan et al [48] compared first year postgraduate doctors’ skills of knot tying using 360º VR video and 2D video. The 360º VR video arm performed significantly better than the 2D arm. In Taiwan, Chao et al [25] compared nursing students learning nasogastric tube feeding with 360º VR video and regular demonstration video on the outcomes of skills, knowledge, satisfaction, and confidence. There was no statistically significant difference between the two groups in skill, knowledge, and confidence; however, the VR group demonstrated higher satisfaction than the traditional video group.

It was expected that the immersiveness, higher engagement and enthusiasm associated with the use of 360º VR video could afford the students a higher possibility of effective learning [49,50,51,52]. The non-significant study outcomes might be that the action-oriented activities involved cognitively demanding details that requires extra attention; highly complex learning environments increase the cognitive load [53]. Based on the students’ feedback, there were concerns on having to move and turn in the virtual space to follow up on the procedure; it is the first time to use such novel technology to learn procedure, and it might be tiring and distracting. One of the students stated that “I had to move my head to see some parts of the procedure, so it was more difficult to understand than a live lecture”; another student noted “the saturation monitor was behind the patient, so I had to turn around to see it.” This could possibly be addressed by providing orientation on what to expect in the virtual space. Moro et al [52] maintained that there is a risk of distraction with the use of VR technology; the participants reported spending more time on exploring the technology rather than learning the contents. Chao et al [25] also maintained similar conclusion. Likewise, the experience of VR symptoms is a convergent finding as it is supported by the quantitative and qualitative data; this could have hampered the learning experience. The result of the VR symptoms revealed that, about 63% (15 out of 24 students) of the students reported VR related symptoms in this study. Moro et al [52] reported that the adverse effects of dizziness, blurred vision, and headaches were felt by 40%, 35%, and 25% of the students, respectively; this could have an impact on learning quality. In this study, one of the students states “It was immersive and realistic, and I could feel what was going on firsthand, but the distortion of the screen and the eye strain and headaches associated with it made it difficult for me to use it on a daily basis”. The participants in Van De Broeck [54] concluded that although the immersiveness with HMDs offer the best user experience, they are associated with cognitive burden, motion sickness and physical discomfort. Somrak et al [55] reported negative association between VR sickness discomfort levels and user experience.

The satisfaction level of the face-to-face group was significantly higher than the 360º VR video group in this study. However, considering the statement on the satisfaction questionnaire, which states, “the teaching materials were motivating and helped me to learn”; there was no statistically significant difference between the scores of the 360º VR video and face-to-face group. This means that both groups equally agreed on the teaching methods being motivating and helpful. For the overall satisfaction, to begin with, the possible reason might be that this study explores the satisfaction of a video-based group, which has been believed by students to lack the opportunity to ask questions and interact with the instructor [56]. Likewise, the teaching method was adopted as an alternate approach rather than blending face-to-face with video for the intervention group. Our qualitative finding suggests that the 360º VR video group prefers the presence of the instructor in face-to-face teaching. Similar to this findings, Alqahtani et al [28] concluded that students were reluctant to replace live demonstration with procedural video; only 40% of the students preferred the procedural video compared to the 59% in face-to-face demonstration. Also, another statement on the satisfaction questionnaire, which states “The way my instructor(s) taught the simulation was suitable to the way I learn”; the face-to-face group scored significantly higher. Our students might have appreciated the interaction opportunity in the face-to-face teaching; the use of VR is a new method of teaching and students are already familiar and accustomed to the face-to-face teaching method. This study compares 360º VR video to face-to-face teaching; It is worth pointing out that the studies demonstrating significantly higher satisfaction for 360º VR video compared two different video-based method (VR versus regular demonstration video). It is difficult to extrapolate these studies' result on satisfaction to our study based on the difference in approach. A three-arm study by Ulrich et al [46] found that traditional teaching was more effective than 360º video and regular video in students’ learning satisfaction. Additionally, the discomfort experienced through the VR sickness symptoms might have affected their satisfaction level. For perception, only a moderate percentage (42.3%) of the 360º VR video group students rated VR useful for learning, and 23% would like to use it in future skills training. Contrarily, in Arents et al [41] a complementary approach, 100% of the students rated it useful, and 83.4% reported that more 360º VR videos should be used in future courses.

Limitations and strengths

The limitation in this study is, to begin with, the intervention was open to the student, and this could have alerted and give the impression of non-conformity to the new technology as a replacement for the established teaching method. This effect and performance bias could have been mitigated with a blinded study, but blinding was not feasible. Moreover, the 360º VR video was only content validated; the production was not validated for use. This could have also affected the students experience of the technology leading to production limitation. Additionally, all the students in our study were females; if the genders were mixed, there could be more generalization as gender factors in the acceptance of technology. Subsequently, our sample was a convenience sample of the nursing students enrolled in the course; a larger sample size could have achieved a normal distribution and mitigate the effect of using a non-parametric test. Also, this is a single-center study in Japan; the findings cannot be generalized to other nursing students.

Conclusion

Although face-to-face demonstration is the established method of teaching psychomotor skills to nursing students, the use of 360º VR video could achieve similar learning effect as an alternative approach. Nevertheless, only a moderate percentage of the students in the 360º VR video group perceived the technology useful for learning. While this is true for learners’ performance, there is need for more studies to explore the students’ satisfaction when used as an alternative. Moreover, factors like ease of use and VR sickness symptoms experienced by users hinder the acceptance of the teaching method.

Data availability

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

CVI:

Item-level content validity index

CVI/UA:

Scale level content validity index of universal agreement

HMD:

Head Mounted Display

ICU:

Intensive Care Unit

MCQ:

Multiple Choice Question

NLN:

National League for Nursing

OSCE:

Objective Structured Clinical Exam

TAM:

Technology Acceptance Model

VR:

Virtual Reality

VLC:

VideoLAN Client

2D:

Two-dimensional

References

  1. Bjørk IT, Larsen K, Ravik M, Reierson IÅ, Sommer I, Stenholt B, et al. Development and testing of an instrument for Summative Assessment of practical skill performance: a Generalizability Theory Approach. J Nurs Meas. 2021;29(3):E162–91.

    PubMed  Google Scholar 

  2. Toriente Relloso J, Abdullah AbuAlula N, Magtalas Medina J, Gatioan Manood E. Nursing Skills Laboratory as Milieu of Clinical Learning and Practice. American Journal of Nursing Research [Internet]. 2021 May 10 [cited 2022 Oct 20];9(4):112–7. Available from: http://pubs.sciepub.com/ajnr/9/4/2/index.html.

  3. Jamshidi N, Molazem Z, Sharif F, Torabizadeh C, Kalyani MN. The Challenges of Nursing Students in the Clinical Learning Environment: A Qualitative Study. 2016; https://doi.org/10.1155/2016/1846178.

  4. Zamanzadeh V, Jasemi M, Valizadeh L, Keogh B, Taleghani F. Lack of Preparation: Iranian nurses’ experiences during Transition from College to Clinical Practice. J Prof Nurs. 2015;31(4):365–73.

    Article  PubMed  Google Scholar 

  5. Bowling AM. The effect of simulation on skill performance: a need for change in pediatric nursing education. J Pediatr Nurs. 2015;30(3):439–46.

    Article  PubMed  Google Scholar 

  6. Nadelson LS, Scaggs J, Sheffield C, McDougal OM. Integration of Video-Based Demonstrations to Prepare Students for the Organic Chemistry Laboratory. J Sci Educ Technol [Internet]. 2015 Dec 3 [cited 2023 Jan 26];24(4):476–83. Available from: https://link.springer.com/article/https://doi.org/10.1007/s10956-014-9535-3.

  7. Teo TW, Tan KCD, Yan YK, Teo YC, Yeo LW. How flip teaching supports undergraduate chemistry laboratory learning. Chemistry Education Research and Practice [Internet]. 2014 Oct 1 [cited 2023 Jan 26];15(4):550–67. Available from: https://pubs.rsc.org/en/content/articlehtml/2014/rp/c4rp00003j.

  8. Pape-Koehler C, Immenroth M, Sauerland S, Lefering R, Lindlohr C, Toaspern J et al. Multimedia-based training on Internet platforms improves surgical performance: A randomized controlled trial. Surg Endosc [Internet]. 2013 Mar 9 [cited 2023 Jan 26];27(5):1737–47. Available from: https://link.springer.com/article/10.1007/s00464-012-2672-y.

  9. Smith W, Rafeek R, Marchan S, Paryag A. The use of video-clips as a teaching aide. European Journal of Dental Education [Internet]. 2012 May 1 [cited 2023 Jan 26];16(2):91–6. Available from: https://onlinelibrary.wiley.com/doi/full/https://doi.org/10.1111/j.1600-0579.2011.00724.x.

  10. Mendez A, Seikaly H, Ansari K, Murphy R, Cote D. High definition video teaching module for learning neck dissection. Journal of Otolaryngology - Head and Neck Surgery [Internet]. 2014 Mar 25 [cited 2023 Jan 26];43(MAR):1–7. Available from: https://journalotohns.biomedcentral.com/articles/https://doi.org/10.1186/1916-0216-43-7.

  11. Bezner SK, Hodgman EI, Diesen DL, Clayton JT, Minkes RK, Langer JC, et al. Pediatric surgery on YouTube™: is the truth out there? J Pediatr Surg. 2014;49(4):586–9.

    Article  PubMed  Google Scholar 

  12. Rapp AK, Healy MG, Charlton ME, Keith JN, Rosenbaum ME, Kapadia MR. YouTube is the most frequently used Educational Video Source for Surgical Preparation. J Surg Educ. 2016;73(6):1072–6.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Salina L, Ruffinengo C, Garrino L, Massariello P, Charrier L, Martin B et al. Effectiveness of an educational video as an instrument to refresh and reinforce the learning of a nursing technique: a randomized controlled trial. Perspect Med Educ [Internet]. 2012 May 1 [cited 2023 Jan 26];1(2):67–75. Available from: https://link.springer.com/article/https://doi.org/10.1007/s40037-012-0013-4.

  14. Davis B, IMAGE LEARNING HIGHER, EDUCATION AND INTERACTIVE VIDEODISC. High Educ New Technol. 1989;409–14.

  15. Ludwig S, Schuelper N, Brown J, Anders S, Raupach T. How can we teach medical students to choose wisely? A randomised controlled cross-over study of video- versus text-based case scenarios. BMC Med [Internet]. 2018 Jul 6 [cited 2023 Jan 26];16(1):1–9. Available from: https://bmcmedicine.biomedcentral.com/articles/https://doi.org/10.1186/s12916-018-1090-y.

  16. Friedl R, Höppler H, Ecard K, Scholz W, Hannekum A, Öchsner W et al. Multimedia-Driven Teaching Significantly Improves Students’ Performance When Compared With a Print Medium. Annals of Thoracic Surgery [Internet]. 2006 May 1 [cited 2023 Jan 26];81(5):1760–6. Available from: http://www.annalsthoracicsurgery.org/article/S0003497505017613/fulltext.

  17. Srivastava G, Roddy M, Langsam D, Agrawal D. An educational video improves technique in performance of pediatric lumbar punctures. Pediatr Emerg Care [Internet]. 2012 Jan [cited 2023 Jan 26];28(1):12–6. Available from: https://journals.lww.com/pec-online/Fulltext/2012/01000/An_Educational_Video_Improves_Technique_in.4.aspx.

  18. Snelson C, Hsu YC. Educational 360-Degree Videos in Virtual Reality: a Scoping Review of the Emerging Research. TechTrends [Internet]. 2020 May 1 [cited 2023 Sep 12];64(3):404–12. Available from: https://link.springer.com/article/https://doi.org/10.1007/s11528-019-00474-3.

  19. Baysan A, Çonoğlu G, Özkütük N, Orgun F. Come and see through my eyes: a systematic review of 360-degree video technology in nursing education. Nurse Educ Today. 2023;128:105886.

    Article  PubMed  Google Scholar 

  20. Tüzün H, Özdinç F. The effects of 3D multi-user virtual environments on freshmen university students’ conceptual and spatial learning and presence in departmental orientation. Comput Educ. 2016;94:228–40.

    Article  Google Scholar 

  21. Ray AB, Deb S. Smartphone Based Virtual Reality Systems in Classroom Teaching - A Study on the Effects of Learning Outcome. Proceedings - IEEE 8th International Conference on Technology for Education, T4E 2016. 2017;68–71.

  22. Ardisara A, Fung FM. Integrating 360° Videos in an Undergraduate Chemistry Laboratory Course. J Chem Educ [Internet]. 2018 Oct 9 [cited 2023 Jan 26];95(10):1881–4. Available from: https://pubs.acs.org/doi/full/https://doi.org/10.1021/acs.jchemed.8b00143.

  23. Harrington CM, Kavanagh DO, Wright Ballester G, Wright Ballester A, Dicker P, Traynor O, et al. 360° operative videos: a randomised cross-over study evaluating attentiveness and Information Retention. J Surg Educ. 2018;75(4):993–1000.

    Article  PubMed  Google Scholar 

  24. Rupp MA, Kozachuk J, Michaelis JR, Odette KL, Smither JA, McConnell DS. The effects of immersiveness and future VR expectations on subjec-tive-experiences during an educational 360° video. https://doi.org/101177/1541931213601477 [Internet]. 2016 Sep 8 [cited 2023 Jan 26];2101–5. Available from: https://journals.sagepub.com/doi/https://doi.org/10.1177/1541931213601477.

  25. Chao YC, Hu SH, Chiu HY, Huang PH, Tsai HT, Chuang YH. The effects of an immersive 3d interactive video program on improving student nurses’ nursing skill competence: a randomized controlled trial study. Nurse Educ Today. 2021;103:104979.

    Article  PubMed  Google Scholar 

  26. Harrison B, Oehmen R, Robertson A, Robertson B, De Cruz P, Khan R et al. Through the eye of the master: The use of Virtual Reality in the teaching of surgical hand preparation. 2017 IEEE 5th International Conference on Serious Games and Applications for Health, SeGAH 2017. 2017.

  27. Yang TY, Huang CH, An C, Weng LC. Construction and evaluation of a 360 degrees panoramic video on the physical examination of nursing students. Nurse Educ Pract. 2022;63:103372.

    Article  PubMed  Google Scholar 

  28. Alqahtani ND, Al-Jewair T, Al-Moammar K, Albarakati SF, Alkofide EA. Live demonstration versus procedural video: A comparison of two methods for teaching an orthodontic laboratory procedure Approaches to teaching and learning. BMC Med Educ [Internet]. 2015 Nov 4 [cited 2023 Jun 22];15(1):1–4. Available from: https://bmcmededuc.biomedcentral.com/articles/https://doi.org/10.1186/s12909-015-0479-y.

  29. Aragon CE, Zibrowski EM. Does Exposure to a Procedural Video Enhance Preclinical Dental Student Performance in Fixed Prosthodontics? J Dent Educ [Internet]. 2008 Jan 1 [cited 2023 Sep 12];72(1):67–71. Available from: https://onlinelibrary.wiley.com/doi/full/https://doi.org/10.1002/j.0022-0337.2008.72.1.tb04454.x.

  30. Nikzad S, Azari A, Mahgoli H, Akhoundi N. Effect of a Procedural Video CD and Study Guide on the Practical Fixed Prosthodontic Performance of Iranian Dental Students. J Dent Educ [Internet]. 2012 Mar 1 [cited 2023 Sep 12];76(3):354–9. Available from: https://onlinelibrary.wiley.com/doi/full/https://doi.org/10.1002/j.0022-0337.2012.76.3.tb05266.x.

  31. Blakeman TC, Scott JB, Yoder MA, Capellari E, Strickland SL. AARC Clinical Practice guidelines: Artificial Airway Suctioning. Respir Care. 2022;67(2):258–71.

    Article  PubMed  Google Scholar 

  32. Elsevier. Elsevier– Clinical Skills Endotracheal Tube and Tracheostomy Tube Suctioning [Internet]. 2023 [cited 2023 Jul 4]. Available from: https://elsevier.health/en-US/preview/endotracheal-tube.

  33. Elsevier. Endotracheal Tube and Tracheostomy Tube Suctioning-CE [Internet]. 2020 [cited 2023 Jul 4]. Available from: https://www.elsevier.com/__data/assets/pdf_file/0008/996749/Endotracheal-Tube-and-Tracheostomy-Tube-Suctioning-Checklist_080520.pdf.

  34. Leeds Teaching Hospitals NHS. Tracheal Suctioning [Internet]. 2022 [cited 2023 Jul 4]. Available from: http://www.lhp.leedsth.nhs.uk/detail.aspx?id=3152.

  35. Centers for Disease Control and Prevention. 2022-05-11 12:30| Archive of CDC Covid Pages [Internet]. 2020 [cited 2023 Jul 4]. Available from: https://public4.pagefreezer.com/browse/CDC%20Covid%20Pages/11-05-2022T12:30/https://www.cdc.gov/coronavirus/2019-ncov/hcp/using-ppe.html.

  36. Elsevier. Elsevier– Clinical Skills Suctioning: Artificial Airway During Mechanical Ventilation (Respiratory Therapy) [Internet]. 2023 [cited 2023 Jul 4]. Available from: https://elsevier.health/en-US/preview/suctioning-artificial-airway.

  37. American Association for Respiratory Care (AARC) Clinical Practice Guidelines. AARC Clinical Practice Guidelines Endotracheal Suctioning of Mechanically Ventilated Patients With Artificial Airways 2010 [Internet]. 2010 [cited 2023 Jul 4]. Available from: https://rc.rcjournal.com/content/respcare/55/6/758.full.pdf.

  38. Polit DF, Beck CT. The Content Validity Index: are you sure you know what’s being reported? Critique and recommendations. Res Nurs Health [Internet]. 2006;29:489–97. Available from: www.interscience.wiley.com.

    Article  PubMed  Google Scholar 

  39. Akiko I. Evaluation of simulation-based learning using simulated patients in nursing students simulation-based learning, nursing student, evaluation of teaching methods, simulated patients. Methodol Fund Nurs Skills [Internet]. 2015;12:51–59.

  40. Peart DJ, Rumbold PLS, Keane KM, Allin L. Student use and perception of technology enhanced learning in a mass lecture knowledge-rich domain first year undergraduate module. International Journal of Educational Technology in Higher Education [Internet]. 2017 Dec 1 [cited 2023 Jul 5];14(1):1–11. Available from: https://educationaltechnologyjournal.springeropen.com/articles/https://doi.org/10.1186/s41239-017-0078-6.

  41. Arents V, de Groot PCM, Struben VMD, van Stralen KJ. Use of 360° virtual reality video in medical obstetrical education: a quasi-experimental design. BMC Med Educ [Internet]. 2021 Dec 1 [cited 2023 Jan 26];21(1):1–9. Available from: https://bmcmededuc.biomedcentral.com/articles/https://doi.org/10.1186/s12909-021-02628-5.

  42. Oculus. Quest 2| Health and Safety Centre| Oculus UK [Internet]. [cited 2023 Sep 12]. Available from: https://www.oculus.com/safety-center/quest-2/.

  43. Content Analysis Method and Examples| Columbia Public Health [Internet]. [cited 2023 Sep 8]. Available from: https://www.publichealth.columbia.edu/research/population-health-methods/content-analysis#.

  44. Zhang Y, Wildemuth BM. Qualitative Analysis of Content.

  45. Holloway I, Galvin K. Qualitative Research in nursing and Healthcare. Fourth. West Sussex, UK: John Wiley & Sons Inc. John Wiley & Sons Inc., editor. Chichester, 2017. pp. 1–360.

    Google Scholar 

  46. Ulrich F, Helms NH, Frandsen UP, Rafn AV. Learning effectiveness of 360° video: experiences from a controlled experiment in healthcare education. Interactive Learning Environments [Internet]. 2021 [cited 2023 Nov 9];29(1):98–111. Available from: https://www.tandfonline.com/doi/abs/https://doi.org/10.1080/10494820.2019.1579234.

  47. Sultan L, Abuznadah W, Al-Jifree H, Anwar Khan M, Alsaywid B, Ashour F. An Experimental Study On Usefulness Of Virtual Reality 360° In Undergraduate Medical Education. Adv Med Educ Pract [Internet]. 2019 [cited 2023 Nov 9];10:907. Available from: /pmc/articles/PMC6826194/.

  48. Yoganathan S, Finch DA, Parkin E, Pollard J. 360° virtual reality video for the acquisition of knot tying skills: a randomised controlled trial. Int J Surg. 2018;54:24–7.

    Article  CAS  PubMed  Google Scholar 

  49. Battulga B, Konishi T, Tamura Y, Moriguchi H. The Effectiveness of an Interactive 3-Dimensional Computer Graphics Model for Medical Education. Interact J Med Res. 2012;1(2):e2 https://www.i-jmr.org/2012/2/e2 [Internet]. 2012 Jul 9 [cited 2023 Sep 9];1(2):e2172.

  50. Limniou M, Roberts D, Papadopoulos N. Full immersive virtual environment CAVETM in chemistry education. Comput Educ. 2008;51(2):584–93.

    Article  Google Scholar 

  51. Shim KC, Park JS, Kim HS, Kim JH, Park YC, Ryu H, Il. Application of virtual reality technology in biology education. http://dx.doi.org/101080/0021926620039655854 [Internet]. 2010 [cited 2023 Sep 9];37(2):71–4.

  52. Moro C, Štromberga Z, Raikos A, Stirling A. The effectiveness of virtual and augmented reality in health sciences and medical anatomy. Anat Sci Educ [Internet]. 2017 Nov 1 [cited 2023 Jul 31];10(6):549–59. Available from: https://onlinelibrary.wiley.com/doi/full/https://doi.org/10.1002/ase.1696.

  53. Frederiksen JG, Sørensen SMD, Konge L, Svendsen MBS, Nobel-Jørgensen M, Bjerrum F et al. Cognitive load and performance in immersive virtual reality versus conventional virtual reality simulation training of laparoscopic surgery: a randomized trial. Surg Endosc [Internet]. 2020 Mar 1 [cited 2023 Sep 12];34(3):1244–52. Available from: https://link.springer.com/article/https://doi.org/10.1007/s00464-019-06887-8.

  54. Van Den Broeck M, Kawsar F, Schöning J. It’s all around you: Exploring 360° video viewing experiences on mobile devices. MM 2017 - Proceedings of the 2017 ACM Multimedia Conference. 2017;762–8.

  55. Somrak A, Humar I, Hossain MS, Alhamid MF, Hossain MA, Guna J. Estimating VR Sickness and user experience using different HMD technologies: an evaluation study. Future Generation Comput Syst. 2019;94:302–16.

    Article  Google Scholar 

  56. Bazyk S, Jeziorowski J. Videotaped versus live instruction in demonstrating evaluation skills to occupational therapy students. Am J Occup Ther [Internet]. 1989 [cited 2023 Jul 31];43(7):465–8. Available from: https://pubmed.ncbi.nlm.nih.gov/2750862/.

Download references

Acknowledgements

We would like appreciate Hiroshima University Hospital Intensive Care Unit (ICU) Nurses and the students for their contribution. In addition, the nursing faculties and registered nurses that evaluated the psychomotor skills (Hiroyuki Sawatari, Tae Watanabe, Mizuho Kidani, Mikako Yoshihara, Haruka Yoshimaru, Emi Kawasaki, Kenji Matsuo).

Funding

The purchase of the HMDs and development of the 360º VR video content was funded by the Ministry of Education, Culture, Sports, Science and Technology, FY2021 Subsidy for Promotion of University Reform (Project for Fostering Medical Personnel for New Healthcare in the “With COVID-19 Era”).

Author information

Authors and Affiliations

  1. Graduate School of Biomedical and Health Sciences, Program of Health Sciences, Hiroshima University, Hiroshima, Japan

    Abdulfatai Olamilekan Babaita

  2. Division of Nursing Science, Graduate School of Biomedical and Health Sciences, Hiroshima University, Kasumi 1-2-3 Minami-ku, Hiroshima, 734-8553, Japan

    Mayumi Kako, Chie Teramoto & Michiko Moriyama

  3. Intensive Care Unit, Hiroshima University Hospital, Hiroshima, Japan

    Miho Okamoto & Yoko Hayashi

  4. Department of Emergency and Critical Care Medicine, Graduate School of Biomedical and Health Sciences, Advanced Emergency and Critical Care Center, Hiroshima University, Hiroshima University Hospital, Hiroshima, Japan

    Shinichiro Ohshimo & Takuma Sadamori

  5. Center for Medical Education, Faculty of Medicine, Hiroshima University, Hiroshima, Japan

    Minoru Hattori

Authors
  1. Abdulfatai Olamilekan Babaita
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mayumi Kako
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chie Teramoto
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Miho Okamoto
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yoko Hayashi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shinichiro Ohshimo
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Takuma Sadamori
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Minoru Hattori
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Michiko Moriyama
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Concept and design: AOB, MK, CT, MO, MM; Development of educational content: AOB, MK, MO, YH, TS, MH, MM; Implementation: CT, MO, MM; Data collection: MK; Data analysis: AOB; Interpretation of data: AOB, SO, MM; Manuscript drafting: AOB, MM; Manuscript review: MK, CT, SO, TS, MH, MM. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Michiko Moriyama.

Ethics declarations

Ethics approval and consent to participate

The study was conducted according to the Declaration of Helsinki and the Ethical Guidelines on Clinical Studies of the Ministry of Health, Labor and Welfare of Japan. This study was reviewed and approved by the Hiroshima University Epidemiological Ethics Review Committee (E2023-0054). One of the researchers who is not part of the adult health nursing course explained the study purpose and data collection procedure, and consent was received from students agreeing to secondary use of the data. It was explained that not consenting to the provision of data obtained in class would not affect the class grade in any way, and there would be no disadvantage on the part of the students; a written informed consent was obtained from all the students. Therefore, to ensure the class instructors would not be able to know which students had consented, consent procedure and data extraction were done by the research coordinator.

Consent for publication

Written informed consent for publication was obtained for use of the images.

Competing interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary Material 2

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Babaita, A.O., Kako, M., Teramoto, C. et al. Face-to-face versus 360° VR video: a comparative study of two teaching methods in nursing education. BMC Nurs 23, 199 (2024). https://doi.org/10.1186/s12912-024-01866-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12912-024-01866-4

Keywords

BMC Nursing

ISSN: 1472-6955

Contact us