Evaluation of the effectiveness of rehabilitation nursing training for clinical nurses based on the Kirkpatrick model

Abstract

Background

According to the “National Nursing Development Plan (2021–2025)” issued by the National Health Commission of China, it was crucial to address the limited research available on rehabilitation nursing training. By 2025, it was expected that at least 90% of urgently needed professional nurses, including rehabilitation nursing, would participate in training. Therefore, it was necessary to explore the effects of rehabilitation nursing training for clinical nurses and provide reference suggestions for rehabilitation nursing training.

Methods

The Kirkpatrick model was used as an evaluation tool to conduct a questionnaire survey of 79 clinical nurses who participated in the 5th term of rehabilitation nursing training in Shandong Province. Paired t tests, factor analysis, and qualitative interviews were used to analyze the participants’ responses, learning styles, behaviors, and results. Four levels were used to evaluate training effectiveness.

Results

The overall satisfaction rate of the trainees was high. There were significant differences in theoretical scores, operational examination scores and related nursing abilities before and after training; these scores before training were significantly lower than scores after training (P < 0.05). Four themes were extracted from the semi structured interviews: improving relevant capabilities, optimizing training, improving safety management awareness and crisis awareness, and expanding career development space.

Conclusions

Rehabilitation nursing training significantly improved the rehabilitation skills and knowledge of clinical nurses. Therefore, it was recommended that the methods and programs used in this study be adopted for the rehabilitation nursing training of clinical nurses in all specialty areas of nursing to improve their nursing skills.

Background

Rehabilitation nursing was an important part of the medical process [1, 2]. It was a comprehensive and complex form of nursing [3] that involved assessing patients, formulating personalized rehabilitation plans, and providing treatment and support. It was designed to promote recovery and/or adapt to their new norm following illnesses or injuries. To achieve this goal, clinical nurses had to receive rehabilitation nursing training to improve their capabilities.

However, the comprehensiveness and complexity of rehabilitation nursing [3] had led to uncertainty regarding the effectiveness of training. This uncertainty stemmed not only from the differentiated needs of individual patients but was also influenced by the professional skills and experience of caregivers. In order to improve the effectiveness of training, individualized training programs had to be developed and optimized through continuous evaluation and feedback mechanisms. Therefore, evaluating the effects of nursing training was crucial for improving the utilization of medical resources and mediating the medical-patient relationship [4]. According to the “National Nursing Development Plan (2021–2025)” [5] issued by China’s National Health Commission, it was expected that by 2025, the proportion of urgently needed professional nurses, such as rehabilitation nurses, participating in training would be no less than 90%, which made research evaluating the effectiveness of nursing training a priority.

Currently, research on nursing training evaluation had focused mainly on Narrative nursing (Narrative nursing belonged to a kind of narrative medicine, which required nurses to respect patients’ life experiences and patiently listen to the stories behind their illnesses during the nursing process, and the core of which was nurses’ empathic ability [6,7,8]. Nurses’ empathic ability was an important guarantee for realizing quality care, and some studies had shown that the implementation of narrative nursing training among nurses could effectively improve the level of nurses’ empathy [9].), operating room nursing, emergency care, and other fields; less research had been conducted on rehabilitation nursing training [10, 11]. Traditional evaluation methods mainly relied on theoretical examinations and operational skill assessments and lacked ways to connect with clinical practice [12]. These factors might have been one-sided and affected patient care and safety. To solve this problem, researchers had proposed various training evaluation methods, such as the CIPP (Context evaluation, Input evaluation, Process evaluation, Product evaluation, CIPP), CIRO (Context evaluation, Input evaluation, Reaction evaluation, Output evaluation), and Kirkpatrick models [13]. The Kirkpatrick model was a comprehensive and scientific tool widely used for evaluating training effects in this field [14, 15]. For example, Kyle E. Chang used the Kirkpatrick model to evaluate the effectiveness of a comprehensive medical Spanish course in a Spanish medical project study [16]. Lavilla-Gracia et al. organized nursing students to participate in elective nursing courses at Spanish universities and used the Kirkpatrick model to classify and evaluate the data after the course was completed, demonstrating the effectiveness of conducting motivational interviewing courses on alcohol abuse [17].

The Kirkpatrick model was a comprehensive evaluation method developed in the 1950s by Donald L. Kirkpatrick, a professor at the Wisconsin University in the United States [18]. The model had been widely used to evaluate the effectiveness of training programs over the past 60 years, and its reliability had been verified. As one of the earliest evaluation models proposed and widely used, the Kirkpatrick model was well known for its systematic, comprehensive, and scientific characteristics. This model emphasized the distinction between learning and behavior. It focused on the transfer of learning into application [19]. The Kirkpatrick model simplified traditional complex assessment methods, demonstrated the interaction between skills and knowledge, and addressed the lack of long-term effect evaluations in previous research methods. Therefore, the Kirkpatrick model was suitable for evaluating hybrid training programmes. Furthermore, to fully utilize the four levels of the Kirkpatrick model, each level could be flexibly combined with other methods to achieve precise research assessments [20]. For example, Rasouil et al. used the three levels of response, learning, and behavior of the Kirkpatrick model combined with quantitative analysis methods to evaluate the effectiveness of virtual nursing critical care training courses [19], which confirmed that virtual training was an effective educational method. Based on the two levels of response and learning in Kirkpatrick’s model, Fury Maulina combined the phenomenological method to evaluate the impact of the workshop on participants’ knowledge and skills [21]. The results indicated that the workshop had a positive impact. Hence, the Kirkpatrick model was important and effective for evaluating the effectiveness of medical training. By utilizing this model, medical training and curricula could be more comprehensively evaluated and improved, enhancing the work capabilities of professionals and providing better medical services.

To ensure that the trainees received high-quality training and achieved ideal rehabilitation nursing training effects, we formulated a detailed and comprehensive training plan before the training began and carefully planned the training process. This study used the Kirkpatrick model as the evaluation method and theoretical basis for measuring the indicators of trainees before and after the training process. The purpose of this research was to prove the effectiveness of rehabilitation nursing training, evaluate the necessity of carrying out such training, and provide reference suggestions for future rehabilitation nursing training.

Methods

Design

This study, a cross-sectional study, was conducted on the participants of the 5th term of rehabilitation nursing training and their grades in training. When this cross-sectional study was approved by the Medical Ethics Committee of Shandong Second Medical University (reference number 2024YX-005), clinical nurses were invited to participate in the training. The sample included clinical nurses from medical institutions in various cities and counties in Shandong Province. After providing written informed consent, participants began receiving four months of rehabilitation nursing training.

Study sample selection

This study investigated clinical nurses in medical institutions in the cities and counties of Shandong Province. This training was conducted through open recruitment and voluntary enrollment. Considering that the number of trainees would affect the effectiveness of the training, this study had limited the number of trainees to 80 and below. At the start of the recruitment process, persons responsible for recruitment set the recruiting criteria. The recruiting criteria were as follows: (1) The need to hold nurse qualifications and nurse practitioner certificates and engage in nursing work; (2) Have some knowledge and skills in rehabilitation nursing, but have not received a certificate of training in rehabilitation nursing; (3) Ability to present clear solutions in the working process, take ownership of tasks and remain efficient, excellent oral and written communication skills. Besides that, persons responsible for recruitment also explained the recruitment criteria, and if recruiter had one of recruiting criteria, could participate in the enrollment. Nurses who did not want to participate or did not meet the criteria were excluded from the study. The exclusion criteria included failure of enrollment materials or failure to complete training or failure of training test scores. Finally, 79 nurses were included as research participants, for a sample size of 79. The response rate was 98.75%.

Implementation of the study

Developed a training plan

The training program in Table 1 strictly followed the requirements of the “Training Outline for Nurses in the Specialty Nursing Field” [22] of the Ministry of Health of China and was formulated based on the actual situation of the rehabilitation nursing profession in Shandong Province.

Table 1 Training content of rehabilitation nurses

Established a training team

The training was divided into two parts: theoretical and clinical practical training. The theoretical training team included teachers from the rehabilitation nursing field, such as Qilu Hospital of Shandong University, and rehabilitation nursing experts from other medical institutions. The clinical practice training team included clinical teachers from each internship base, for example, Jining First People’s Hospital, The Affiliated Hospital of Binzhou Medical University, The Affiliated Hospital of Qingdao University, The Second Affiliated Hospital of Shandong University of Traditional Chinese Medicine, Dezhou People’s Hospital, The Second Hospital of Shandong University and The Affiliated Hospital of Jining Medical University. These institutions trained students in multiple aspects, angles, and levels to ensure the achievement of internship teaching goals of training clinically applied rehabilitation specialist nurses with solid theoretical knowledge of rehabilitation specialties and proficiency in specialized nursing skills and conducted assessments based on students’ mastery.

Implemented the training plan

This training was conducted in the form of face-to-face teaching and clinical practice, and the students were intensively trained. Theoretical training adopted the form of face-to-face teaching. Students must participate in communication, learning, and answering questions in class. Materials and courseware were shared online by uploading them to the WeChat group chat. Clinical practice training was divided into two phases: the first phase was internship in clinical teaching bases, mainly focusing on learning from basic theory, condition observation, clinical operation, and other dimensions. After the internship ended, trainees reported to the nursing department of each internship base. The second phase involved writing rehabilitation nursing-related academic papers, with the type of paper being a review. The content should have covered the development of the current situation of rehabilitation nursing, as well as the obstacles encountered in the development and the main measures to cope with them. Each trainee was requested to submit one paper. Students must abide by the training requirements and maintain a positive learning attitude during the training process. In the clinical practice training stage, each internship base trained students in basic theory, disease observation, and clinical operational skills from multiple perspectives and levels through direct teaching, practical operations, and internships. After the first stage, there was a rehabilitation nursing technical operation assessment, and students had to submit an academic paper before the final week. Teachers at each base rated the trainees’ internship performance, and the Nursing Department submitted the trainees’ clinical internship evaluation form to the Shandong Provincial Association of Rehabilitation Medicine. After the training, a graduation ceremony was held, and training certificates were issued.

Training effect evaluation methods

Reactivity level evaluation

Reactive-level evaluation referred to the degree to which trainees responded to various elements of training [23]. A satisfaction questionnaire was designed to evaluate student satisfaction with the training. The questionnaire used in this study was developed and designed for this study by the researcher after referring to the relevant literature and consisted of two parts (see supplementary file). The first part covered five aspects—teaching methods, training format, training effect, time arrangement, and overall satisfaction—using the Likert 5-point scoring method [24]. The content reliability of the questionnaire was high (Cronbach’s α = 0.972). The second part was an open-ended question designed to collect the trainees’ suggestions for this training and further understand their level of satisfaction.

Learning level evaluation

Learning-level evaluation involved trainees’ knowledge, attitudes, and skills. Measurement scores could be used to evaluate student learning before and after training. It was important to note that knowledge and attitudes were measured differently from skills; however, the measurement methods used before and after training should be consistent [23]. Learning level evaluation scores could be used to evaluate students’ mastery of the training content and also to some extent could reflect the effectiveness of the participants’ training. Qilu Hospital of Shandong University organized teaching experts to prepare questions, establish an examination question bank, and determine the content of examination papers based on the training content, purpose, and expert opinions. The same examination question bank was used before and after training to build examination paper, for a total score of 100 points. The operational assessment standards were led by Qilu Hospital of Shandong University. Teachers who participated in specialist nurse teaching qualification training had unified operational standards and were assessed according to these standards, with a total score of 100 points. The percentage system was used for the full score of both parts, so a participant was considered to have mastered this training when the weighted total score for both components was ≥ 60. Both the theoretical and operational assessments were proctored by full-time teachers. The rehabilitation practice ability assessment was conducted clinically by teachers at each practice base through direct observation to evaluate the nurses’ rehabilitation practice ability after taking up their posts.

Behavioral level evaluation

Behavioral-level evaluation involved changes in the trainee’s behavior. Measurements could be made before and after training or at some time after training (usually three months) through model interviews or written surveys [23]. The fourth part of the questionnaire that was developed by Shandong Association of Rehabilitation Medicine, Shandong Nursing Association and Qilu Hospital of Shandong University used in this study was based on the core competencies of nurses and included 69 indicators, such as “I have a good mastery of basic theoretical knowledge related to rehabilitation” and so on to evaluate the extent to which trainees applied the training content in practice. (Due to the number of indicators on the questionnaire, we used factor analysis to create 3 dimensions that changed the behavioral evaluation tool and the following findings about the behavioral level would be based on these 3 dimensions). A reliability test was also conducted among the nurses, and the results were good (Cronbach’s α = 0.990). The questionnaire adopted a 5-point Likert scale ranging from 1 (completely disagree) to 5 (completely agree). The higher the total score is, the stronger the corresponding ability. The Star Questionnaire was used to fill in and collect questionnaires online before and after the training.

Results-level evaluation

The resulting level of evaluation involved changes in performance. Assessments were conducted before and after training using the same measures, and the results were assessed using subject-specific outcome measures. It should be noted that the posttest could not be administered immediately after training. A certain period of time (usually 6 months or half a year) must be set aside to show the results [23]. Semistructured interviews [25] were used to conduct in-depth interviews with trainees to understand the changes in their work skills and other related abilities after training. With the consent of the interviewees, face-to-face interviews were conducted with the interviewees using the interview outline developed for this study (see supplementary file), and the entire interview process was recorded and transcribed to create the research data. Stratified sampling was used to select six rehabilitation nurses, three head nurses from the 79 research subjects. The interview outline for ordinary students, teachers, and head nurses included the following content: opinions on whether training was needed for rehabilitation nursing work; whether training would help them become competent in rehabilitation nursing work; rehabilitation theoretical knowledge; practical skills; whether scientific research, teaching ability, and comprehensive ability had changed; and suggestions for follow-up training. After the interviews were completed, we refined the interview results based on the recordings made during the interviews to improve the completeness and accuracy of the interview results. Finally, themes were extracted by analyzing the interview data.

Statistical methods

Our data came from the questionnaire and we planned to process the data using SPSS 22 software. Statistical descriptive methods were used to describe the participants’ baseline information and the results of the response level, and paired t-tests to analyze the results of the learning level. When analyzing the results of the behavioral level, due to the excessive number of indicators, factor analysis was used to dimensionalize the indicators, and finally divided the indicators into three dimensions, and then the paired t-test was used to analyze the results of the three dimensions. When analyzing the result level, group discussion method was used.Members of the subject group conducted interviews with interviewees simultaneously in a one-to-one format. At the end of the interviews, organized by the tutor, the subject group members expressed their views on the results of the interviews, and four themes were finally extracted from the interview results after thorough discussion. Besides, reliability analysis was used [26]. The test level was α = 0.05.

Results

Baseline information of the research sample (see supplementary file 1)

The research subjects were from 14 departments, including the Departments of Rehabilitation, Orthopedics, Internal Medicine, Physiotherapy, Neurological Rehabilitation, and Traditional Chinese Medicine. Among them, 76 were women and 3 were men; 40 were on contract, 29 were in the corporate establishment, and 10 were personnel agents. The working experience of the participants ranged from 10 to 31 years, with an average of 12.8 years, their age ranged from 24 to 48 years, with an average of 35.3 years, 5 people were unmarried, and 74 people were married; 2 people had a college degree, 77 had a bachelor’s degree or above; and a total of 30 people have junior titles, 42 people have intermediate titles, 7 people have senior titles. There were 40 general nurses, 19 head nurses, and 20 teaching teachers; the hospital levels of the research subjects included 1 first-level hospital, 26 s-level hospitals, and 52 third-level hospitals. The types of hospitals where the research participants were located included 64 general hospitals, 2 traditional Chinese medicine hospitals, 3 integrated traditional Chinese and Western medicine hospitals, 7 rehabilitation hospitals, and 3 other hospitals.

Reaction level evaluation results

The results of the descriptive analysis of training satisfaction in Table 2 showed that 1 person (1.52%) rated it on average, 14 (17.74%) rated it as satisfied, and 64 (80.76%) as very satisfied. Based on these results, the proportion of very satisfied respondents was greater. Notably, none of the participants was very dissatisfied or dissatisfied.

Table 2 Rehabilitation nurses’ satisfaction evaluation of training programs (n = 79) people (%)

Learning level evaluation results

The results of the two-sided paired t test in Table 3 based on the students’ theoretical and operational assessment scores before and after the training showed a significant difference (P < 0.001), indicating that the training of rehabilitation nursing specialist nurses achieved significant results.

Table 3 Comparison of theoretical and operational assessment scores of rehabilitation specialist nurses before and after training (points, x̅(s))

Behavioral level evaluation results

We used factor analysis to downscaled 69 indicators, including “I can be proficient in basic theoretical knowledge related to rehabilitation”, to extracted three common factors: clinical practice, scientific research and teaching, and out-of-hospital service abilities. A paired t test was conducted on the scores of these three common factors before and after the training. The results in Table 4 as showed a significant difference in scores (P < 0.001), indicating that training could improve nurses’ professional abilities.

Table 4 Scores of corresponding ability evaluation of rehabilitation nurses before and after training (points, x̅(s))

Results level evaluation results

Before starting the discussion on the interview data, the subject group restricted the themes to be extracted to nurses, competencies, and training based on the purpose of the study. Through analysis of the interview data, this study identified the following themes: (1) The relevant abilities improved, including clinical rehabilitation operations and interpersonal communication skills; (2) There was still room for further training optimization, such as less ward management training content and clinical training. There were fewer learning opportunities and a single teaching mode; (3) safety management awareness and crisis awareness improved; and (4) Training broadened the career development space for clinical nurses.

Discussion

The Kirkpatrick model was gradually being used in the field of nursing training

Researchers had been studying how to establish scientific, objective, systematic, and comprehensive training evaluation methods. As a new method, the Kirkpatrick model had been widely used for training evaluation [27] to construct a training evaluation index system. For example, Li et al. applied the Kirkpatrick model in a nurse training program in an emergency surgery department based on clinical demand during the COVID-19 pandemic (coronavirus disease 2019) [28]. Heydari et al. used Kirkpatrick’s model to measure the effect of a new teaching and learning methods workshop for health care staff [29]. The interview method was used to construct an evaluation index system for the standardized training of new nurses. These indicator systems could be used to evaluate the effectiveness of nurse training. In addition, training effects were evaluated. For example, Rasouli et al. used the Kirkpatrick model to evaluate the training effects of supporting nurses in intensive care wards [19]. Ando et al. used the Kirkpatrick model to guide the evaluation of airway management training [30]. In summary, this research showed that the Kirkpatrick model had been well applied in the field of nursing training, could provide effective evaluation for training, and could promote progress in the nursing industry.

Training could improve the rehabilitation nursing capabilities of clinical nurses

Although the response-level evaluation results could not be used as the main evaluation indicator for training, this could provide reference suggestions for improving training. As shown in Tables 2 and 79 trainees were satisfied with the program. Learning-level evaluations mainly assessed the effectiveness and quality of training from two perspectives: knowledge and skill mastery [31]. There was a significant difference in the students’ theoretical and operational performance before and after training. The scores after the training were higher than those before the training, indicating that the training was effective. The behavioral evaluation results showed that after the training, the students’ clinical practice, scientific research and teaching, and out-of-hospital service abilities significantly improved (P < 0.05), indicating that the training of rehabilitation nurses could effectively improve related abilities [32]. The evaluation results of the outcome layer were consistent with the above levels. The interview results showed that standardized rehabilitation nursing specialist nurse training could expose nursing staff to richer, more professional, and more targeted knowledge; improve the nursing staff’s rehabilitation nursing technical operation level; and thus provide patients with better services.

Suggestions for optimizing rehabilitation nursing training for clinical nurses

In the current environment of high-quality and rapid development of hospitals, it was urgent and necessary to strengthen the rehabilitation nursing ability training of nurses for the rapid recovery of patients, reduce the cost of rehabilitation, and improve the rehabilitation concepts, professional levels, and professional quality of nursing staff [33]. This study used the Kirkpatrick model to conduct an empirical analysis of the effectiveness of rehabilitation nursing training for clinical nurses. The results showed that the training program was scientific, effective, and highly practical. Moreover, the Kirkpatrick model revealed weak links during training and provided a reliable basis for subsequent improvements in training projects. Based on this, the following suggestions were proposed for future rehabilitation nursing training: (1) Improving the training effect evaluation index system; (2) Strengthening training implementation management and conducting post training follow-up evaluation management; (3) Using flexible teaching methods and combining online and offline teaching and (4) Paying attention to a combination of theory and practice.

Limitations

The small sample size limited the generalizability of the results. Further research could improve the reliability and validity of this study by increasing the sample size. In addition, our trainees varied widely in terms of job title, age and work experience, and our study lacked subgroup analysis. In the future, we might have consider refining the analysis according to trainees’ characteristics, which would have made the results of the study more relevant.

Conclusions

This study demonstrated that rehabilitation nursing training for clinical nurses could effectively improve their skill levels. In particular, there was a level of competence in specialized rehabilitation care. Therefore, it was recommended that the methods and protocols used in this study be referred to for the rehabilitation nursing training of clinical nurses to improve their nursing skills.