Research Paper Critical Analysis

1
Research Paper Critical Analysis

• 10 ways to look at a research paper
  systematically for critical analysis.

2
NUR 599T Human Factors In Healthcare Engineering

• Dr. Vincent Duffy
• Research Paper Analysis
• Submitted by
• RuthAnn Smolen, RN, MS Ed, HNP,
• July 19, 2007

3
STUDY CRITIQUE Title Assessment of
Clinical Performance during Simulated Crises
Using Both Technical and Behavioral
RatingsAuthors Gaba, D., Howard, S.,
Flanagan B., Smith, B., Fish, K., Botney, R.
4
Research Ideas or Question(Study and Purpose)

• Study Assessment in clinical performance Gaba,
  et.al.)
• The purpose is clearly identified in the
  introduction -
• to evaluate ratings of performance (technical
  and behavioral), and variability of ratings made
  by multiple independent observers viewing
  videotapes of simulated crises
• The use of human patient simulators (HPS) in
  health-care training is increasing to gain
  knowledge, teach technical skills, complete drill
  training, and human factors training for single
  or multidisciplinary groups

5
2. Background

• Anesthesiology is recognized as the leading
  medical specialty in addressing patient safety
  with HPS
• Pioneered development of computer screen and
  mannequin based interactive patient simulators
  and training plans (Schwid, 1987 Small, Wuetz,
  Simon, Shapiro, Conn, Setnik, 1999).
• A persistent initiative in graduate education is
  to find more effective methods to assess
  specialists skills
• Techniques are needed to assess
  anesthesiologists performance when responding to
  critical events (Gaba, 1998)
• Traditionally there has been concern about the
  potential subjectivity of expert raters.

6
3. Theoretical Basis/Hypothesis

• Helmreich, RI Theory underlying CRM training
  Psychological issues in flight crew performance
  and crew coordination (NASA, 1986) in teams
• Behavioral psychologists and learning theories
  were used originally
• In anesthesia, patient safety depends on a
  complex interaction between a skilled
  practitioner and technology
• A human patient simulator may reduce the risk of
  learning-curve error in anesthesiologists
  technical and behavioral clinical performance
  without subjecting patients to risk

7
4. Literature Search

• 13 reports published between 1980 2000 4 were
  designed to investigate validity or reliability
  of HPS systems overview of existing evidence of
  the problem is addressed in follow-up searches of
  today
• In this study the literature review is supportive
  of purpose and results, current for 1998, and
  thorough
• Summaries of other comparative studies are
  discussed, relevant to the study and includes
  strengths and weaknesses
• Chopra, et al. (pg. 13)
• Devitt, et al. (pg. 13)
• Law and Sherman (pg. 15)

8
5. Applicability

• The aviation industrys commitment to simulation
  and similarities between anesthesia and aviation
  was a key rationale for developing courses in
  crisis management and using simulation in
  anesthesia education
• Generalization among various acute events in
  simulation teaching and evaluation environments
  across healthcare disciplines is a reality today

9
6. Methodologies

• Raters scored the videotapes of 14 different
  teams that were managing two scenarios malignant
  hyperthermia (MH) and cardiac arrest. Technical
  performance and crisis management behaviors were
  rated.

10
7. Statistical Analyses Assumptions

• Using three observers, the study shows good
  interrater reliability in scoring technical
  achievement with less agreement between five
  raters scoring ACRM behaviors.
• The authors analyze the statistics of interrater
  reliability in depth and note that the scoring
  system requires further development before it can
  be used for assess clinical competence for
  residency/graduation, or board certification

11
8. Presentation of Results

• What do the results really mean?
• Team performance varied and correlations among
  interraters were better for technical markers
  than for behavioral markers

12
9. Conclusions (are they reasonable?)

• The rating system needs refinement before it can
  be used to assess clinical competence for
  residency/graduation, or board certification
• This study suggests that simulation can be used
  to measure more complex skills expected in
  specialty training
• Similar to other studies assessing complex
  technical and behavioral performance multiple
  encounters are needed to estimate skill
  effectively and accurately

13
10. Future Work/Research Direction

• Studies emphasize the cognitive and social
  aspects of designing and operating advanced
  technologies yet to be developed
• The behavioral rating system can be improved
  difficulty aggregating a single rating that
  fluctuated with time
• Human Factors and ergonomics link quantitative
  skills to the integrated systems and make sure
  design of systems act in accordance with
  requirement of users includes interactions with
  technologists
• Questions
• Is there a need for simulation in assessment of
  clinical performance?
• Do current simulators match the fidelity of a
  real-live clinical setting? Are they realistic?
• What are the potential uses of clinical
  simulation in medical/healthcare education?
• What role exists for simulation in healthcare
  professional training and assessment?
• Do collaborative opportunities exist?