ASSESSMENT and EVALUATION

1
ASSESSMENT andEVALUATION

• R. Danzinger M. Burrows
• 2006

2
Pre-Test Sample Questions

• Do you know
• 
  
• Where to find the objectives for the Undergrad.
  medical programs?
• The objectives of the undergraduate programs in
  which you teach?
• Where to find the ITETS/FITERS for your
  undergrad. teaching?
  
• Your responsibility in completing undergrad.
  ITERS/FITERS?
• The difference between feedback and summative
  assessment?
  
• B. Are you comfortable in
  
• Your role as a teacher of Medical students?
  
• Giving feedback to students?
  
• Evaluating students?

3
The Magic Wand

• You have just found a magic wand that allows you
  to make a change related to assessment and
  evaluation.
• What would you change for the
• Postgraduate Program?
• Undergraduate Program?

4
Agenda - Objectives

• Review Curriculum Objectives
• Identify the Purpose of Assessment Evaluation
• Examine Some Tools - FITER
• Examine Feedback Characteristics
• Practice Specific A E Skills Tasks

5
Evaluation in UGME PGME
POLICY
RESOURCES
Planned Clinical Experiences ITER
Report Card ITER

• Mid-Point
• Remedial
• Learning Primer
• Debriefing
• Observational Learning

UGME PGME Program Planning Assessment
6
Curriculum Expectations

• Objectives for Rotation

7
Assessment Evaluation
8
Purpose of Assessment Evaluation

• The improvement of student learning.

9
Assessment is -

• The gathering and interpreting of information
  using a variety of tools and techniques.
• It is the act of describing student performance.
• It is the ability to compare performance with
  objectives.
• Teacher feedback is essential during this process
  to enhance student learning.

10
Evaluation is -

• The process of making judgments about students at
  a particular point in time, based on information
  from various sources.
• Is based on a specific set of learning
  objectives.
• A reflection of student performance - should
  reflect their most consistent and recent efforts.
  
• Is done for reporting purposes High Stake.

11
Objectives -

• Forms a written contract between the learner and
  the teacher(s).
• Used for assessment and evaluation.
• Allows students to continually self-evaluate
  their performance against the objective.
• Feedback can provide students with indicators of
  where they currently are and what they must do to
  achieve the objective at the required performance
  level.

12
Objectives contd -

• Assessments evaluations are objective, not
  subjective based on performance.
• Students know the objectives before the rotation.
  
• Objectives form the basis of the clinical
  experience.
• Students are assessed and evaluated during the
  rotation using the FITER scale that gives a clear
  judgment of student ability and performance.

13
Assessment Three Types

• Diagnostic
• Formative
• Summative

14
Diagnostic -

• Assessment designed to give information about a
  students skill level in a particular area, prior
  to instruction.

15
Formative -

• An ongoing process
• Provides direction for student improvement
• Allows the teacher to modify the experience
  environment
• Provides a basis for course directors to evaluate
  the program

16
Summative

• Used at the end of a unit of instruction or a
  prescribed period of time to inform the student,
  the teacher and the program about the extent of
  the students learning progress.
• Performance is compared to a standard the
  objectives.
• Reporting FITER - high stake

17
Principles of Assessment

• The principles of assessment have been
  established to ensure that assessment and
  evaluation are valid and reliable and lead to an
  improvement in student learning.

18
Principles of Assessment

• Address what how well
• Appropriate
• Communicated clearly
• Fair
• Varied and over time
• A S - K
• Use of observed performance as evidence
• Directions for improvement
• Self directed assessment

19
Task the deliverable

• Continuous Diagnostic Learning Assessment
• Mid-point evaluation for student that is
  progressing at the expected rate.
• Intervention Strategy for student who has a
  Problem
• Summative Evaluation
• FITER Written comments

20
Diagnostic AssessmentProblem Area
Identification Collaborative Action Plan
21
Problem Identification

• Identify the General Problem Area
• Provide comments/examples related to general
  problem area
• Emphasize Standard of performance the Objective
• Observe students response to problem area
  identification

22
Collaborative Action Plan to Address Problem

• Review Behavioral Expectations
• Formulate action steps time frame
• Outline evaluation process time frame
• Record consequences process if problem not
  resolved
• Follow up next meeting

23
Formative throughout rotation

• Learning primer or rehearsal
• Debriefing
• Address what how well
• Appropriate
• Communicated clearly
• Fair

24
Summative End of RotationEvaluation
25
FITER Areas of Evaluation

• Clinical expertise
• Communicator
• Professional
• Scientist/Scholar
• Core criteria total scores
• Overall evaluation
• Comments for the Deans letter.

26
(No Transcript)
27
(No Transcript)
28
Potential Evaluation Problems

• Halo Effect
• Observations not made
• No mid-point evaluation fair warning
• Learner may become defensive
• Be able to justify your comments be specific
  about what you saw, read, heard

29
Type Frequency of Problems (Hunt, Carline,
Tonesk, Jergan, Loebel, 1989)

• Type I - Frequent and Difficult
• Student characteristics
• bright
• poor interpersonal skills
• excessively sky
• non-assertive

30
Type Frequency of Problems

• Type II - Frequent and Not Difficult
• Student characteristics
• poor integration skills
• over eager
• disorganized
• disinterested
• poor knowledge base

31
Type Frequency of Problems

• Type III - Not Frequent and Difficult
• Student characteristics
• untrustworthy
• has psychiatric problem
• has abuse problem
• manipulative

32
Type Frequency of Problems

• Type IV - Not Frequent and Not Difficult
• Student characteristics
• hostile
• rude
• too casual informal
• avoids work
• challenges everything
• all thumbs

33
Type Frequency of Problems (Hendren, 1988)

• 1- Interpersonal Problems . . highest risk for
  failure not open to intervention
• 2- Academic Problems . . next highest risk
• 3- Intrapersonal conflicts anxiety . . best
  success rate open to intervention
• 4- Intrapersonal academic Problems . . best
  success rate open to intervention

34
Evaluation in UGME PGME
POLICY
RESOURCES
Planned Clinical Experiences FITER
Report Card FITER

• FEEDBACK
• Mid-Point
• Remedial
• Learning Primer
• Debriefing
• One Minute Teacher
• Observational Learning

UGME PGME Program Planning Assessment
35
Feedback for Learners

• Effective Techniques and Strategies

36
Definition of Feedback
The process by which the teacher provides
learners with information about their performance
for the purpose of improving their performance.
37
Feedback Types - A - S - K

• Attitudes
• Evidence of professionalism
• Skills
• Rotation specific
• Knowledge
• Has the information comprehension
• Can use the information application
• Can analyze and synthesize the information
  problem solving

38
Feedback is -

• First elicited from the learner
• Descriptive rather than evaluative
• Specific rather than general
• Directed toward behavior the student can do
  something about

39
Characteristics of Effective Feedback

• Specific use examples
• Timing frequency - watch setting
• Positive/Negative feedback sandwich
• Learner Reaction
• Action Plan - interactive

40
Opportunities for Feedback

• Read and respond to students patient notes
• Include the student in procedures
• Look at MRIs, xrays, slides etc.
• Case presentations and physical exam by student
• Debriefing after clinical experience
• Observation of students encounters with patients
  and other health care providers

41
Giving Feedback - Summary

• Find opportunities for giving feedback that is
  timely and ongoing
• Give frequent and specific feedback
• Provide reinforcing and corrective feedback
• Involve learner in self-assessment and reaction
  to feedback
• Involve the program director in the process
• Develop an action plan on improving future
  performance in regards to knowledge, skills and
  attitudes.

42
Task 1

• You are completing your second rotation as an
  R2, feeling frustrated at the lack of feedback or
  formal evaluation, and are not really sure how
  you are doing.
• Evaluation comments from attending staff
• Performance appropriate to clinical level. . .
  Keep up the good work.
• How do you respond to this type of evaluation?

43
Task 2

• Write narrative comments for an evaluation on one
  of the following areas
• Clinical expertise
• Communicator
• Professional
• Scientist/Scholar
• Positive comment
• Area for improvement comment

44
Effective Teaching Behaviors

• Think about teachers you liked . . . .
• What did they do that make you learn?
• NOW
• Think about teachers you didnt like . .
• What did they do that turned you off learning?

45
How do you rate as a teacher?
46
Effective Teaching Behaviors

• Encourages active student participation
• Enthusiastic about teaching
• Demonstrated patient care skills
• Emphasis on applied problem solving

47
Effective Teaching Behaviors

• Assigned responsibility for patient care
  appropriately
• Student-centered instructional strategy
• Humanistic orientation
• Barrows, 1990 Irby et al 1991Bordage, 1995

48
Evaluation in UGME PGME
POLICY
RESOURCES
Planned Clinical Experiences FITER
Report Card FITER

• FEEDBACK
• Mid-Point
• Remedial
• Learning Primer
• Debriefing
• Observational Learning

UGME PGME Program Planning Assessment
49
TO SUMMARIZE -

• As an educator
• Know the learning objectives
• Provide an orientation where expectations are
  clarified
• Diagnose learning problems, document give
  feedback
• Develop an action plan for change in behavior
  planned for with time frame and consequences
• Mid Point documentation feedback as
  required
• Evaluation students fulfills or does not
  fulfill contract FITER Grading

50
Questions
51
Selected Bibliography

• Andrew, B.J. (1977). The use of behavioural
  checklists to assess physical examination skills.
  Journal of Medical Education 52, 589-590.
• APGO (1994). Promoting clinical excellence How
  to strengthen your clerkship program. Author
  Washington, DC.
• Barrows, H.S. (1990). Inquiry the pedagogical
  importance of a skill central to clinical
  practice. Medical Education, 24 3 - 5.
• Bordage G. (1995) Where are the history and
  physical? Canada Medical Association Journal
  152(10), 1595-1597
• Brookfield, S.D. (1994). Adult learners Motives
  for learning and implications for practaice. In
  Kenneth A. Feldman andMichael B. Paulsn, Teaching
  and learning in the college classroom. Ginn
  Press Needham Heights, MA.
• Cohen, A. Cohen, R. (1990). Developmentof
  competence inclerkship. Medical Teacher, 12(1),
  47-55.

52
Selected Bibliography

• Coles, C.R. (1990). Elaborated learning in
  undergraduate medical education. Medical
  Education, 24 14- 22.
• Coles, C.R. (1990). Helping students with
  learning difficulties in medical and health-care
  education. Medical Education, 24 300-312.
• Davis, M.H., and Dents, J.A. (1994). Comparison
  of student learning in the out-patient clinic and
  ward round. Medical Education, 28 208-212.
• Dent, JA., Davis, M.H. (1995). Role of
  ambulatory care for student-patient interaction
  the EPITONE model. Medical Education 29, 58-60.
  
• Edwards, J.C., Kissling, G.E., Plauche, W.C., and
  Marier, R.L. (1988). Evaluation of a teaching
  skills improvement programme for residents.
  Medical Education, 22 514-517.
• Ende, J. (1983). Feedback in clinical medical
  education. JAMA 250 (6), 777-781.
• Foley, R., Smilansky, J., Yonke, A. (1997).
  Teacher-student interactions in the medical
  clerkship. Journal of Medical Education, 54,
  622-626.

53
Selected Bibliography

• Gravdal, J., Glasser, M. (1987). The integration
  of the student into ambulatory primary care a
  decade of experience. Family Medicine 19 (6),
  457-462.
• Greer,T., Schneeweiss, R. Baldwin, L.M. (1993).
  A comparison of student clerkship experiences
  in community practices and residency based -
  clinics. Family Medicine, 25(5), 322-326.
• Harper A.C., Roy W.E., Norman, G.R., Rand,
  C.A., Feightner, J.W. (1983). Difficulties in
  clinical skills evaluation, Medical Education 17,
  24-27.
• Hendren, R.L. (1988). Predicting success and
  failure of medical students t risk for dismissal.
  Journal of Medical Education 63, 597-601.
• Hunskaar, S., Seim, S.H. (1984). The effective
  of a checklist on medical students exposures to
  practical skills. Medical Education 18, 439-442.
  
• Hunt, D., Carlisle, J., Tonesk, X., Yurgan, S.J.,
  Siever, M., Loveel, P. (1989). Types of problem
  students encountered by clinical teachers on
  clerkship. Medical Education 23, 14-18

54
Selected Bibliography

• Hutter, N.J., Dungy, C., Zakus, G., Moore, V.,
  Ott, J.B., Favrety, A.C. (1977). Interviewing
  skills A comprehensive approach to teaching and
  evaluation. Journal of Medical Education 52,
  328-333.
• Irby, D. (1983). Evaluating instruction in
  medical education. Journal of Medical Education
  58, 844-849.
• Irby, D., Shannon, N.F., Sher, Y., Peckham, P.,
  Ko, K., Davis, V. (1977). The use of medical
  student ratings for multi-instructor courses.
  Journal of Medical Education 52, 668-673.
• Irby, D.M., Gillmore, G.M., Ramsey, P.G. (1987).
  Factors affecting ratings of clinical teachers by
  medical students and residents. Journal of
  Medical Education 62, 1-7.
• Kirz, H.L., Larcen, C. (1986). Costs and
  benefits of medical student training in the
  health maintenance organization. JAMA 256 (6),
  734-739.
• Klos, M., Reuler, J.B., Nardone, D.A., Girard,
  D.E. (1983). An evaluation of training
  performance in the case presentation. Journal of
  Medical Education 58, 432-433.
• Kong, H., Robb, K. , Cleave-Hogg, D. Evans, K.
  (1991). Achievement of clinical objectives
  Internal medicine fourth year clinical clerkship.
  Medical Teacher, 13(1),29-37.

55
Selected Bibliography

• Links, P.S. Foley, F., Feltham, R. (1988). The
  education value of student encounter logs in a
  psychiatry clerkship. Medical Teacher, 10(1),
  33-40.
• Moeller, P. (1984). Clinical supervision
  Guidelines for managing the problem student.
  Journal of Allied Health, August, 205-210.
• Mooney, G.A., Bligh, J.G., Leinster, S.J.,
  Wareniux, H.M. (1995). An electronic study guide
  for problem-based learning. Medical Education
  29, 397-402.
• Neher, J. O., Gordon, K. C., Meyer, B.,
  Stevens, N. (1992). A five-step "microskills"
  model of clinical teaching. Journal of the
  American Board of Family Practice, 5, 419-424.
• Newble, D.I., and Entwistle, N.J. (1986).
  Learning styles and approaches implications for
  medical education. Medical Education, 20
  162-175.
• Newble, D.I., Entwistle, N.J., Hejka, E.J.,
  Jollys, B.C., and Whelans, G. (1988). Towards the
  identification of student learning problems the
  development of a diagnostic inventory. Medical
  Education, 22 518-526.
• Newle, D.I. (1988). Eight years experience with
  a structured clinical examination. Medical
  Education 22, 200-204.
• Newle, D.I., Jaeger, K. (1983). The effective
  assessments and examinations on the learning of
  medical students. Medical Education 17, 165-171.
  

56
Selected Bibliography

• Newle, D.I., Swanson, D.B. (1988). Psychometric
  characteristics of the objective structured
  clinical examination. Medical Education 22,
  325-334.
• Noel, G.L. (1987). A system for evaluating and
  counselling marginal students during clinical
  clerkship. Journal of Medical Education 62,
  353-355.
• Norman, G., Tugwell, P., Feightner, J. (1982). A
  comparison of residents performance on real and
  simulated patients. Journal of Medical Education
  57, 708-715.
• Norman, G.R. (1988). Problem-solving skills,
  solving problems and problem-based learning.
  Medical Education, 22 279-286.
• Pactione, G., Cohen, E., Schwartz, E. (1989). A
  new teaching model in ambulatory medicine.
  Archives of Internal Medicine 149, 2407-2411.
• Patel, Vimla L., Groen, G.J., and Frederiksen,
  C.H. (1986). Differences between medical students
  and doctors in memory for clinical cases. Medical
  Education, 20 3-9.
• Paul, S., Bojanczyk, M., and Lanphear, J.H.
  (1994). Learning preferences of medical students.
  Medical Education, 28 180-186.

57
Selected Bibliography

• Phelan, S., Obenshain, S.S., Galey, R. (1993).
  Evaluation of the non-cognitive professional
  traits of medical students. Academic Medicine
  68(10), 799-803.
• Pratt. D., Magill, M.K. (1983). Educational
  contracts A basis for effective clinical
  teaching. Journal of Medical Education 58,
  462-467.
• Quill, T.E. (1987). A cross-sectional study of
  the influence of the ambulatory receptor as a
  role model. Archives of Internal Medicine 147,
  971-973.
• Rutala, P., Witzke, D.B., Leko, B.O., Fulginiti,
  J.B., Taylor, P. (1991). Sharing of information
  by students in an objective structured clinical
  examination. Archives of Internal Medicine 151,
  541-544.
• Stritter, F.T., Hain, J.H. (1977). A workshop in
  clinical teaching Journal of Medical Education
  52, 155-157.
• Stritter,F.T., Hain, J.D. Grimes, D.A. (1975).
  Clinical teaching re-examined. Journal of
  Medical Education, 50, 876-882.
• Sweig, S., Glenn, J., Reid, J., Williamson, H.,
  Garrett, E. (1989). Activities of the attending
  physician in the ambulatory setting What part
  is teaching? Family Medicine 21 (4), 263-267.

58
Selected Bibliography

• Tonesk, X., Buchanan, R.G. (1987). AAMC pilot
  study by ten medical schools of clinical
  evaluation of students. Journal of Medical
  Education 62, 707-718.
• Tonest, X. (1979). The house officer as a
  teacher What schools expect and measure.
  Journal of Medical Education 54, 613-616.
• Weinholtz, D. (1983). Directing medical student
  clinical case presentations. Medical Education
  17, 364-368.
• Whitman, N. (1988). A management skills workshop
  for chief residents. Journal of Medical
  Education 63, 442-446.
• Williams, R., Barrows, H.S., Vu, N.V., Verhulst,
  S.J., Colliver, J.A., Marcy, M., Steward, D.
  (1987). Direct standardized assessment of
  clinical competence. Medical Education 21,
  482-489.
• Willoughby, T.L., Gammon, L.C., Jonas, H.S.
  (1979). Correlates of clinical performance
  during medical school. Journal of Medical
  Education 54, 453-460.

59
Post-Test Sample Questions

• Do you know
• 
  
• Where to find the objectives for the Undergrad.
  medical programs?
• The objectives of the undergraduate programs in
  which you teach?
• Where to find the ITETS/FITERS for your
  undergrad. teaching?
  
• Your responsibility in completing undergrad.
  ITERS/FITERS?
• The difference between feedback and summative
  assessment?
  
• B. Are you comfortable in
  
• Your role as a teacher of Medical students?
  
• Giving feedback to students?
  
• Evaluating students?

60
Evaluation of the Resident/Fellow Workshop
onAssessment Evaluation Sample Questions

• Was the format suitable?
• Suggest changes for improvement.
• Was the content appropriate?
• What was not included that should have been?
• What was included that was unnecessary?
• Has this workshop helped you in your role as a
  teacher?
• What should follow this workshop to assist you
  become a better teacher/evaluator?
• What should follow this workshop to assist you
  become a better teacher/evaluator?
• What should follow this workshop to assist you
  become a better teacher/evaluator?