Assessment, Evaluation and All that Jazz

1
Assessment, Evaluation and All that Jazz

• OR
• The Squeaky Wheel
• Gets the Grease

Craig Scanlan, EdD, RRT, FAARC Professor,
MSHS/PhD Programs UMDNJ-SHRP (Newark)
2
(No Transcript)
3
(No Transcript)
4
(No Transcript)
5
(No Transcript)
6
(No Transcript)
7
Sallys Lesson Learned

• Good evaluation should be based on explicit
  criteria that are shared with students in advance

8
Pretest

• The most common question students ask is __
• Dont teach to the _____
• Whats central to evaluation? _____
• Dont use a sledge hammer to swat a ____
• Dont loose the forest through the _____
• Columbus discovered America in ____
• Is there any new thing under the sun?

9
The Most Common Question Students Ask?

• Is this going to be on the test?

Lesson AssessmentDrives Learning
Brissenden, G. Slater, T. Assessment primer.
In College Level One (CL-1) Team. Field-tested
learning assessment guide. Available at
http//www.flaguide.org.
10
Dont Teach to the _____

• Wrong test!

Lesson A valid reliable test can be the basis
for teaching and learning
11
Whats Central to Evaluation?
e
ation
valu
E
Lesson Evaluation Requires MakingValue
Judgments
12
Dont Use a Sledge Hammer to Swat a ___
Fly
Lesson Use the right tool for each job
13
Law of the Instrument Fallacy

• Molding the evaluation problem to fit a given tool

Give a small boy a hammer and he will find that
everything he encounters needs
pounding Kaplan, A. (1964). The conduct of
inquiry. San Francisco Chandler Sharp
14
Dont Loose the Forest Through the _____
Trees
Lesson Always keep the big picture in mind,
i.e., professional competence
15
Fallacy of False Quantification

• The tendency to focus on those competencies or
  abilities that are easiest to measure

16
Important Attributes and Characteristics Needed
by Future Respiratory Care Practitioners
O'Daniel C, Cullen DL, Douce FH,et al. (1992).
The future educational needs of respiratory care
practitioners a Delphi study. Respir Care, 37,
65-78.
17
Columbus discovered America in ____
a Boat
Lesson Allow for and encourage divergent (vs.
convergent) thinking
18
Is There Any New Thing Under the Sun?
NO! (Ecclesiastes 19)
Tyler, R. W. (1950). Basic principles of
curriculum and instruction. Chicago University
of Chicago Press.
19
Ancient History

• Proficiency testing likely began as early as 2200
  B.C., when the employees of the Chinese emperor
  were tested to determine their fitness for
  serving the emperor.

Gregory, R. (1992). Psychological testing,
History, principles, and applications. Boston
Allyn and Bacon.
20
Assessment vs. Evaluation
From Apple, D.K. Krumsieg. K. (1998). Process
education teaching institute handbook. Pacific
Crest
21
Classroom Assessment

• A particular type of assessment designed to
  several related purposes
• motivating and directing learning
• providing feedback to students on their
  performance
• providing feedback on instruction and/or the
  curriculum
• ensuring standards of performance/progress are
  met

22
Classroom Assessment

• Help us answer the following questions
• To what extent are my students achieving the
  stated goals?
• How should I allocate class time for the current
  topic?
• Can I teach this topic in a more efficient or
  effective way?
• What parts of this course/unit are my students
  finding most valuable?
• How will I change this course/unit the next time
  I teach it?

23
Classroom Assessment

• For students, learner assessment helps answers
  the following questions
• Do I know what my instructor thinks is most
  important?
• Am I mastering the course content?
• How can I improve my studying?
• How am I progressing in this course?

24
Examples of Classroom Assessment Techniques

• Minute Paper/Muddiest Point
• Think-Pair-Share
• Student Recaps
• Student- generated Test Questions
• Exam Evaluations
• Application Ideas

Angelo, T.A. Cross, P.K. (1993). Classroom
Assessment Techniques (2nd ed.). San Francisco
Jossey-Bass
25
Impact of Classroom Assessment Techniques

• High quality formative assessment has a powerful
  impact on student learning
• Black and William report effect sizes on
  standardized tests of between 0.4 and 0.7
• Formative assessment is particularly effective in
  improving the performance of lower achieving
  students

Black, P, William, D. (1998). Inside the black
box Raising standards through classroom
assessment. Phi Delta Kappan, 80(2).
26
Minute Paper/Muddiest Point

• "What is the most important point you learned
  today?"
• ???
• ???
• ???
• ???

• "What point remains least clear to you?
• ???
• ???
• ???
• ???

27
Evaluation
From Apple, D.K. Krumsieg. K. (1998). Process
education teaching institute handbook. Pacific
Crest
28
What To Evaluate?

• Professional
• Competency

29
Health Professional Competency

• 1960s (Late) Early Concerns
• Back to Basics Educational Movement
• US DHEW recommends periodic MD re-examination
• 1970s Behavioralism/Competency-Based Education
• 1980s Mandatory Continuing Education
• 1990s Demand for Better Regulatory Controls
• Reports (Pew Health Professions Commission, IOM)
• QA/CQI and Consumerism Movements
• JCAHO Competency Assessment Standards (1997)
• State/Provincial Laws e.g., Alberta Health Prof
  Act (1999)
• 2000s Mandatory Skills Assessment, e.g., NBME

30
What is Professional Competency?
31
Much descriptive work has defined competencies
and outlined processes that can be used in
creating competency-based curricula. Assessment
tools to evaluate competence have received less
attention. The creation of tools that are valid,
reliable, and predictive of future success is our
immediate challenge.

• Carraccio, C, Wolfsthal, SD, Englander, R,
  Ferentz, K, Martin, C. (2002). Shifting
  paradigms From Flexner to competencies. Academic
  Medicine 77, 361-367.

32
Newble, DI, Dawson, B, Dauphinee, WD, et al.
(1994). Guidelines for assessing clinical
ompetence. Teach Learn Med, 6, 213-20.
33
In-Vivo Performance In-Vitro
Performance Competence
Knowledge
Miller, GE. (1990). The assessment of clinical
skills/ competence/ performance. Acad Med,
65(Suppl)S63-7.
34
Tools vs. Levels
Wass, W, Van der Vleuten, C, Shatzer, J and
Jones, R. (2001). Assessment of clinical
competence. Lancet, 357, 945-949.
35
Required Attributes of an Evaluation Process

• Reliability
• Validity
• Acceptability
• Feasibility
• Educational Impact

Van der Vleuten, C.P.M. (1996). The assessment
of professional competence Developments,
research and practical implications. Adv Hlth Sci
Educ,1, 4167
36
Assessment Attributes by Level
Acceptability
Feasibility
Ed Impact
Reliability
Validity
DOES
SHOWS HOW
KNOWS HOW
KNOWS
37
The Evaluation Toolbox

• Knows/Knows How
• Written Tests of Knowledge
• Oral/Online QA
• Shows How
• Process Evaluation (OSCE)
• Product Evaluation (e.g., Case Study, Report,
  Paper)
• Does (Provides Service or Care)
• Performance Observation
• Records Review/Audit

38
Development of Good Written Tests/Exams

• Reliability
• Validity
• Content Validity
• Criterion-Related Validity
• Concurrent
• Predictive
• Conclusion Validity

39
Conclusion Validity

• How Do You Establish Cut Scores?
• Arbitrary, e.g. 70-80-90
• Grading on a Curve, e.g. Percentiles
• Formal Subject Matter Expert Review, e.g.,
  Angoff, Ebel, Nedelsky methods
• Predictive Validity

40
Oral/Online QA Quantitative Rubric
41
Oral/Online QA Qualitative Rubric

• A portion of your course grade is based on the
  quantity and quality of your involvement in
  posted discussion topics. Specific consideration
  is given to the following
• Continuous involvement throughout course
  (participate at least once in all posted
  discussion topics)
• The accuracy of statements made in response to
  posted questions or topics.
• The relevance of statements made to the topics
  under discussion
• The logic of the statements or arguments raised
  in discussion
• The cognitive levels of the statements or
  arguments raised in discussion
• The ability to build on other students
  statements or arguments.
• The degree to which dialogue is based on unit
  assignments
• The willingness to listen to opinions and views
  that differ from ones own.
• The willingness to change or consider other
  views, when one's arguments have been found
  faulty

42
Product Specification/Evaluation

• Prepare an executive summary of your research
  study that includes the following
• Research question or hypothesis (sentence or two)
• Description of subjects (sentence or two)
• Description of the intervention (sentence or two)
• Description of data analysis procedures (one
  paragraph)
• Selection/justification of your basic statistical
  approach
• Test(s) of assumptions underlying the statistical
  approach, to include your multigroup scatterplot
  with accompanying narrative
• Description of findings (paragraph or two)
• The final ANCOVA table (simplified!)
• A table of the 'raw' and adjusted post-test means
  and their SEs
• An accompanying narrative statement summarizing
  the results of the ANCOVA
• A summary of your conclusions, i.e., is the new
  rehab treatment better than the old?

43
Product Specification/Evaluation
Criteria for Students
Grade Calculator for Faculty
44
Observational Evaluation of Performance (Example)
Patient Care Resident provides compassionate
care that is effective for the promotion of
health, prevention, treatment, and at the end of
life
Rating
45
Observational Evaluation of Performance (Example)
46
The Special Problem of Assessing AffectorThat
student has an attitude problem
Gordon, M.J. (1978). Assessment of student
affect A clinical approach. In Morgan, M.K.
Irby, D.M. (1978). Evaluating Clinical Competence
in the Health Professions, St. Louis, CV Mosby.
47
Take-Home Points

• Assessment evaluation require criteria
• Assessment drives learning
• Evaluation requires value judgments
• Assessment/evaluation should not merely be done
  to students, but also for students
• For evaluation, focus on the big picture, i.e.,
  graduate competence
• Select and properly apply the right tool for
  each specific job
• The higher the stakes, the greater your
  responsibility
• Develop a consistent approach to assessing affect