PerformanceBased Assessments

1
Performance-Based Assessments

• Jennifer Weber
• Arizona Department of Agriculture

2
Presentation Overview

• Overview of standardized multiple-choice exam and
  performance based exam formats
• Activity involving each type of exam
• Compare and contrast formats based on your test
  results
• Describe characteristics of pesticide handlers
• Draw conclusions about effectiveness of these two
  types of exams for the agricultural community

3
Assessing Knowledge. . .
through a hands-on activity
through multiple-choice questions on a written
test
4
Assessing Knowledge. . .
Hmm, now what is wrong with this picture?
Hmm, now which of these is the correct answer?
5
Assessing Knowledge. . .
Individually answering questions about how to
respond to pesticide-related situations
Working together in small groups to demonstrate
how to respond to a pesticide-related situation
6
O.K. class, lets begin.
7
Correct Answers

• Your Results?

8
Multiple-Choice Exam Activity

• You will receive a 5 question, written,
  multiple-choice exam
• Please read each question thoroughly and circle
  the correct answer
• You will have 3 minutes to complete the exam
• Do not discuss questions or answer options with
  others in the room
• I will provide you with the correct letter but
  will not explain the answer itself.
• You may begin your test

9
Performance-Based Exam Activity
Do we have a volunteer?
10
Test Format Preference
11
Discussion

• How many people in this session preferred the
  multiple-choice exam format?
• Why?
• How many people in this session preferred the
  performance-based exam format?
• Why?

12
Test Format Preference Quote

• Personally I like multiple choice as it is not
  open to interpretation and judgment of the person
  administering the exam and embarrassment, should
  I not have a clue about the answer. I have a 25
  chance of getting the correct answer with
  multiple choice.
• (Jack Peterson, personal communication, April 11,
  2007)

13
Advantages of Multiple-Choice Exams Over
Performance-Based Exams

• Easier to administer
• Less time consuming for department staff
• Can be scored quickly and easily if a Scantron
  form and scoring machine are used
• More economical
• Fewer logistics
• Can test more individuals in a short amount of
  time

14
Multiple-Choice Exams for Pesticide Applicators

• May be the ideal format for a fluent
  English-reader who grew up on a farm, has
  completed high school in the United States and is
  accustomed to taking multiple-choice exams
• Anyone without the aforementioned characteristics
  could struggle with the multiple-choice exam
  format.
• (Brenda Ball and Jack Peterson, personal
  communication, April 11, 2007)

15
Diversity of Agricultural Community

• Language skills
• Literacy levels
• Test taking experience
• Learning styles

• Educational background
• Work experience
• Age

16
What is an Effective Way to Measure Knowledge and
Skills?
The story of the three brothers
17
The story continues. . .
18
Language and Literacy

• The following table is from the 2005 NAWS report
  and displays the participants English speaking
  and reading abilities by place of birth and
  ethnicity data.
• English Speaking and Reading Ability, by Place of
  Birth and Ethnicity

19
Language and Literacy

• Transferring knowledge and terminology from ones
  native language to English can be difficult,
  especially when the same knowledge was acquired
  in a different context. In this type of
  situation, standardized exams can become tests of
  English reading and comprehension abilities and
  not of content knowledge and skills.
• (Carlon Ami, Relevance of Assessment and Culture
  in Evaluation (RACE) Conference. Tempe, AZ 2004)

20
Language and Literacy

• Thinking processes aside, non-English speakers
  often must learn additional words. For example,
  while all of them know that they must protect
  their eyes and skin during pesticide application,
  ocular and dermal are two more words that
  have to be learned to pass many exams.
• (Gerald Kinro, personal communication, May 6,
  2007)

21
Multiple-Choice Exams - Spanish

• Private applicator licensing exams are offered in
  Spanish in Oregon, Washington, and California.
• Results
• Oregon 55 pass rate among pesticide applicators
  who took the Private Applicators exam in English
  and a 3-5 pass rate among those who take the
  same exam in Spanish
• (Myron Shenk, personal communication, 2001)
• Monterey County, California 80 of the English
  test takers and 15 of the Spanish test takers
  passing the Private Applicator exam on their
  first attempt
• (Maria Vidauri, personal communication, May 6,
  2004)

22
Learning Styles and Test Taking Experience

• Native American students are holistic, visual
  learners, a contrast to the more logical and
  linear thinking skills that are often needed to
  excel on multiple-choice exams.
• They possess unique intelligences and a vast
  amount of knowledge and skills that cannot be
  captured through standardized exams.
• (Carlon Ami, RACE Conference. Tempe, AZ 2004)

23
Work Experience and Knowledge

• Becky Hines provides a test preparation course at
  WSU for landscapers
• Most have been working in the industry for
  several years and have vast amount of knowledge
  about pesticides and pest control
• Excel in group work, individual tasks, and
  practice quizzes during the three day test
  preparation course

24
Years of Experience Handling Pesticides

• Very few have passed the pesticide applicators
  licensing exam that is administered immediately
  following the course
• Its so frustrating! Theyre not stupid. They
  know the information but they just cant seem to
  pass these tests.
• (Becky Hines, personal communication, March 30,
  2007).

25
Equity Concerns

• The problem arises when these single scores are
  low despite ones experience and expertise in the
  area of study.
• Pesticide handlers who have limited English
  language and/or test taking abilities may
  struggle with the traditional multiple-choice
  format and be unable to display their true
  knowledge on the subject.

26
Educational Background and Test Taking Experience

• The multiple-choice tests are hard for the
  Spanish-speaking growers from Mexico. Where I
  grew up in Mexico, we didnt have multiple-choice
  exams or the Scantron form. Its hard to adjust
  to these types of tests.
• (Maria Vidauri, personal communication, 2004)

27
Possible Solutions

• One might think that a simple solution would be
  to present a course on multiple-choice test
  taking skills, to teach people how to use process
  of elimination tactics, pace themselves on timed
  tests, and fill in Scantron bubbles clearly and
  correctly
• While such a session might be helpful for some,
  efforts should really be made on finding the
  right tool or tools to assess what pesticide
  handlers already know or need to learn about
  pesticide safety

28
Alternative Form of Assessment

• The provision of an alternative form of
  assessment, such as performance-based exam, would
  allow test takers who might have difficulties
  passing a multiple-choice exam with an
  opportunity to demonstrate their knowledge and
  skills through authentic, hands-on activities,
  case studies, and question and answer sessions
  designed to reflect actual workplace situations.

29
Examples of Performance-Based Exam Programs

• Crane operators exam
• Business school admission requirement
• Drivers license driving demonstration
• Computer technician laboratory
• Clinical skills for nurses and veterinarians
• Air traffic controllers communication skills
  test

30
Additional Advantages of Performance-Based Exams

• A performance-based exam provides an opportunity
  to exchange safety information as well as assess
  individual knowledge and skill.
• In the long run, this assessment process has the
  potential to prevent hazardous situations at the
  work site.

31
Conclusion and Suggestions

• Develop and offer two testing formats
• a multiple-choice exam, which may be the desired
  format for people who have attended school in the
  U.S. and are accustomed to this testing format
• a performance-based exam that would enable test
  takers from various educational backgrounds with
  diverse language and test taking skills equal
  opportunity to demonstrate their knowledge on the
  safe and effective use of pesticides