Dr' Nitza Davidovitch

1
(No Transcript)
2
Dr. Nitza Davidovitch Dr. Nitza Davidovitch
holds a doctorate degree, awarded by Bar Ilan
University, focused on the developmental trends
of regional colleges and their impact on the
higher education system in Israel. Dr.
Davidovitch currently heads the Department of
Academic Development at Israel's largest public
college, the Academic College of Judea and
Samaria. Dr. Davidovitch is active in community
services in issues relating to the Holocaust and
Jewish identity, cultivating the cultural history
of Jewish sects, and moral education..
3
2ND ASIA-PACIFIC EDUCATIONAL INTEGRITY
CONFERENCE VALUES IN TEACHING, LEARNING
RESEARCH. 2-3 DECEMBER ? 2005
4
How to Ensure Reliable Assessments of Students'
Achievements on Exams? A Case Study Application
of Statistical Tools in Computerized Marking of
Multiple-Choice Exams and Questionnaires Nitza
Davidovitch d.nitza_at_ycariel.yosh.ac.il .
5

• Emphasis on Quality
• Previous approaches to quality in education
• Number of graduates
• Number of drop-outs
• Number of complaints
• Measurement of client service quality costs

6

• Recent approaches to quality in education
• Quality drives market share and profitability
• Quality is a key factor in client's trust in
  service delivery
• Quality leads to repeat purchases
• Demand the best of yourself and meet your own
  expectations
• Everyone is involved in quality
• A collective approach to achieve improvement

7

• Efforts to achieve quality progressed on several
  levels
• Improve quality by improving processes (e.g. TQM)
• Self-assessment
• Development of performance scores (e.g.
  benchmarking, balance scorecard)
• The common denominator QUANTIFY AND MEASURE

8

• To improve quality quantify
• "When you can measure what you are speaking
  about, and express it in numbers, you know
  something about it but when you cannot measure
  it, when you cannot express it in numbers, your
  knowledge is of a meager and unsatisfactory
  kind."
• Lord Kelvin, 1883

9

• Motives
• Greater awareness
• Political pressure
• Financial pressure
• As organizations grow, their ability to control
  quality declines
• Globalization measures for accreditation

10

• Aims of the Computerized Testing Program
• Measure the level of student achievement
• Identify students failing to achieve mastery of
  content
• Provide teachers with diagnostic information
• Assist school systems in identifying strengths
  and weaknesses in order to establish priorities
  in planning educational programs.

11

• From the APA CODE OF FAIR TESTING PRACTICES IN
  EDUCATION
• "Fairness is a primary consideration in all
  aspects of testing. Careful standardization of
  tests and administration conditions helps to
  ensure that all test takers are given a
  comparable opportunity to demonstrate what they
  know and how they can perform in the area being
  tested. Fairness also extends to the accurate
  reporting of individual and group test results.
  Fairness is not an isolated concept, but must be
  considered in all aspects of the testing
  process."

12

• Four critical areas of action to ensure fair
  testing
• Developing and Selecting Appropriate Tests
• Administering and Scoring Tests
• Reporting and Interpreting Test Results
• Informing Test Takers

13

• Quality Control Assurance at the Academic
  College of Judea and Samaria (ACJS)
• Students complete Instructor Assessment Surveys
• Workshops and training for instructors
• Integration of computerized services to support
  instruction
• ACJS website portal for instructors
• Training in academic course development
• Database of videotaped courses
• Assistance in publishing teaching materials
• Research grants
• Collaboration efforts with institutions overseas
• Participation in national and international
  conventions AND
• The ACJS Project Computerized Multiple Choice
  Exam System

14

• Aims of the ACJS Project
• Improve quality of service delivery to students
• Improve reliability of grades
• Improve fairness of the marking process
• Provide feedback for instructors on
• Quality of exams
• Quality and effectiveness of instruction

15

• Direct and Indirect Clients
• Students data on individual achievements, by
  item class rank
• Instructors indices of students' assessments
  exam reliability, average scores, SD, range,
  percent passed, by class and group (i.e. tutor
  group, retest)
• Course and curricula developers reliable index
  of course quality
• Administrative staff administration of
  students' scores by course, by instructor

16

• Our Partners
• Tel Aviv University
• Developed the method in the Faculty of Medicine
• For the last 20 years Dr. Neta Notzer was in
  charge of exam analysis at Tel Aviv University.
• Today commercial companies have assumed this
  function

17

• Image net
• World leader in sophisticated image processing
  based on optical scanning and video systems.
• Supplier of advanced technological solutions to
  large and small companies that lead to
  significant savings in time and costs, improve
  performance efficiency and reliability.
• Provides the following services to the ACJS
• Software and hardware for precise data entry
• System operators high customer-service
  orientation high level of confidentiality
• Ongoing support

18

• Acuform
• Developed in collaboration with the Chief
  Scientist
• Automatic scanning and identification of forms
  containing handwriting, barcodes, printed text
  and marks
• Automatically deciphers/corrects errors,
  completes missing data and issues reports of raw
  data

19

• Instructors' Workshop on Drafting Multiple Choice
  Exams
• How to structure the exam
• How to formulate questions
• How to administer multiple-choice exams
• Introduction to statistical analysis of exams and
  criteria
• How to draw conclusions from statistical reports
  
• How to analyze students' achievements
• How to analyze course instruction
• How to analyze exam quality

20

• Exam Review and Analysis Software
• Reads students' answers and identifying data
  error-free and accurate
• Graphic and verbal representation of analyses
• Multiple choice question bank, by subject

21

• Computerized Exams ensure that exams are
• Fair Produce valid and reliable test scores for
  all students, regardless of race, national
  origin, gender or disability
• Reliable Scores - (accuracy of assessment) test
  scores ensure consistency and lack of
  measurements errors
• Valid Interpretation-- Measure the same
  knowledge of content for all students who take
  the test
• Convenient to use
• Cost-effective
• Provide rapid results
• Provide conclusions that are fair and accurate
  for all students

22

• Exam reports provide verbal and graphic data
• Exam reliability measurement errors and
  consistency
• Exam items -
• Dispersion of responses
• Discriminatory power
• Level of difficulty
• Combinations (average scores by level of
  difficulty)
• Students' achievement levels
• Individual reports by exam item
• Average
• Class rank

23

• Some results of our recent QA efforts
• Finding No link between number of items on exams
  and reliability and validity scores
• Conclusion Instructors have yet to learn to
  improve the quality of their exams.
• Action New training sessions scheduled for
  instructors.

24

• Cross-checking instructors' exam quality scores,
• and
• instructors' assessments scores from the student
  survey
• We found no link found between instructors'
  popularity scores and students' final grades.
• If anyone was worried - difficult exams don't
  detract from instructors' popularity!

25

• Multiple choice questions and standardized tests
  may provide a solution to the problem of fairness
  
• --BUT--
• Multiple choice exams do not necessarily
  constitute authentic assessment methods that
  reflect the skills we wish our students to
  develop.

26

• We need to re-conceptualize testing to find more
  effective ways to link successful test
  performance - - to - - student effort and
  effective instruction

27

• Criticism against standardized testing
• Is unable to test a students' true ability in
  that they are not free to speak their minds
• Narrows down what students are being taught

28

• "Not everything that counts can be counted, and
  not everything that can be counted counts."
  Albert Einstein
• "I don't think there's a way to build a multiple
  choice question that allows students to show what
  they can do with what they know." Roger Farr,
  Professor of Education at Indiana University
• "Standardized tests can't measure initiative,
  creativity, imagination, conceptual thinking,
  curiosity, effort, irony, judgment, commitment,
  nuance, goodwill, ethical reflection, or a host
  of other valuable dispositions and attributes.
  What they can measure and count are isolated
  skills, specific facts and functions, the last
  interesting and least significant aspects of
  learning" Bill Ayers

29

• Thank you very much for your time and attention.