Apply Handheld Devices to Formative Evaluation in Classroom

1
Apply Handheld Devices to Formative Evaluation in
Classroom

• Kuo-En Chang
• Department of Information and Computer Education
• National Taiwan Normal University

2
Introduction

• Nowadays, evaluations used by teachers are mostly
  summative evaluation, which focuses on evaluating
  students final learning results.
• Hyland (2000) indicated that 90 of students
  think that if feedback and learning advices can
  be provided from evaluations, the learning effect
  can be enhanced.

3
Introduction

• 92 of students wish for feedbacks that tell them
  how to improve their learning.
• 91 of students wish for evaluation feedback that
  explains their errors.
• The objective of formative evaluation is to
  provide these feedbacks for students in the
  purpose to enhance learning.

4
Introduction

• Applications of handhelds on formative evaluation
• Classroom Response System, CRS
• Classroom Wizard
• EduClick system

5
Introduction

• Disadvantages
• Not enough feedback, only right or wrong is
  available.
• Student only has the function of sending
  information.
• Student requires special equipment and this
  equipment only accommodates limited activities.
• No further exploring the application effect of
  handheld devices on the formative evaluation.

6
Introduction

• Objectives
• Develop a mobile system to assist teachers to
  implement formative evaluation activities in
  classroom teaching.
• Study empirically the effect on students
  progress of test scores after involving formative
  evaluation using handhelds (tablet PC or PDA) in
  classroom.
• Explore how mobile facilities and wireless
  environment are used in actual teaching.

7
Formative evaluation activities using mobile
devices

• Student evaluation
• Summary of student evaluation
• Student Score Distribution
• Student Score Ranking
• Evaluation Statistics
• Statistical Results of Each Question

8
Study 1

• Subjects
• A high school English teacher and two classes of
  first grade students.
• The experimental group has 42 students. The
  control group has 39 students.
• The experimental group is the class with poorer
  English performance, and the control group has
  better English performance.
• This research expects to improve the performance
  of the English learning of the experimental group
  through experimental treatment.

9
Study 1

• Research design
• Study 1 adopts the nonequivalent group pre- and
  post-test design to compare the English scores of
  the two groups between two tests.
• The independent variable is whether use
  formative evaluation software in classroom
  teaching.
• The dependent variable is English score in the
  second exam.
• The covariate is English score of last semester.

10
Study 1

• Experimental procedure
• The experiment was implemented over a four-week
  long, which each week has a 50 min-long session
  of English grammar teaching.
• The facility used in Study 1 is tablet PCs and
  wireless network. Each student of experimental
  group has a tablet PC.
• The teacher used the last 15 minutes for every
  session to conduct the formative evaluation
  activities.
• Both groups were implemented with identical
  teaching.

11
Study 1

• Results
• After the result of ANCOVA excluded the effects
  of English score of last semester, within-class
  comparison reaches the significant level (F4.86,
  p lt .05).
• after Study 1, the result shows a significant
  difference on the English scores of the second
  exam (Adjusted means experimental group 66.67,
  control group 71.02).

12
Study 1

• Observation on the teacher
• After evaluation, the teacher discussed the
  questions one by one based on the statistical
  results to determine what should be reinforced in
  the next class.
• In the early two weeks of experiment, the
  evaluations like quizzes, not formative
  evaluating.
• The teacher tried to send the evaluation results
  back to students tablet PCs, but this attempt
  caused students to look down to their computers.
• The teacher only conducted classroom discussion
  when the student counts of wrong answers with
  long answering time and high ineffective prompts.

13
Study 1

• Observation on the students
• Students sitting at the back of classroom were
  distracted with the tablet PCs and its internet
  functions.
• Some students used tried and error to answer the
  questions and behaved in a brash way.
• Students wish to know their performance instantly
  after the evaluation.
• The students not concentrated enough on the
  evaluation activities.
• Many students who made wrong answers spent very
  little time on thinking of answering the
  questions.

14
Study 2

• Improvements
• Change hardware connection to reinforce the
  effectiveness of the evaluation.
• Connected serially a new AP and moved the server
  to the same area network (that is, in an
  Intranet, not Internet).
• Tablet PCs used in Study 1 were bulkier
  therefore, they require 2 to 3 minutes for
  startup. we replaced the equipments with PDAs.
• Solving the problem that the handhelds diverted
  students attentions to the teachers lecture.

15
Study 2

• Improvements
• Solving the problem that the handhelds diverted
  students attentions to the teachers lecture.
• We only require that students turn on PDAs for
  evaluation.
• When the teacher was lecturing, a projector was
  used to project the questions and students
  answers on the wall for discussions of the
  evaluation results.

16
Study 2

• Subjects
• same as Study 1
• Experimental design
• Compare the scores of the third exam with the
  scores of the first and second exams.
• A 2 (Groups, experimental and control groups) ? 3
  (Exams, the first, second, and third exams)
  two-way mixed design was used.
• The dependant variable is students scores of the
  third exam.

17
Study 2

• Results
• After treatment in Study 2, within-class
  comparison did not show a significant difference
  on the scores of the third exam (the adjusted
  average values of control and experimental groups
  are 68.95 and 67.41).
• after the Study 2, the two groups did not show
  significant difference in grammar scores of third
  exam (Adjusted means of experimental and control
  groups are 12.99 and 13.17).

18
Study 2

• Results
• Both scores of the first exam or the second exam
  in the experimental group and control group have
  significant difference and the students of the
  experimental group perform poorer.
• There is no significant difference between both
  groups in the third exam.
• The gap in the scores of the experimental group
  and control group was closing up, and to the
  third exam, the gap became almost non-existent

19
Study 2

• Observation on the teacher
• The teachers first viewed students score
  distribution (Figure 3) to understand roughly the
  students answering status, the teacher then
  viewed the detail answer distribution of each
  question (Figure 6) to prepare for individual
  question discussion in the next class.
• The teacher paid particularly attention to
  students answering time.

20
Study 2

• Observation on the teacher
• After the post-evaluation discussion, the teacher
  looked at students score ranking.
• The evaluation software is capable of displaying
  students answering statuses in real time
  therefore, the teacher walked around the
  classroom during the evaluation to observe the
  students.

21
Study 2

• Observation on the Students
• Compared to Study 1, students, who used PDAs,
  became more proactive. All students were willing
  to participate in the evaluation activities and
  interacted with the teacher better.
• The feedback function of the evaluation software
  enabled students to concentrate much on answering
  questions. The ratio of students right answers
  was higher than that of Study 1.
• When students knew that PDA evaluation activities
  were to be hold in the class, they became more
  concentrated.

22
Study 2

• Observation on the Students
• In the post-evaluation discussion, some students
  sitting at the two sides and the back of the
  classroom would play games in the PDAs and
  interacted less with the teacher.
• The startup time for PDAs is short (0.8 second).
  Evaluations could be conducted without any delay.

23
Conclusions

• This software improves the shortcomings of other
  evaluation systems, adds the feedback function in
  the formative evaluations, and exploits the
  effects of computation functions in handhelds.
• The experimental results showed that
  differentials between the experimental group and
  the control group were becoming minimized.
• The teacher and students became more interactive
  after implementation of the evaluation
  activities, and students became more concentrated
  in the lecture.

24
Figure 1
25
Figure 2
26
Figure 3
27
Figure 4
28
Figure 5
29
Figure 6