Improving assessment quality using electronic testing

1
Software Engineering Education and Reverse
Engineering
Improving assessment quality using electronic
testing
Goce Armenski, M.Sc
Institute of Informatics - FNSUniversity Ss.
Ciryl and Methodious Skopje, Macedonia
2
1.
Content
CONTENT
- INTRODUCTION
- ?OTIVATION AND GOALS
- ????? NEW SYSTEM FOR ELECTRONIC TESTING
- BASIC CONCEPTS OF THE SYSTEMS FOR ??ESTING
- ?RCHITECTURE, CONCEPTS AND FUNCTIONALITY OF
??EST
- APLICATION OF ??EST
- RESULTS
- CONCLUSION
Goce Armenski
3
2.
Introduction
1. INTRODUCTION
Changes in the way people live influenced by-
Globalization- Increased meaning of knowledge-
Information and communication revolution
Industrial society ? Information
societyInformation society ? Knowledge
society
50 of working skills are getting old in 3-5 years
Goce Armenski
4
3.
Introduction
Changes in educationTeacher ? Student centered
education
5
4.
Introduction
Changes in the education (lot more students)-
Lecturing process- Delivery of materials-
Knowledge assessment

• What are the assessment functions?
• Check the successfulness in achieving learning
  goals
• Provide feedback
• Improve learning process

Keys achieved 14 improvement of student results
when assessing their knowledge once a week
instead once a month. Pikunas and Mazzota also
mention improvements in results of 10, when
tests are delivered every week instead every 6
weeks
Goce Armenski
6
5.
Introduction
Institute of Informatics (FNS)- every year 150
students- 4 questions (assignments) x 150
students 600 questions- 5 min for marking a
question- 3000 min / 50 hours
How long the marking will take???
Ontime feedback???
Personalized feedback???
Objectivity???
Goce Armenski
7
6.
Motivation and goal
2. MOTIVATION AND GOAL
Development of a system for assessment of large
number of students (more that 150) every month,
which can be used in distance learning, as well
as any other form of knowledge or skill
assessment.

• Motivation for test construction
  Different tests for ecery student with same
  weight Lowering the possibility for
  memorizing

• Motivation for simple data entry
  Possibility for bulk entering of data

• Motivation for reports creation
  Statistical analyses of gathered information

Goce Armenski
8
7.
Motivation and goal
Goal
Focus of the researchTo implement a system for
computer based assessment and evaluate the
results from its use Global goalIdentify the
influence of this system to the teachers and
students Main goalIs computer based
assessment more effective and objective than the
traditional assessment
Goce Armenski
9
8.
??est concepts and functionality
2. ??EST CONCEPTS AND FUNCTIONALITY
Basic concepts of systems for eTesting
QuestionMark, BlackBoard, WEB CT, Top Class,
EduSystem
- Question bank, - Algoritms for test creation,
- Systems for data presentation, - Result
reports
Goce Armenski
10
9.
??est concepts and functionality
Question bank
Database of unique questions, with needed
characteristics for simple selection during the
test construction.some of the parameters are
created dynamicaly

• Standards for question bank development.-
  Exchange of questions of university level

Goce Armenski
11
10.
??est concepts and functionality
Fig. 2 Types of questions

• Fixed answers questions (objective) -
  multichoice - short entry - questions
  with graphical selection (hotspot)- Free text
  answers - programming code as answer -
  essay answer

Goce Armenski
12
11.
??est concepts and functionality
Algoritms for test creation
Fig. 3 Algoritms for test creation
Main difference between them in the adaptation to
the characteristics of the person which knowledge
is assessed
Goce Armenski
13
12.
??est concepts and functionality
Systems for data presentation
Content depending on the characteristics of the
monitor and the network
Marking and reporting
In which way the system shows the results?-
shows true answers?- the result is in points or
percents?- negative marking???
Goce Armenski
14
13.
??est concepts and functionality
??est Technology
Web based sollutions VS desktop based sollutions

• Web based application- Active Server Pages
  (ASP)- JavaScript- SQL Server 2000- NT Server
  and Win 2000 compatible- IIS 4.0 or newer

Web Browser (Netscape 4.x or Internet Explorer
4.x, and above)
JScript
Email (SMTP)
SQL Server Access
IIS 4.0
ASP
Win 2000 Server
Goce Armenski
15
14.
??est concepts and functionality
??est Architecture
Fig. 4 Three tier architecture of the eTesting
system
Goce Armenski
16
15.
??est concepts and functionality
????? Concepts

• Types of users- Course organisation- Types of
  questions- Test creation algoritm- System for
  data presentation- Marking and reporting

Goce Armenski
17
16.
??est concepts and functionality
Types of users
Fig. 5 Types of users
Goce Armenski
18
17.
??est concepts and functionality
Course organisation

• learning objects- three structure

Fig. 6 Course organisation
Goce Armenski
19
18.
??est concepts and functionality
Types of questions

• Multichoice questions (choose one of many, choose
  many of many, yes/no answers)
• Short entry answer (text or numerical)
• Essay answer.

• Questions can have pictures or graphs in the text
  or offered answers

Goce Armenski
20
19.
??est concepts and functionality
Multichoice questions
Fig. 7 Choose one of many
Fig. 8 Choose many of many
Goce Armenski
21
20.
??est concepts and functionality
Short entry answers
??. 9 Short entry
Goce Armenski
22
21.
??est concepts and functionality
Questions with essay answer
Fig. 10 Essay answer

• these answers are not evaluated by the system-
  lowering the objectivity- Project Essay Grade
  (PEG)- Intelligent Essay Assessor (IEA),-
  Erater,- Bayesian Essay Test Scoring System
  (BETSY).

Goce Armenski
23
22.
??est concepts and functionality
Test creation algoritm

• dynamic linear tests (fixed number of questions)

System for data presentation

• adjusted to Web standards (800x600)-
  possibility for picture and graph display

Marking and reporting

• results are shown at the end of the test-
  negative marking

Goce Armenski
24
23.
??est concepts and functionality
Statistical data analyses

• Identification of content which is not well
  presented
• Personalized feedback to students
• Identification of week questions which need to be
  revised before used again
• Identifying the individual weeknesses of students.

Goce Armenski
25
24.
??est concepts and functionality
Inportant statistical data about questions

• difficulty how difficult a question is
• discrimination how well the question
  diferentiate good from bad students
• guessing how students answer the corect answer
  without knowing it
• Different achievements what are the results of
  diferent user groups.

Goce Armenski
26
25.
Application of eTest
4. APPLICATION OF ??EST
- Integration in the process of learning -
Controled learning
Fig. 12 Way of passing the learning objects

• Succesfull strategy??
• all questions
• N questions in a row
• N wright questions
• 3 wright questions in a row

Statistical analyses of the user activities
Goce Armenski
27
26.
Results
5. RESULTS
Prctical Implementation

• Institute of Informatics , FNS (2001)
• ?.D. Mobimak (2002)
• UNDP (2003-2004)

The use of technology in education is very
dependent of the organization - logistics
- socal changes Syncronisation with other
systemsTechnical infrastructure
Goce Armenski
28
27.
Results
Data gathering
Institute of Informatics, FNS (january 2001)
- 26 courses - 12391 questions - 589
scheduled assessments - 9861 generated tests
Does eTesting provides more effective and more
objective assessment compared to the traditional
forms, and does it help the learning process
Goce Armenski
29
28.
Results
Cost savings
Decreasing the use of paperDecreasing the use of
print materialDecreasing the time for conducting
the assessment
Technical infrastructure Software development
and maintenanceTraining of administratorsTechnic
al personal
Goce Armenski
30
29.
Results
Teacher perspective
Creating of question bank is time consuming -
more than 1300 per courseTime saving after the
creation
Student perspective
Questionary with 10 questions - Questions with
Likert type of answers with 5 values - 236
students were enroled in the survey
Goce Armenski
31
30.
Results
Question 1
Goce Armenski
32
31.
Results
Question 2
Goce Armenski
33
32.
Results
Question 3
Goce Armenski
34
33.
Results
Question 4
Goce Armenski
35
34.
Results
Question 5
Goce Armenski
36
35.
Results
Question 6
Goce Armenski
37
36.
Results
Question 7
Goce Armenski
38
37.
Results
Question 8 and 9
Goce Armenski
39
38.
Results
Question 8 and 9 cont
Goce Armenski
40
39.
Results
Question 10
Goce Armenski
41
Results
40.
Summary of the results
??????? N ??. ??. Std
The use of the system for electronic testing is 235 3,7149 0,8865
The electronic testing is __________ compared to the traditional one 233 3,7296 0,7368
I prefere assessment using the system for electronic testing, than the traditional one 236 4,3686 0,9154
Marking on the system for electronic testing is objective (same for all) 233 4,3305 0,8748
The use of the module for online learning with often knowledge assessment is helping with the material 236 4,3814 0,7428
Using the module for online learning with often knowledge assessment helps me achive more knowledge 235 4,0128 0,8986
I recommend use of the system for electronic testing for assessment in other courses 233 4,4378 0,8288
42
41.
Conclusion
6. CONCLUSION
System implementation

• Dependent of the institution in which it is
  implemented

Improvements in the assessment process

• Possibility to assess knowledge of more than 150
  students
• Immediate feedback
• Lowering the subjectivity
• Self testing possibility before the official
  assessment
• Analyses of the gathered data
• Possitive effects on the learning process
• Incresed security???

Combining this method with other method of
assessment can give great results
Goce Armenski