Interoperability: future-proofing CAA

1
Interoperability future-proofing CAA
Heriot-Watt University 6th February, 2002

• Niall Sclater CETIS
• Manager, Educational Systems
• University of Strathclyde

2
Why do I need to know about QTI?

• Im authoring a bank of questions in my subject
  area
• Im involved in the selection of a CAA system for
  my institution
• I have a general interest in online learning and
  assessment

3
The basics

• IMS QTI is a means for exchanging assessment
  information
• It uses XML
• 2 key parts of the spec
• Assessment Section Item
• Results Report
• Also QTI lite
• Can be implemented separately

4
(No Transcript)
5
Response types

• Multiple choice
• True false
• Multiple response
• Image hot spot
• Fill in the blank
• Select text
• Slide
• Drag object
• Drag target
• Order objects
• Match item
• Connect the points

6
The concept of rendering

• The same item can be rendered in different ways
  to allow for different
• delivery platforms
• technical capabilities
• participant challenges (eg visual, audio,
  physical etc)

7
Rendering example
8
Basic response render types

• Response types
• logical identifier
• xy coordinate
• string
• number
• logical group

• Render types
• choice
• hot spot
• fill in blank
• slider
• object

9
True false example
10
True false example XML
ltquestestinteropgt ltqticommentgt This is a
simple True/False multiple-choice example. The
rendering is a standard radio button style. No
response processing is incorporated. lt/qticomment
gt ltitem ident"IMS_V01_I_BasicExample001a"gt
ltpresentation label"BasicExample001a"gt
ltmaterialgt ltmattextgt Paris is the
Capital of France ? lt/mattextgt
lt/materialgt ltresponse_lid
ident"TF01" rcardinality"Single" rtiming"No"gt
ltrender_choicegt
ltresponse_label ident"T"gt
ltmaterialgtltmattextgt True lt/mattextgtlt/material
gt lt/response_labelgt
ltresponse_label ident"F"gt
ltmaterialgtltmattextgt False lt/mattextgtlt/materia
lgt lt/response_labelgt lt/render_choi
cegt lt/response_lidgt
lt/presentationgt lt/itemgt lt/questestinteropgt
11
Response processing
ltitemgt ltresprocessinggt ltoutcomesgtltdecvar/gtlt/ou
tcomesgt ltrespcondition title"Correct"gt ltcondit
ionvargt ltvarequal respident"TF01"gtTlt/varequalgt
lt/conditionvargt ltsetvar action"Set"
gt1lt/setvargt ltdisplayfeedback feedbacktype"Respo
nse" linkrefid"Correct"/gt lt/respconditiongt lt/res
processinggt ltitemfeedback ident"Correct"
view"Candidate"gt ltmaterialgtltmattextgtYes, you
are rightlt/mattextgtlt/materialgt lt/itemfeedbackgt
ltitemgt
12
Multiple choice example
13
Multiple choice example XML
ltquestestinteropgt ltitem title"Standard MC Item"
Ident"IMS_V01_I_BasicExample002"gt
ltpresentation label"BasicExample002"gt
ltmaterialgtltmattextgtWhich one of the listed
standards committees is responsible
for developing the token ring specification ?
ltmattextgt lt/materialgt
ltresponse_lid ident"MC01" rcardinality"Single
" rtiming"No"gt ltrender_choice
shuffle"Yes"gt ltresponse_label
ident"A"gt ltmaterialgtltmattextgt
IEEE 802.3 lt/mattextgtlt/materialgt
lt/response_labelgt ltresponse_label
ident"B"gt ltmaterialgtltmattextgt
IEEE 802.5 lt/mattextgtlt/materialgt
lt/response_labelgt ltresponse_label
ident"C"gt ltmaterialgtltmattextgt
IEEE 802.6 lt/mattextgtlt/materialgt
lt/response_labelgt ltresponse_label
ident"D"gt ltmaterialgtltmattextgt
IEEE 802.11 lt/mattextgtlt/materialgt
lt/response_labelgt ltresponse_label
ident"E" rshuffle"No"gt
ltmaterialgtltmattextgt None of the above.
lt/mattextgtlt/materialgt lt/response_label
gt lt/render_choicegt lt/response_lidgt
lt/presentationgt lt/itemgt lt/questestinteropgt
14
Response processing
ltitemgt ltresprocessinggt ltoutcomesgt ltdecvar
vartype"Integer" defaultval"0"/gt lt/outcomesgt lt
respcondition title"Correct"gt ltconditionvargt
ltvarequal respident"MCb_01"gtAlt/varequalgt lt/con
ditionvargt ltsetvar action"Set"
gt1lt/setvargt ltdisplayfeedback feedbacktype"Respo
nse" linkrefid"Correct"/gt lt/respconditiongt lt/res
processinggt ltitemfeedback ident"Correct"
view"Candidate"gt ltmaterialgtltmattextgtYes, you
are right.lt/mattextgtlt/materialgt lt/itemfeedbackgt
ltitemgt
15
Multiple response example
16
Image hot spot example
17
Fill in blank (string) example
18
Numerical example
19
Composite example
20
Results reporting

• Specific to a single participant
• 4 constructs
• summary (eg max score, num attempts)
• assessment
• section
• item
• Each construct contains a context which specifies
  details such as name of participant

21
Where does QTI fit in with other specs?

• IMS Meta-data Specification
• IMS Content Packaging Specification
• IMS Learner Information Packaging Specification
• IMS Digital Repositories Interoperability
  Specification

22
IMS Spec Development Process
IMS Members
Elect
Public
Technical Board
Final
Approval
Feedback
Scope
Drafts
Working Group
Scope
Base Document
Public Draft
23
Whats next with QTI?

• V1.2 currently being approved
• selection and ordering of sections and items
• scoring roll-up in sections and assessments (from
  question scores to test scores)
• results reporting
• Question and test metadata
• Harmonisation with other specs
• Integration with ADL SCORM
• Implementations now needed
• V1.3 early in 2002

24
UK Interoperability Developments

• Strong representation in QTI working group
• CETIS QTI SIG
• meetings
• online discussions
• repository of open source tools
• hosting workshops
• CAA Centre SCAAN
• JISC MLE CAA Project
• e3an

25
Products which say they are implementing QTI

• E-Test 3, RIVA Technologies Inc
• Question Mark Perception
• Can studios
• Oracle i-learning
• Blackboard
• WebCT
• Ultimate assessment
• E3AN

26
IMS and other initiatives

• IMS Europe
• SCORM
• Open Knowledge Initiative
• Prometeus
• Questionmark -gt Blackboard WebCT

27
www.imsproject.org
28
QTI page
29
QTI some problems

• For vendors
• implementing QTI is costly
• stops them from protecting their user base
• Vendors want simple specifications users want
  broad, well-defined ones
• It is difficult to define specifications which do
  not have an effect on functionality

30
Finally why bother with QTI?

• Avoid lock in
• allow data to be moved in standard formats
• allow one system to replace another
• Integrate systems from different vendors allow
  choice of best assessment tool
• Facilitate exchange of questions and tests
  between institutions, allow market to develop
• Enable separation of content from presentation
• Standard assessment records will facilitate
  student mobility

31
Further information

• www.sclater.com
• www.imsproject.org
• www.cetis.ac.uk/assessment