Presentation for IEEE

1
Presentation for IEEE
IMS Question and Test Interoperability

• Presentations by
• Eric Shepherd, Team Lead
• Non-Presentation (but to acknowledge) byColin
  Smythe, Technical Team Lead

Montreal, Canada June 27, 2000
2
Agenda

• Eric Shepherd
• Terminology and Vocabulary
• Who produced the QTI specification?
• Benefits of QTI Specification
• Who benefits from the specification?
• The problems solved for
• A college perspective
• A business perspective
• A publisher perspective
• Learning Management Systems perspective
• Specification documents
• Examples

3
What is IMS?

• The IMS is a consortium that promotes an on-going
  collaboration among educational, corporate, and
  government organizations to produce relevant
  specifications for learning resources, systems,
  products, and services.

4
IMS Role in Standards Evolution
Unanimous Adoption
Mature standards
Initial specifications
Mature products
Emerging Consensus
First implementations
5
Terminology and Vocabulary

• Every industry has their vocabulary
• Item the combination of the question, choices,
  scoring scenario and feedback that might be given
  
• Assessment the method of asking questions and
  gathering results such as in a test, exam,
  survey, needs analysis, etc.
• Participant the person who answers the
  questions such as a respondent, candidate, student

6
Why Question and Test Interoperability?

• Although Question and Test are only a subsets of
  item and assessment, these words provide the
  greatest understanding to the widest audience
• While purists can smile knowingly!
• Interoperability? The goal wasand still isto
  provide interoperability between disparate systems

7
Who Are The Players in the Assessment Process?
8
Benefits of QTI Specification

• Provides infrastructure for building assessment
  applications now
• Supports a wide array of question and response
  types
• Extensible and customizable
• Allow items and assessments to be exchanged
  between systems
• Enables new markets for the trade of assessment
  content
• Built by experts from leading market players and
  institutions

9
Who Produced the Specification?

• The IMS QTI specification was produced by an
  international group that contributed their time,
  energy and expenses freely
• The specification is a result of numerous
  conference calls and meetings around the world
• A sample of its members include
• Educational Testing Services (ETS)
• Question Mark
• Lotus Development (IBM)
• Miami-Dade Community College
• Blackboard
• Fretwell Downing

10
Who Benefits from the Specification?

• Application developers have access to
• Templates for structuring data
• Systematic approach for building assessments
• Allows items banks (libraries) to be easily
  constructed
• Opens new learning markets for assessment content
• Publishers can sell well-proven assessments
• Purchase the best assessments from reputable
  sources
• High quality items can be sold, exchanged and
  bartered
• Helping us to deliver good and fair assessments
• Items and assessment can be authored in one
  system and delivered via another

11
Who Benefits from the Specification?A College
Perspective

• College can buy their authoring systems, delivery
  and reporting systems from the best supplier
• College Professor can buy or trade assessments
  that correspond to the material being taught
• Reduces time to think up questions
• Reduces risk of invalid questions
• Standardized exams results can be compared

12
Who Benefits from the Specification?A Business
Perspective

• Different business will use different assessment
  development tools, assessment delivery engines
  and different reporting systems
• For consistency Companies, Divisions and Business
  Partners will need to exchange item banks
• Product knowledge tests developed for employees
  can now be shared with business partners
• Organizations can
• Purchase test banks from Certification
  Authorities
• Purchase test banks along with their learning
  content for more consistent assessments of
  learning objectives

13
Who Benefits from the Specification?A Publishers
Perspective

• New Market to sell assessment content along with
  learning content
• Publishers do not need to provide delivery
  engines, just the data which can run on any
  delivery engine
• Valid assessments add value to learning materials

14
Who Benefits from the Specification?Software
Developers Perspective

• Application developers have access to
• Templates for structuring data
• Systematic approach for building assessments
• Develop knowing software will interoperate with
  complementary systems
• Market will start to demand IMS QTI compatible
  systems as content publishers release QTI content
• Separation of Authoring, Delivery and
  Reporting systems allow for focus and
  specialization

15
Specification Documents

• Information Model
• Explaining the theory behind the specification
• XML Binding Document
• Explaining and defining the data elements
• Best Practice Guide
• A good place to start
• Examples of XML

16
Technical Aims Objectives

• To supporting the exchange of Questions (Items)
  and Tests (Assessments)
• Objectives
• Support current common question types
• Support the more esoteric question types
• Cover the functionally low-lying fruit
• Enable rich feature support in the future
• Allow proprietary extensions

17
Interoperability
18
Response Types

• Sometimes thought of as question-type
• Response type is more meaningful
• Specification allows for proprietary extension
  but these are supported

• Multiple choice
• True false
• Multiple response
• Image hot spot
• Fill in the blank
• Select text
• Slide
• Drag object
• Drag target
• Order objects
• Match item
• Connect the points

19
Response-type Taxonomy
20
The Concept of Rendering

• The same item can be rendered different ways to
  allow for different delivery platforms, technical
  capabilities and participant challenges (e.g.
  visual, audio, physical, etc.)

21
Rendering Example
The same basic question with different rendering.
Not the same educational objective !!
22
Basic Response Render Types

• Response types
• Logical identifier
• XY co-ordinate
• String
• Number
• Logical group

• Render types
• Choice
• Hot-spot
• Fill-in-blank
• Slider
• Object

23
Item XML Structure
24
Section XML Structure
25
Assessment XML Structure
26
True/False Example
27
True/False Example XML
ltquestestinteropgt ltqticommentgt This is a
simple True/False multiple-choice example. The
rendering is a standard radio button style. No
response processing is incorporated. lt/qticomment
gt ltitem ident"IMS_V01_I_BasicExample001a"gt
ltpresentation label"BasicExample001a"gt
ltmaterialgt ltmattextgt Paris is the
Capital of France ? lt/mattextgt
lt/materialgt ltresponse_lid
ident"TF01" rcardinality"Single" rtiming"No"gt
ltrender_choicegt
ltresponse_label ident"T"gt
ltmaterialgtltmattextgt True lt/mattextgtlt/material
gt lt/response_labelgt
ltresponse_label ident"F"gt
ltmaterialgtltmattextgt False lt/mattextgtlt/materia
lgt lt/response_labelgt lt/render_choi
cegt lt/response_lidgt
lt/presentationgt lt/itemgt lt/questestinteropgt
Question
Render format
True
False
28
Response Processing
ltitemgt ltresprocessinggt ltoutcomesgtltdecvar/gtlt/ou
tcomesgt ltrespcondition title"Correct"gt ltcondit
ionvargt ltvarequal respident"TF01"gtTlt/varequalgt
lt/conditionvargt ltsetvar action"Set"
gt1lt/setvargt ltdisplayfeedback feedbacktype"Respo
nse" linkrefid"Correct"/gt lt/respconditiongt lt/res
processinggt ltitemfeedback ident"Correct"
view"Candidate"gt ltmaterialgtltmattextgtYes, you
are rightlt/mattextgtlt/materialgt lt/itemfeedbackgt
ltitemgt
Response
Comparison
Feedback trigger
Feedback
29
Multiple Choice Example
30
Multiple Choice Example XML
ltquestestinteropgt ltitem title"Standard MC Item"
Ident"IMS_V01_I_BasicExample002"gt
ltpresentation label"BasicExample002"gt
ltmaterialgtltmattextgtWhich one of the listed
standards committees is responsible
for developing the token ring specification ?
ltmattextgt lt/materialgt
ltresponse_lid ident"MC01" rcardinality"Single
" rtiming"No"gt ltrender_choice
shuffle"Yes"gt ltresponse_label
ident"A"gt ltmaterialgtltmattextgt
IEEE 802.3 lt/mattextgtlt/materialgt
lt/response_labelgt ltresponse_label
ident"B"gt ltmaterialgtltmattextgt
IEEE 802.5 lt/mattextgtlt/materialgt
lt/response_labelgt ltresponse_label
ident"C"gt ltmaterialgtltmattextgt
IEEE 802.6 lt/mattextgtlt/materialgt
lt/response_labelgt ltresponse_label
ident"D"gt ltmaterialgtltmattextgt
IEEE 802.11 lt/mattextgtlt/materialgt
lt/response_labelgt ltresponse_label
ident"E" rshuffle"No"gt
ltmaterialgtltmattextgt None of the above.
lt/mattextgtlt/materialgt lt/response_label
gt lt/render_choicegt lt/response_lidgt
lt/presentationgt lt/itemgt lt/questestinteropgt
Question
Options
31
Response Processing
ltitemgt ltresprocessinggt ltoutcomesgt ltdecvar
vartype"Integer" defaultval"0"/gt lt/outcomesgt lt
respcondition title"Correct"gt ltconditionvargt
ltvarequal respident"MCb_01"gtAlt/varequalgt lt/con
ditionvargt ltsetvar action"Set"
gt1lt/setvargt ltdisplayfeedback feedbacktype"Respo
nse" linkrefid"Correct"/gt lt/respconditiongt lt/res
processinggt ltitemfeedback ident"Correct"
view"Candidate"gt ltmaterialgtltmattextgtYes, you
are right.lt/mattextgtlt/materialgt lt/itemfeedbackgt
ltitemgt
Response
Condition group
Parameter setting
Feedback content
32
Multiple Response Example
33
Multiple Response Example XML
ltquestestinteropgt ltqticommentgt This is a
multiple-response example. The rendering is a
standard check button style. lt/qticommentgt
ltitem title"Standard Multiple Response Item
ident"IMS_V01_I_BasicExample005"gt
ltpresentation label"RS05"gt
ltmaterialgtltmattextgtWhich of the following
elements are used to form water ?
lt/mattextgtlt/materialgt ltresponse_lid
ident"MR01" rcardinality"Multiple"
rtiming"No"gt ltrender_choice
shuffle"Yes" minnumber1 maxnumber4gt
ltresponse_label ident"A"gtltmaterialgtltmattextgt
Hydrogen lt/mattextgtlt/materialgt
lt/response_labelgt ltresponse_labe
l ident"B"gt ltmaterialgtltmattextgt Helium
lt/mattextgtlt/materialgt lt/response_labe
lgt ltresponse_label ident"C"gt
ltmaterialgtltmattextgt Carbon lt/mattextgtlt/materialgt
lt/response_labelgt
ltresponse_label ident"D"gt
ltmaterialgtltmattextgt Oxygen lt/mattextgtlt/materialgt
lt/response_labelgt
ltresponse_label ident"E"gt
ltmaterialgtltmattextgt Nitrogen lt/mattextgtlt/materialgt
lt/response_labelgt
ltresponse_label ident"F"gt
ltmaterialgtltmattextgt Chlorine lt/mattextgtlt/materialgt
lt/response_labelgt lt/render_choic
egt lt/response_lidgt
lt/presentationgt lt/itemgt lt/questestinteropgt
Question
34
Image Hot-spot Example
35
Image Hot-spot Example XML
ltquestestinteropgt ltqticommentgt This is a
standard image hotspot example. No response
processing is incorporated. lt/qticommentgt ltitem
title"standard image hotspot item" ident"IMS_
V01_I_BasicExample011"gt ltpresentation
label"BasicExample011"gt ltresponse_xy
ident"IHS01" rcardinality"Single"
rtiming"No"gt ltmaterialgtltmattextgt Identify
the vdu. lt/mattextgtlt/materialgt ltrender_hotspotgt
ltmaterialgt ltmatimage
imagtypeimage/gif uriihsvdu.gif
x0"0" width"300" y0"0" height"400"gt lt/ma
timagegt ltmaterialgt ltresponse_label
ident"A" rarea"Rectangle"gt 50,200,250,350 lt/
response_labelgt lt/render_hotspotgt lt/respons
e_xygt lt/presentationgt lt/itemgt ltquestestinteropgt
Question
Render
Image
36
FIB (String) Example
37
FIB (String) Example XML
ltquestestinteropgt ltqticommentgt This is a
standard fill-in-blank (text) example. No
response processing is incorporated. lt/qticomment
gt ltitem title"Standard FIB string Item"
ident"IMS_V01_I_BasicExample012"gt ltpresentation
label"BasicExample012"gt ltresponse_str
ident"FIB01" rcardinality"Single"
rtiming"No"gt ltrender_fib fibtype"String"
prompt"Dashline" maxchars"6"gt ltmaterialgt
ltmattextgtComplete the sequence Winter,
Spring, Summer, lt/mattextgt lt/materialgt lt
response_label ident"A"gt lt/response_labelgt
ltmaterialgt ltmattextgt .
lt/mattextgt lt/materialgt lt/render_fi
bgt lt/response_strgt lt/presentationgt lt/itemgt lt
questestinteropgt
Render
Question
Question
38
Numerical Example
39
Numerical Example XML
ltquestestinteropgt ltqticommentgt This is a
standard numerical fill-in-blank using slider
rendering. No response processing is
incorporated. lt/qticommentgt ltitem
title"Standard FIB numerical Item with Slider
rendering" ident"IMS_V01_I_BasicExample017"gt
ltpresentation label"BasicExample017"gt ltrespon
se_num ident"NUM04" rcardinality"Single"
rtiming"No"gt ltmaterialgt ltmattextgtHow
many degree are there in a triangle ?
lt/mattextgt ltmaterialgt ltrender_slider
lowerbound"1" upbound"360" step"1"
startval"90" steplabelNo"
orientationHorizontal"gt ltresponse_label
ident"A"gt lt/response_labelgt lt/render_sli
dergt lt/response_numgt lt/presentationgt lt/itemgt
lt/questestinteropgt
Question
Slider render
40
Response Processing
ltitemgt ... ltresprocessinggt ltoutcomesgt ltdecvar
varname"SCORE1" vartype"Integer"
defaultval"1"/gt lt/outcomesgt ltrespconditiongt lt
qticommentgtScoring for the correct answer.
lt/qticommentgt ltconditionvargt ltvarequal
respident"NUM04"gt 180 lt/varequalgt lt/conditionva
rgt ltsetvar action"Add" varname"SCORE1"gt 9
lt/setvargt ltdisplayfeedback feedbacktype"Respons
e" linkrefid"Correct"/gt lt/respconditiongt lt/respr
ocessinggt ltitemfeedback ident"Correct"
view"Candidate"gt ltmaterialgtltmattextgtYes, 180
degrees. lt/mattextgtlt/materialgt lt/itemfeedbackgt ...
lt/itemgt
Variable
Response link
Action
41
Composite Example
42
Composite Example XML
ltquestestinteropgt ltqticommentgtThis is a
composite item.lt/qticommentgt ltitem
title"Composite Item" Ident"IMS_V01_I_CompExampl
e001"gt ltpresentation label"CompExample001"gt
ltmaterialgtltmattextgtWhich city is the
capital of England and name another city in
England ?ltmattextgtlt/materialgt
ltresponse_lid ident"Comp_MC01"
rcardinality"Single" rtiming"No"gt
ltrender_choice shuffle"Yes"gt
ltresponse_label ident"A"gt ltmaterialgtltmattextgt
Sheffield lt/mattextgtlt/materialgt
lt/response_labelgt ltresponse_label
ident"B"gt ltmaterialgtltmattextgt London
lt/mattextgtlt/materialgt lt/response_label
gt ltresponse_label ident"C"gt
ltmaterialgtltmattextgt Manchester lt/mattextgtlt/materia
lgt lt/response_labelgt
ltresponse_label ident"D"gtltmaterialgtltmattextgt
Edinburgh lt/mattextgtlt/materialgt
lt/response_labelgt lt/render_choicegt
lt/response_lidgt ltresponse_str
ident"Comp_FIB01" rcardinality"Single"
rtiming"No"gt ltrender_fib fibtype"String"
prompt"Box"gt ltmaterialgtltmattextgtAnother
citylt/mattextgtlt/materialgt ltresponse_label
ident"A/gt lt/render_fibgt lt/response_strgt
lt/presentationgt lt/itemgt lt/questestinteropgt
Question
Choice selection
String input
FIB render
43
Section Example
ltquestestinteropgt ltsection ident"IMS_V01_S_Examp
le001gt ltitem ident"IMS_V01_I_BasicExample001"gt
ltpresentation label"RS01"gt
ltmaterialgt ltmattextgt Paris is the
Capital of France ? lt/mattextgt
lt/materialgt ltresponse_lid
ident"TF01"gt ltrender_choicegt
ltresponse_label ident"T"gt
ltmaterialgtltmattextgt True lt/mattextgtlt/materia
lgt lt/response_labelgt
ltresponse_label ident"F"gt
ltmaterialgtltmattextgt False
lt/mattextgtlt/materialgt lt/response_labe
lgt lt/render_choicegt lt/response_lidgt
lt/presentationgt lt/itemgt ltitemref
linkrefid "IMS_V01_I_BasicExample002"gt ltitemref
linkrefid "IMS_V01_I_BasicExample003"gt
lt/sectiongt lt/questestinteropgt
Reference to external items
44
Assessment Example
ltquestestinteropgt ltassessment identIMS_V01_A_Ex
ample301gt ltsection identIMS_V01_S_Example301
gt ltitem ident"IMS_V01_I_BasicExample301"gt
ltpresentation label"RS01"gt
ltmaterialgtltmattextgt Paris is the Capital of
France ? lt/mattextgtlt/materialgt
ltresponse_lid ident"TF01"gt
ltrender_choicegt ltresponse_label
ident"T"gt ltmaterialgtltmattextgt
True lt/mattextgtlt/materialgt
lt/response_labelgt
ltresponse_label ident"F"gt
ltmaterialgtltmattextgt False
lt/mattextgtlt/materialgt lt/response_lab
elgt lt/render_choicegt
lt/response_lidgt lt/presentationgt
lt/itemgt lt/sectiongt ltsectionref linkrefid
"IMS_V01_S_Example302"gt ltsectionref linkrefid
"IMS_V01_S_Example303"gt lt/assessmentgt lt/questes
tinteropgt
Reference to external sections
45
Extensions Facility

• assessproc_extension
• condition_extension
• itemproc_extension
• render_extension
• respcond_extension
• scorecondition_extension
• response_extension
• mat_extension
• response_na
• sectionproc_extension
• var_extension

• Content declared as ANY.
• Recommended naming convention

46
Conformance Statement
47
Resource Kit

• Specifications
• Support documentation e.g. tutorials
• DTDs XDRs
• Example XML files
• Export validation tool
• Revision release notes
• Links to other key resources

48
Set of DTDs XDRs

• Developed using XML Authority
• Functionality
• Full (with and without comments)
• Core (no extensions or V2.0 hooks)
• Item only and Item/Section only
• Platform
• File formats for MacOS, IBM and Unix

49
Set of Examples

• Basic examples
• With and without response processing (30)
• Composite examples
• Limited range of known types (5)
• Advanced examples
• Full range of ASIs (10)
• Examples with know errors
• Testing for input parser (30)

50
Version 2.0 Issues

• Results interoperability
• Reporting the results
• Integrated scoring method
• Between Assessment, Sections and Items
• Dynamic/static selection and sequencing
• Section and Items

51
IMS Web-site
Click here for the specs
URL http//www.imsproject.org
52
IMS QTI Web-site
Click here to download the specs
53
QTI Documents Registration
Click here to download the specs
54
Questions?

• Or do you want a test?

55
Contributor Perspective

• Russell Almond (ETS)
• Topics
• ETS Process Model
• E-rater Example
• Chinese Example
• Implementing the specification

56
Four Process Delivery System
Presentation Process
Activity Selection Process
Task/Evidence Composite Library
Examinee
Administrator
Summary Feedback
Task Level Feedback
Evidence Identification Process
Evidence Accumulation Process
57
Short Essay Question
Answer the question below in the form of an essay
of no more than 250 words. Make sure your
arguments are structured in proper sentences and
paragraphs. Include details to support your
arguments.
Describe in 250 words or less the causes that
lead to the US Civil war. Include political and
economic causes for both the North and the South.
Where possible, support your answers with
examples. Make sure that your answers are in the
form of well formed paragraphs.
1. Supporting E-rater and human rater,
simultaneously. 2. Diagnostic Scoring.
58
Essay Purpose Instructions
ltitem title"Essay with Human or Machine
Scoring Options." ident"ETS_Item_Essay1"
label"Essay"gt ltqticommentgt This is a
classic essay test, but it shows how to use
proprietary extensions to hide scoring data.
lt/qticommentgt ltitemobjectives view"Author"gt
ltmaterialgtltmattextgt This question
attempts to gather information about the students
understanding of forces causing the American
Civil War. lt/mattextgtlt/materialgt
lt/itemobjectivesgt ltitemrubric
view"Candidate"gt ltmaterialgt
ltmattextgtAnswer the question below in the form of
an essay of no more than 250 words. Make sure
your arguments are structured in proper
sentences and paragraphs. Include details to
support your arguments. lt/mattextgt
lt/materialgt lt/itemrubricgt
Start with information about the purpose of the
item.
59
Essay Essay Prompt
ltpresentation label"Essay"gt
ltmaterialgt ltmattext label"Prompt"
texttype"text/plain" charset"ascii-us"gt
Describe in 250 words or less the causes that
lead to the US Civil war. Include
political and economic causes for both
the North and the South. Where possible, support
your answers with examples. Make sure
that your answers are in the form of
well formed paragraphs. lt/mattextgt
lt/materialgt ltresponse_str ident"essaytext"
rtiming"No"gt ltrender_fib charset"ascii-us"
encoding"UTF_8" fibtype"String"
rows"66" maxchars"2500" columns"80"gt lt/render_
fibgt lt/response_strgt lt/presentationgt
This is the prompt. We capture the response in a
variable called "essaytext" which is what is
expected for e-rater.
60
E-rater Response Processing
ltresprocessing scoremodel"E-Rater"gt
ltqticomment e-dtype"string"gt This is a
prototype for E-rater scoring. Note,
these instructions would provide the prompt
specific data which would need to be sent to
e-rater along with the essaytext.
lt/qticommentgt ltoutcomesgt ltdecvar
varname"Score" vartype"Integer" minvalue"0"
maxvalue"6"gtlt/decvargt ltinterpretvar
varname"Score" view"All"gt ltmaterialgt
ltmatref linkrefid"ScoreRubric"gtlt/matrefgt
lt/materialgt lt/interpretvargt lt/outcomesgt
ltitemproc_extensiongt ltets_erater_prompt
ets_erater_waitreq"300"gtCivWar13lt/ets_erater_data
gt lt/itemproc_extensiongt lt/resprocessinggt
E-rater data is mostly just prompt ID.
61
Chinese String Example
62
Questions?

• Or do you want a test?

63
Thank You for Your Attention

• For detailed information please refer to
• http//www.imsproject.org/question/
• Please direct comments on this presentation to
• Ed Walker edwalker_at_ectwalker.com
• Eric Shepherd eric_at_questionmark.com
• Colin Smythe colin_at_dunelm.com