TAO General Presentation

1
TAO General Presentation

• Thibaud Latour,
• CRP Henri Tudor - Luxembourg

2

• A Modular and Versatile Platform for
  Collaborative and Distributed Computer-based
  Assessment Management
• Thibaud Latour1, Romain Martin2
• 1Centre de Recherche Public Henri Tudor, Centre
  for IT Innovation
• 2University of Luxembourg, EMACS Research Group

Contact Thibaud Latour thibaud.latour_at_tudor.lu R
omain Martin romain.martin_at_uni.lu
Centre de Recherche Public Henri Tudor Centre for
IT Innovation (CITI) Reference Systems for
Certification and Modeling unit (REF) Avenue John
F. Kennedy, 29 L-1855 Luxembourg-Kirchberg LUXEMBO
URG Tel [phone] Fax [phone]
Université du Luxembourg EMACS Research
Group FLSHASE Campus Walferdange B.P.2 L-7201
Walferdange LUXEMBOURG Tel  [phone] Fax 
352-333256
3
Presentation of the platform
4
Competency assessment context

• Allowing an efficient collaboration among
  different actors situated at different
  institutional levels and in different
  geographical locations
• Researchers
• Item developers
• Knowledge domain experts
• Large-scale survey managers
• School / ministerial responsibles
• Corporate subsidiaries
• Trainers and teachers
• Human resource managers
• Managing complex database structures
• Items
• Tests
• Subjects
• Groups
• Efficient test delivery and data collection
• Rapid creation of items and tests
• Enabling complex interaction and behaviour
  collection
• Cost and time reduction of the authoring
  delivery analysis process

5
CBA management process

• Computer-Aided Assessment
• Computer-Based Testing

6
CBA domain top-level model
Six major knowledge domains can be distinguished
in the CBA problem
Each domain corresponds to an independent data
domain that can be managed by different actors
possibly distributed in various remote locations
7
CBA domain top-level model
Groups are composed of Subjects and are assigned
a series of Tests. Tests are in turn composed of
Items that contain multimedia content. The
Results can be put in relation with any feature
of Groups, Subjects, Tests, or Items.
8
CBA domain top-level model
This top-level model enables the separation of
concern with respect to the different types of
resources involved in the testing process. This
separation is the keystone of modularity and
component reusability in the creation,
management, end delivery of tests.
9
CBA domain top-level model
PRIVATE
Separation between subjects and groups is a
convenient way to preserve privacy ? personal
data are considered separately from the strict
CBA problem (represented by the grouptest
relation).
10
CBA domain top-level model
Testitemmultimedia resource composition scheme
also separates clearly the problems related to

• the measurement (Test),
• the elementary interaction between the subject
  and a piece of evaluated knowledge (Items)
• the graphical presentation used in the
  interaction (Multimedia resource).

11
CBA domain top-level model
For analysis purpose, results can be put in
relation with the other types of resources.
12
CBA domain top-level model
Test can be delivered according to particular
schedules, or organised in the form of campaigns
gathering several different tests. Constraints
regarding the number of possible passes per
subjects can also be established, as well as
multi-pass result combination policies.
13
Application architecture

• Platform split in six types of specialized
  modules
• Each module dedicated to a specific kind of
  resource
• Modules provide basic functionalities
• domain modelling
• creating hierarchies of categories
• user-defined properties
• populating categories with instance data
• complete definition of a test (items, time and
  scoring algorithms, ) and its context (subjects,
  groups, )
• Modules enable sharing and inter-connection of
  resources
• User and access right management
• Connections resources used by the Test Delivery
  Server
• analyse a test description
• collect all constituting elements over the nodes
  distributed on the Web
• Test delivered to the subjects web browser as a
  flash animation
• Result report sent back to the Test Delivery
  Server and re-dispatched towards affiliated
  Result modules
• Users connected to the delivery server are
  subjects managed by the subject module

14
Application architecture

• Task and resource domain-oriented modules
• Subject management
• Group management
• Test definition and management
• Item creation and management
• Result management and analysis
• Deliver tests
• ? Resource Management
• Key technology
• Semantic Web standards and tools
• P2P network

15
Application architecture

• Generis Kernel

16
Application architecture

• Assessment process-centred instead of test
  execution-centred platform
• Factorisation of common IT features
• Concentrate the IT effort on psychometric
  value-added features
• All effort concentrated on versatility and
  openness

17
Application architecture
18
Functional coverage

• Subject modelling and registration
• Group subject together and assign tests ( model)
• Build tests from item bank ( model)
• Sequential, adaptive
• IRT ML and EAP scoring Classical weighted or
  not
• Author items ( model)
• Type MultipleChoice, Kohs, CAMPUS, C-Test
• IRT models Rash, Birnbaum, 3-parameters
• Analyse results ( model)
• Correlate results (score, endorsement, event log
  ) with models
• exports excel sheets
• Deliver tests
• Planning (dates), constraints (number of
  authorized passes), result dispatching, cache
  management
• Multimedia
• Rich text, Sound, Image, video, Flash animation
  (including communication with test and event
  logging)

19
Item and test authoring

• Current Item and test architecture

20
Item and test authoring

• Current Item and test architecture
• Creating new item type or modifying test
  framework
• Client-side Item engine ? Flash developer
• Model ? Analyst
• Server-side authoring tool (web application) ?
  Flash or PHP developer
• Creating items and tests
• Lambda user with basic TAO experience

21
Item and test authoring
22
Item Types
23
Current item types

• Standard item (QCM MCQ)
• Block-design item (Kohs)
• C-test items
• Campus items
• Item types or components under development
• Cascade
• Open ended question
• Reordering
• Association
• WS delegation ? SQL and XML programming

24
Current item types

• Standard Item (1/2)

25
Current item types

• Standard Item (1/2)

Flash animation
26
Current item types

• Block-design (Kohs)

27
Current item types

• C-Test

28
Current item types

• Delegation to Web Service TAO-XML example (1/4)

29
Current item types

• Delegation to Web Service TAO-XML example (2/4)

Answer is an XML piece of code
Evaluation is made using a web service
30
Current item types

• Delegation to Web Service TAO-XML example (3/4)

Feed-back from the web service the piece of XML
is correct from a syntax point of view
31
Current item types

• Delegation to Web Service TAO-XML example (4/4)

Feed-back from the web service the piece of XML
contains a syntax error at line 9
32
Current item types

• Association

33
Test Delivery (to come)
34
Delivery Campaign
35
Delivery Campaign
Delivery server
36
Delivery Campaign
Planning Campaign
37
Delivery Campaign
Precompiled deliveries
38
Delivery Campaign
Delivery server (client prototype)
Delivery server (client prototype)
39
Future directions

• Graphical User interface
• Service-based (plug-ins)
• Widget-Based
• Oriented towards end-user in context
• Personal desktop (like apple dashboard/netvibes)
• Orchestration of activities ? workflow
• Based on XUL and SVG (with RDF binding)
• Result analysis
• Extract data and meta-data
• Interface with R for analysis, plotting, and
  automatic report generation
• What else with R?
• IRT models and algorithms, scoring,
  externalised as Web Services

40
Future directions

• General item architecture
• One model One authoring tool
• BLC (Business/Logic/Content) model
• WYSIWYG authoring application ? draw the item
  and test
• Self-executable (even out of TAO)
• Various test sequencing
• Random
• Sequence
• Adaptive (IRT)
• Conditional branching (depend on instantaneous
  score, time, answer, endorsement, )

41
Future directions

• Questions about QTI
• Only as a manifest
• Not as a rendering standard
• Non-automatic (human-supervised) imports/exports
• Integration of SCORM
• Test (small classical ones) or item as LO
• API

42
Deployment
43
Infrastructure and deployment topology

• Deployment topology fits with organisational
  structure

44
Infrastructure and deployment topology

• Tests can be passed locally on LAN when no
  Internet connection is available

45
TAO Server Farm
46
Rapid exploration
47
Rapid exploration
48
Getting started

• LoginUserName pisaPassword pisa

49
Choosing a module

• PISAItems

50
I. Modelling an Item

• QCM

51
II. Composing a test

• Selecting items and defining test modes

52
III. Managing subjects

• Construct your subject data sheet

53
IV. Defining groups

• Groups gather subjects according specific
  criteria
• Tests are assigned to groups

54
V. Delivery of tests

• Compiling tests
• Executing tests

55
VI. Exploitation of result

• Extracting, comparing, exporting results

56
Questions
57
Overview of TAO Use Cases
58
Projects using TAO

• Already realized projects
• Univ. Lux. evaluation of competencies in
  electrotechnics in secondary technical schools
• CLL evaluation of language skills at the entry of
  training programs
• Projects to be realized
• Univ. Lux. Computer-Assisted Mathematical
  Problems Understanding and Solving (CAMPUS)
• Univ. Lux. Computer-assisted scientific
  competencies assessment and dynamic evaluation
  (CASCADE)
• Univ. Lux. evaluation of language skills profiles
  at the end of secondary school (C-tests)
• Lux. Min. Edu. school system monitoring and
  school evaluation
• Lux. Min. Edu. automatized feedback for
  diagnostic tools in mathematics
• Projects starting
• Ger. Min. Edu. Res. Industrialization of TAO to
  provide general tool for psychometric research
  programmes and school monitoring
• Lux. Nat. Res. Fund A Quality Control System for
  Assessment Processes by Means of a Collaborative
  Internet-Based Assessment Platform

59
Monitoring
60
First data collection for Luxembourg school
monitoring

• 24 schools
• Paper and pencil student motivation and attitudes
  questionnaire September 05 N3932
• Paper and pencil student motivation and attitudes
  questionnaire July 06 N3762
• Cognitive tests July 06 N3476 (mainly TAO based)

61
Cognitive tests July 06 language skills
62
Cognitive tests July 06 Mathematics
63
The future

• Cognitive tests in German, French and Mathematics
  at the end of each schooling cycle measurement
  of student attitudes and motivation
• 2nd grade
• 4th grade
• 6th grade
• 8th grade
• Longitudinal data base

64
(No Transcript)
65
School monitoring coding and reporting

• Teacher authentication

66
School monitoring coding and reporting

• Editing test results

67
School monitoring coding and reporting

• View consolidated results

68
School monitoring coding and reporting

• Compare results between class and school averages

69
Recent development of TAO
70
TAO and Luxembourg school monitoring

• Revision of web-based delivery bootable CDs for
  increased test security
• Updating of TAO server farm
• Automatization of feedback and reporting

71
TAO related research activities

• CASCADE (computer-assisted scientific
  competencies assessment and dynamic evaluation)
• CAMPUS (computer-assisted mathematical problem
  understanding and solving)
• Multilingual testing
• TAO-QUAL