DLXXI Courseware Guidelines Workshop

1
DLXXI Courseware Guidelines Workshop

• The University of Southern California
• Center for Cognitive Technology
• Institute for Creative Technology
• Dr. Richard E. Clark,
• Rich DiNinni, Kenneth Yates
• Lesson 3

2
Lesson Goals

• You will learn to use the checklist to determine
  whether DL designs contain adequate
• Practice exercises to help trainees apply what
  they learn
• Feedback to help them adjust their learning so
  that it is accurate
• Assess learning to insure that ELOs and TLOs
  are achieved

3
Reasons

• Practice is seldom provided, which causes
  problems with learning and effective transfer to
  the mission environment
• Designers often give very difficult problems for
  practice, which damages trainee confidence
• Practice seldom reflects objectives, assessment
  or mission environment

4
Reasons

• Feedback often focuses trainees on the fact that
  they made a mistake, or were wrong this slows
  and decreases learning
• Assessment of training is most often a memory
  test NOT adequate for use/apply goals
• Objectives, knowledge type/use, practice AND
  assessment must be aligned

5
Practice

• Part task to whole task practice
• After four or five small (part-task) practices,
  present a big problem that requires the use of
  all previously practiced tasks (whole-task)
• Set practice in mission environment
• Simple tasks, then more complex novel
• Practice must reflect learning objectives
• Practice until they succeed or exceed limit

6
Feedback

• Give frequent feedback during practice so
  mistakes do not accumulate
• Feedback should
• Acknowledge success great job up to this point
• Do not say you made an error instead say Your
  strategy did not work on this section/part
• Give opportunity to revise their strategy, and to
  review relevant parts of module
• Align practice with learning objective, mission
  environment and assessment of performance

7
Assessment

• Do assessments ask trainees to apply what they
  have learned (Tell and/or Use)?
• Are the assessments aligned with the ELOs and
  TLOs and the exercises?
• Do assessments reflect the mission environment
  (transfer) as much as possible?

8
Practice, Feedback, and Assessment Design
On page 33 Practice and tests will occur in
several places throughout each module and lesson,
to enhance and reinforce learning but also to
prepare students for the testing process. Types
of quiz questions and type of feedback will vary
by which type of quiz is being developed as
follows Drill/practice or Obstacles These
will be checks on learning to test both knowledge
acquisition and to provide early practice for
application of that knowledge to scenarios based
on the Irvington terrain. (See Notes Below)
9
Practice, Feedback, and Assessment Design
It appears that all lessons will be practiced.
10
Practice, Feedback, and Assessment Design
After describing how to fell trees and remove
stumps with explosives, the course covers the
construction of an abatis using trees An abatis
can be constructed rapidly using demolition to
fell the trees. The trees should remain attached
to the stumps to make the obstacle more effective
and difficult to clear. Is the abatis practice
also a larger practice of the previous training
on how to feel trees and uproot stumps?
11
Abatis
12
Practice, Feedback, and Assessment Design
We believe that this design offers the
opportunity for constant practice of larger and
larger chunks of learning. However, it might be
a good idea to bring this up with the contractor
to insure that they provide a whole practice at
the end of the course where trainees have to
remove a number of obstacles in a real or
simulated field setting.
13
Practice, Feedback, and Assessment Design
From the design on page 33 Skills are trained
for implementation in the geographic region
nearest to the site of practical exercise
implementation. provide early practice for
application of that knowledge to scenarios based
on the Irvington, KY terrain Is this statement
an attempt to mirror the mission environment?
14
Practice, Feedback, and Assessment Design
Skills are trained for implementation in the
geographic region nearest to the site of
practical exercise implementation. While this may
not mirror the mission environment, one could
conclude that the designers thought carefully and
practically about the transfer environment they
could anticipate.
15
Practice, Feedback, and Assessment Design
Does this appear to be an attempt to sequence by
difficulty level? In the design document, on
pages 6-7 KNK01410 - Calculate, Designate, and
Employ Demolitions 052-193-2015 Place
Timber-Cutting Charges 052-193-2016 Place
Steel-Cutting Charges 052-193-2017 Place
Breaching Charges 052-193-2018 Place Cratering
Charges 052-193-3022 Calculate Timber-Cutting
Charges 052-193-3023 Calculate Steel-Cutting
Charges 052-193-3024 Calculate Breaching Charges
16
Practice, Feedback, and Assessment Design
The designers did not state that they sequenced
tasks by difficulty levels.
17
Practice, Feedback, and Assessment Design
Will sequencing by difficult level also lead to
practice of Increasingly novel problems? In the
design document, on pages 6-7 KNK01410 -
Calculate, Designate, and Employ
Demolitions 052-193-2015 Place Timber-Cutting
Charges 052-193-2016 Place Steel-Cutting
Charges 052-193-2017 Place Breaching
Charges 052-193-2018 Place Cratering
Charges 052-193-3022 Calculate Timber-Cutting
Charges 052-193-3023 Calculate Steel-Cutting
Charges 052-193-3024 Calculate Breaching Charges
18
Practice, Feedback, and Assessment Design
Since the contractor did not state that more
familiar practice exercises would be placed
before less familiar exercises, this item
receives a NO-GO.
19
Practice, Feedback, and Assessment Design
Does this statement indicate an attempt to
connect learning objectives with practice
exercises? Page 33, 34 in the course design
states End of ELO tests or quizzes These will
cover at least one question per learning
objective. Questions will be performance/scenario
based. Feedback will again be remedial but may
not be as detailed (i.e. each answer does not
need a feedback statement), and should at least
refer the student back to the reference relating
to the lesson.
20
Evaluating Practice, Feedback, and Assessment
We believe that the designers will attempt to
connect learning objectives with practice
exercises.
21
Practice, Feedback, and Assessment Design
Does this statement describe a comprehensive
strategy for practice of ALL types and uses of
knowledge taught in the course? From page
33 General Practice Strategies Practice and
tests will occur in several places throughout
each module and lesson, to enhance and reinforce
learning but also to prepare students for the
testing process.
22
Practice, Feedback, and Assessment Design
The contractor describes frequent practice, but
not of all types and uses of knowledge taught in
the course.
23
Feedback

• Give frequent feedback during practice so
  mistakes do not accumulate
• Feedback should
• Acknowledge success great job up to this point
• Do not say you made an error instead say Your
  strategy did not work on this section/part
• Give opportunity to revise their strategy, and to
  review relevant parts of module
• Align practice with learning objective, mission
  environment and assessment of performance

24
Practice, Feedback, and Assessment Design
Page 16 in design Feedback will be used
throughout this program/course. Each lesson
will contain feedback tools to assist the student
in his retention of training materials. Feedback
will utilize variety, humor, and interactivity to
enhance student motivation and shape an online
environment conducive to learning Does this
statement indicate that feedback is provided
during practice?
25
Practice, Feedback, and Assessment Design
We believe that the contractors design clearly
indicates that feedback will be provided during
practice
26
Practice, Feedback, and Assessment Design
Did the preceding notes section describe feedback
that is frequent enough to correct errors?
27
Feedback Example 1
28
Feedback Example 2
29
Practice, Feedback, and Assessment Design
We believe that feedback will be frequent enough
so that errors do not accumulate
30
Practice, Feedback, and Assessment Design
Look at example on next slide
31
Does this focus on errors or strategy?
32
Practice, Feedback, and Assessment Design
33
Practice, Feedback, and Assessment Design
Page 33 in many instances students will be
required to do the exercise over until they
perform correctly Where appropriate, students
will be allowed two tries at the correct answer
and then if incorrect will receive
remediation. Does these statements indicate that
students will have the opportunity to correct
their strategy?
34
Practice, Feedback, and Assessment Design
It appears that students will have the
opportunity to correct their strategy
35
Practice, Feedback, and Assessment Design
Review the description of the exercises below to
see if The designers intended to allow the
trainees to review the demonstrations.

36
Practice, Feedback, and Assessment Design
It appears that trainees are able to go back
within modules at any time, in order to review
relevant portions of the demonstration.
Additionally, they may repeat any section or
module of the course, both before and after
module tests.
37
Assessment

• Do assessments ask trainees to apply what they
  have learned (Tell and/or Use)?
• Are the assessments aligned with the ELOs and
  TLOs and the exercises?
• Do assessments reflect the mission environment
  (transfer) as much as possible?

38
Practice, Feedback, and Assessment Design
Four specific tests for record will occur in the
DL phase, testing the OPFOR/IPB content, the ERO
content, the CADD content, and the RS content.
Each of these four tests for record will contain
some computer-graded knowledge portion as well as
application based products that are to be sent to
the SGI for grading. All portions of the tests
for record will be performance-oriented so that
even the knowledge portion will use graphics and
scenarios as well as standard computer-graded
question formats.
39
Practice, Feedback, and Assessment Design
We believe that the lesson uses application
exercises to assess the trainees learning.
40
Practice, Feedback, and Assessment Design
We determined earlier that objectives were
aligned with practice exercises this asks if
both practice and objectives are aligned with
assessments (tests, scored practice). What is
your decision after reading the description in
the Notes Section below (taken from page 33-35 in
the design).
41
Practice, Feedback, and Assessment Design
The design promises that The learner will
complete an MPA at the end of each module. They
will be tested to determine if they acquired the
knowledge presented within the associated module.
If they passed the MPA, they are granted credit
for the module. However, if they fail, they will
need to retake the MPA until they pass. They will
receive feedback as to which topics they failed
and will receive a recommendation to retake that
portion of the Module.
42
Practice, Feedback, and Assessment Design
Check the description from the design in the
Notes section below for the Course Practical
Exercise Does it meet this checklist
requirement?
43
Practice, Feedback, and Assessment Design
The design states After completing all
recommended modules, the learner will take the
Course Practical Exercise (and) will be placed in
a simulated environment where they will apply all
of the skills and knowledge acquired in the
course(and) Skills are trained for
implementation in the geographic region nearest
to the site of practical exercise implementation
(and) provide early practice for application of
that knowledge to scenarios based on the
Irvington, KY terrain . This seems to fit the
checklist requirement
44
Team Exercise 3Evaluating Practice, Feedback
and Assessment

• Each team selects
• A Team Leader (coordinator, timekeeper)
• A Reporter (summarizes and presents teamwork to
  rest of us)
• Assign responsibility to find examples of each
  introduction item (3.1 to 3.3.2) in the design
  for the Armor Captains Career Course
• Report those examples to the rest of the workshop
  when it is your turn

45
Practice, Feedback, and Assessment Design
3.1. Do all lessons include practice? 3.1.1. Is
part-task practice followed with whole-task
practice? 3.1.2. Does whole-task practice mirror
the mission environment? 3.1.3. Does practice
begin with simple problems and progress to more
complex problems? 3.1.4. Does practice require
trainees to solve increasingly novel
problems? 3.1.5. Is there consistency between
practice exercises and the learning objectives?
3.1.6. Are there an adequate number of practice
exercises to master each skill? 3.2. During
practice, is feedback provided? 3.2.1. Is
feedback frequent, so that errors do not
accumulate? 3.2.2. When trainees make mistakes
does feedback 3.2.2.1. Acknowledge correct use
of strategies or steps and not focus on their
mistakes? 3.2.2.2. Provide trainees the
opportunity to correct their strategy? 3.2.2.3.
Provide the opportunity to review the relevant
parts of the demonstration? 3.3. Does the
lessons provide a method of assessing the
trainees learning by asking them to apply what
was learned? 3.3.1. Are assessments aligned with
the learning objectives and practical
exercises? 3.3.2. Do the assessments reflect the
performance of the learning objectives in the
mission environment as closely as media will
permit?
46
Practice, Feedback, and Assessment Design
47
Practice, Feedback, and Assessment Design
48
Practice, Feedback, and Assessment Design
49
Practice, Feedback, and Assessment Design
50
Practice, Feedback, and Assessment Design
51
Practice, Feedback, and Assessment Design
52
Practice, Feedback, and Assessment Design
53
Practice, Feedback, and Assessment Design
54
Practice, Feedback, and Assessment Design
55
Practice, Feedback, and Assessment Design
56
Practice, Feedback, and Assessment Design
57
Practice, Feedback, and Assessment Design

58
Practice, Feedback, and Assessment Design
59
Practice, Feedback, and Assessment Design
60
Practice, Feedback, and Assessment Design