Building a Better

1
Building a BetterX! Test!
6th Annual Career Cluster Institute Atlanta,
GA June 17, 2007 Session D6 E7

• Presenter
• Ms. Brenda Hattaway
• Assistant Executive Director VTECS Atlanta,
  Georgia

2
SESSION FOCUS Critical Components of Career
Cluster Implementation
9 Multi-Measure Assessment 15 Student
Centered Learning
3
Since 1973 . . .
A Consortium for Innovative Career and
Workforce Development Resources
4
VTECS develops resources . . .
. . . for CTE.
Tools
5
Procedures (Steps)
Item Banks
Duty/Task Lists
Academic Skills
Instructional Tools
Objectives
6
(No Transcript)
7
What kind of test?
Summative or Formative? Its all about purpose!
8
SUMMATIVE ASSESSMENT Assessment OF Learning
Used to determine how much students have learned
at a particular point in time in order to report
achievement status.
9
FORMATIVE ASSESSMENT Assessment FOR Learning
A planned process in which teachers or students
use assessment-based information to adjust what
they're doing.
10
Does the pilot test process we use really tell us
anything about how well our instruction works?
What types of tasks characterize the typical day
of a submarine officer?
Personality inventories indicate that our
programmers tend to have higher extroversion
scores that introversion.
List the steps in the accounting cycle.
11
How can we improve student learning in the
classroom and raise student performance on
high-stakes tests? The key is continuing
assessment and evaluation throughout the school
year, as well as a commitment to the success of
all students . . .
Using Data to Drive Student Achievement in the
Classroom and on High-Stakes Tests
T H E Journal (Technological Horizons in
Education) January, 2003
12
1
Why are you testing in the first place?
13
2
Communicate expectations to your students!
14
3
Make Sure Your Students Know the Material

• 1. Create an assessment instrument that looks
  like the job, not one that looks like other tests
  that you have taken or seen.
• Keep test items and instruction directly
• related to the skill standards.

15
4
Develop Great Test Items!
16
Objectives
CONDITION PERFORMANCE STANDARD (CRITERIA)
Using a pencil, ruler, tape and a single edge
razor, mount photographs for framing with
a margin of 1/4 inch on all sides.
17
Be Sure to Assess . . .

• Knowledge
• Performance
• Workplace Skills

18

• Review of Item Writing Guidelines
• Multiple Choice
• Performance
• Scenario

19
Multiple Choice
20
Stem
Which of the following is the inch equivalent of
a 16d nail? a. 1 ¼" b. 2" c. 2 ½" d. 3 ½"
Options
Distractors
Correct Answer
21
MULTIPLE-CHOICE ITEMS WRITING and REVIEWING
GUIDELINES

• Options
• There are three or four options.
• The options are presented in a logical order, if
  one exists.
• The options are independent not overlapping.
• The options for an item are homogeneous in
  content.
• The length of the options is fairly consistent.
• Options are phrased positively, not negatively.
• There are no clues through the use of faulty
  grammatical
• construction. (Grammatical inconsistencies
  involving "a," or "an,"
• for example, give clues to the correct
  answer.)

The item Is based on a clearly stated,
valid skill or standard. Uses correct
grammar, punctuation, and spelling. Is not a
trick item that would mislead or deceive
examinees into answering incorrectly.
Uses either the best answer or the correct
answer format Does not have textbook,
verbatim phrasing. Is not based on
opinions. Is based on important aspects of
the content area not on trivial
material. Does not contain potentially
insensitive content or language.
ITEMS
OPTIONS
The stem of the item Is stated in either
question form or completion form. Is clearly
worded and lets the examinee know exactly what
is being asked. Includes enough material
to make the problem/question clear.
Includes the central idea and most of the
phrasing.
Distractors Are plausible and logical.
Incorporate common errors of students. Use
familiar yet incorrect phrases. May be true
statements that do not correctly answer the item.
DISTRACTORS
STEM
The correct option Is positioned so that it
appears about the same number of times in each
possible position for a set of items. Is the
one and only correct, or clearly best, answer on
which experts would agree.
ANSWER
22
Guidelines for Developing Multiple Choice Items

• General multiple choice item-writing (procedure)
• 1. Use either the best answer or the correct
  answer format.
• 2. Avoid the complex multiple choice format.
• 3. Format the item vertically, not horizontally.
• Allow time for editing and revising items.
• 5. Use correct grammar, punctuation, and
  spelling.
• 6. Minimize examinee reading time in phrasing
  each item.
• Avoid trick items that mislead or deceive
  examinees
• into answering incorrectly.

23
Guidelines for Developing Multiple Choice Items

• General multiple choice item-writing (content)
• Base each item on a clearly stated objective or
  valid skill standard.
• Focus on a single problem.
• Keep the vocabulary as simple as possible.
• Avoid cuing one item with another keep items
  independent.
• Avoid over specific knowledge when developing the
  item.
• Avoid textbook, verbatim phrasing when developing
  the item.
• Avoid items based on opinions.
• Develop items that measure higher level thinking.
• Base items on important aspects of the content
  area no trivial material.
• Avoid potentially insensitive content or language.

24
Guidelines for Developing Multiple Choice Items

• Stem development
• 1. State the stem in either question form or
  completion form.
• When using the completion format, dont leave a
  blank for
• completion in the beginning or middle of
  the stem.
• Ensure that the directions in the stem are clear,
  and that
• wording lets the examinee know exactly
  what is being asked.
• Include only the material needed to make the
  problem clear.
• Don't add extraneous information.
• Word the stem positively avoid negative
  phrasing.
• If an item must be stated negatively,
  underline or
• capitalize the negative word.
• 6. Include the central idea and most of the
  phrasing in the stem.

25
Guidelines for Developing Multiple Choice Items

• General option development
• 1. Include four options. (Tip 3 options is
  fine!!)
• Place options in a logical order, if one exists
• (e.g., numerical, alphabetical).
• Keep options independent
• options should not be overlapping.
• 4. Keep all options in an item homogeneous in
  content.
• 5. Keep the length of options fairly consistent.
• 6. Avoid, or use sparingly, the phrase all of
  the above.
• 7. Avoid, or use sparingly, the phrase none of
  the above.
• 8. Avoid the use of the phrase I dont know.

26
Guidelines for Developing Multiple Choice Items

• General option development (continued)
• 9. Phrase options positively, not negatively.
• Avoid distractors that can clue test-wise
  examinees
• for example, avoid clang associations,
  absurd options,
• formal prompts, or semantic (overly
  specific or
• overly general) clues.
• Avoid giving clues through the use of faulty
  grammatical
• construction. (Grammatical inconsistencies
  involving
• "a," or "an," for example, give clues to
  the correct answer.)
• 12. Avoid specific determiners, such as never
  and always.

27
Guidelines for Developing Multiple Choice Items

• Correct option development
• Position the correct option so that it appears
• about the same number of times in each
• possible position for a set of items.
• Make sure there is one and only one correct,
• or clearly best, answer on which experts
• would agree.

28
Guidelines for Developing Multiple Choice Items

• Distractor development
• Use plausible distractors avoid illogical
  distractors.
• 2. Incorporate common errors of students in
  distractors.
• 3. Use familiar yet incorrect phrases as
  distractors.
• 4. Use true statements that do not correctly
  answer the item.
• 5. Avoid the use of humor when developing options.

29
Applying the Guidelines
30
Acceptable devices for measuring are A.
Scales B. Measuring cups C. Ladles D. All
the above
31
The income statement compares A. sales and
expenses. B. assets and liabilities. C. cash
flows. D. owners investments in the business
and withdrawals.
32
A budget _______. A. is a plan to match
expected income with expected expenses B. is a
plan to match spending with saving C. is a plan
to increase income D. is a plan to decrease
your wants and needs
33
A budget is a plan to ___________________. A.
match expected income with expected expenses B.
match spending with saving C. increase
income D. decrease your wants and needs
34
Workplace Skills
COMMUNICATING ON THE JOB Communicate orally
with others. Use telephone etiquette. Interpre
t the use of body language. Prepare written
communication. Follow written directions. Ask
questions about task.
35
Workplace Skills
A company's most reliable tool of internal
communication is the a. company
bulletin board. b. community daily
newspaper. c. local gossip network.
d. company newsletter.
36
Workplace Skills
Tom is employed at a toy store and his task today
is to assemble a bicycle. He should


a. read the directions
completely before starting.
b. read Step 1 and start assembling the bike.
c. look at the diagram
and start assembling the bike.
d. read the directions if he encounters
difficulties.
37
Workplace Skills
You are conducting business on a phone with no
hold button and must leave the phone unattended
to obtain some information. The best thing to do
with the phone receiver is to

a.
hang up.
b. place the receiver between
the rests. c. lay
the phone receiver flat in the middle of the
desk. d. put the receiver next to
a radio that is playing music.
38
S C E N A R I O S
From The VTECS Perspective
39
What are scenarios?

• A realistic work situation with
• pre-established criteria that measures an
  individuals achievement of
• career cluster
• foundation, pathway, or specialty
  knowledge/skills.

40
Two Types of Scenarios

• Performance with a Rubric
• Situation Multiple Choice

41
Performance with a Rubric
COMMUNICATIONS Skill F02.7
42
Situation Multiple Choice
You are an employee at the Eastside auto parts
store. The sales manager for the corporate
office has shared with the sales team this chart
which gives the sales rate for the first four
months of the current year for all three
storesEastside, Downtown, and The Mall. He
wants you to analyze how your store compares to
other stores in the company.

• Which store was consistently the most productive?
• The Mall store
• The Mall and Downtown were the same
• Eastside
• Downtown

• Based on this chart, what is the approximate
  difference between Eastsides March sales rate
  and its April sales rate?
• 30
• 40
• 50
• 60

43
Why use scenarios?
44
How many buses does the army need to transport
1,128 soldiers, if each bus holds 36 soldiers?
45
This is from a national mathematics assessment
for 8th grade students. Almost one-third of the
8th graders answered the question 31 remainder
12
46
So, why use scenarios?
Use scenarios . . . . . . for instruction to
help students put what they learn into a
context. . . . for assessment to provide
evidence of learning that reflects
worthwhile content.
47
Three Principles for Developing Well Designed
Scenarios

• 1 Scenarios require learners to transfer
  knowledge and skill learned in school to real
  life work situations.

48
Three Principles for Developing Well Designed
Scenarios

• 2 Scenarios place learners in a simulated job
  role and setting.

49
Three Principles for Developing Well Designed
Scenarios

• 3 The measurement criteria is critical to the
  evaluation of the scenario and provides the basis
  for the scoring guide (rubric or checklist).

50
SCENARIODEVELOPMENT
51
Planning/Organizing

• Identify Knowledge/Skill
• Identify Business Contacts

52
Developing the Scenario

• Obtain real situation through
• face to face contact with
• business/industry representative
• Obtain employer expectations for
• successful completion of tasks in the
• scenario.
• (continued)

53

• Developing the Scenario
• (continued)

• Write workplace content and identify
  knowledge/skills, set-up requirements, academic
  and workplace skills.
• Determine performance elements measurement
  criteria
• Develop scoring guiderubric or performance
  checklist based on measurement criteria.

54
SCENARIO FORMAT

• A. Scenario Title
•  B. Foundation Knowledge/Skills
  Pathway Knowledge/Skills
• Specialty Knowledge/Skills
• C. Workplace Context/Situation

55
SCENARIO FORMAT (continued)

• D.  Time for Completion
• E. Academic Skills Required
• F. Workplace Skills

56
SCENARIO FORMAT (continued)

• G. Notes to the Teacher
• H. Performance Elements Measurement Criteria
• I. Scoring Guide
• (Rubric/Checklist

57
Weve learned that . . .
It is critical that the . . .

• workplace context,
• performance elements,
• measurement criteria, and
• scoring rubric or checklist

. . . Are aligned.
58
Weve learned that . . .

• Scenarios should allow enough
  to be implemented in different localities or
  situations without compromising the content and
  criteria.

flexibility
59
Lets create one right now!!
60
So, how do you make a cup of coffee?
61
VTECS WEBSITE
www.vtecs.org
62
Building a BetterX! Test!
Presenter Brenda Hattaway hattaway_at_vtecs.org