Building a Better

1
Building a BetterX! Test!
5th Annual Career Cluster Institute Phoenix,
AZ June 12, 2007 Session E41

• Presenter
• Ms. Brenda Hattaway
• Assistant Executive Director
• VTECS Atlanta, Georgia

2
Critical Component 9 Multi-Measure Assessment
3
Since 1973 . . .
A Consortium for Innovative Career and
Workforce Development Resources
4
A Consortium of States
5
VTECS develops resources . . .
. . . for CTE.
Tools
6
Procedures (Steps)
Item Banks
Academic Skills
Instructional Tools
Objectives
7
(No Transcript)
8
Does the pilot test process we use really tell us
anything about how well our instruction works?
What types of tasks characterize the typical day
of a submarine officer?
Personality inventories indicate that our
programmers tend to have higher extroversion
scores that introversion.
List the steps in the accounting cycle.
9
How can we improve student learning in the
classroom and raise student performance on
high-stakes tests? The key is continuing
assessment and evaluation throughout the school
year, as well as a commitment to the success of
all students . . .
Using Data to Drive Student Achievement in the
Classroom and on High-Stakes Tests
T H E Journal (Technological Horizons in
Education) January, 2003
10
1
Why are you testing in the first place?
11
2
Communicate expectations to your students!
12
3
Make Sure Your Students Know the Material

• 1. Create an assessment instrument that looks
• like the job, not one that looks like other
• tests that you have taken or seen.
• Keep test items and instruction directly
• related to the skill standards.
• 3. Assess beyond the recall/memorization level.

13
4
Write Good Objectives Develop Great Test
Items!
14
Objective (Example)
CONDITION PERFORMANCE STANDARD (CRITERIA)
Using a pencil, ruler, tape and a single edge
razor, mount photographs for framing with
a margin of 1/4 inch on all sides.
15

• Review of Item Writing Guidelines
• Multiple Choice
• Performance
• Scenario

16
A Comment on True / False Test Items
17
Multiple Choice
18
Guidelines for Developing Multiple Choice Items

• General multiple choice item-writing (procedure)
• 1. Use either the best answer or the correct
  answer format.
• 2. Avoid the complex multiple choice format.
• 3. Format the item vertically, not horizontally.
• Allow time for editing and revising items.
• 5. Use correct grammar, punctuation, and
  spelling.
• 6. Minimize examinee reading time in phrasing
  each item.
• Avoid trick items that mislead or deceive
  examinees
• into answering incorrectly.

19
Guidelines for Developing Multiple Choice Items

• General multiple choice item-writing (content)
• Base each item on a clearly stated objective or
  valid skill standard.
• Focus on a single problem.
• Keep the vocabulary as simple as possible.
• Avoid cuing one item with another
• keep items independent of one another.
• Avoid over specific knowledge when developing the
  item.
• Avoid textbook, verbatim phrasing when developing
  the item.
• Avoid items based on opinions.
• Develop items that measure higher level thinking.
• Base items on important aspects of the content
  area
• avoid trivial material.
• Avoid potentially insensitive content or language.

20
Guidelines for Developing Multiple Choice Items

• Stem development
• 1. State the stem in either question form or
  completion form.
• When using the completion format, dont leave a
  blank for
• completion in the beginning or middle of
  the stem.
• Ensure that the directions in the stem are clear,
  and that
• wording lets the examinee know exactly
  what is being asked.
• Include only the material needed to make the
  problem clear.
• Don't add extraneous information.
• Word the stem positively avoid negative
  phrasing.
• If an item must be stated negatively,
  underline or
• capitalize the negative word.
• 6. Include the central idea and most of the
  phrasing in the stem.

21
Guidelines for Developing Multiple Choice Items

• General option development
• 1. Include four options. (Tip 3 options is
  fine!!)
• Place options in a logical order, if one exists
• (e.g., numerical, alphabetical).
• Keep options independent
• options should not be overlapping.
• 4. Keep all options in an item homogeneous in
  content.
• 5. Keep the length of options fairly consistent.
• 6. Avoid, or use sparingly, the phrase all of
  the above.
• 7. Avoid, or use sparingly, the phrase none of
  the above.
• 8. Avoid the use of the phrase I dont know.

22
Guidelines for Developing Multiple Choice Items

• General option development (continued)
• 9. Phrase options positively, not negatively.
• Avoid distractors that can clue test-wise
  examinees
• for example, avoid clang associations,
  absurd options,
• formal prompts, or semantic (overly
  specific or
• overly general) clues.
• Avoid giving clues through the use of faulty
  grammatical
• construction. (Grammatical inconsistencies
  involving
• "a," or "an," for example, give clues to
  the correct answer.)
• 12. Avoid specific determiners, such as never
  and always.

23
Guidelines for Developing Multiple Choice Items

• Correct option development
• Position the correct option so that it appears
• about the same number of times in each
  possible
• position for a set of items.
• Make sure there is one and only one correct,
• or clearly best, answer on which experts
  would agree.

24
Guidelines for Developing Multiple Choice Items

• Distractor development
• Use plausible distractors avoid illogical
  distractors.
• 2. Incorporate common errors of students in
  distractors.
• 3. Use familiar yet incorrect phrases as
  distractors.
• 4. Use true statements that do not correctly
  answer the item.
• 5. Avoid the use of humor when developing options.

25
Applying the Guidelines
26
Incorrect and improper collection of samples for
bacterial testing could result in
what type of test result(s)?

a. negative

b. false positive
c. false negative
d.
b and c
Food Science/Processing Occupations
27
The primary type of milk consumed in the United
States is obtained from dairy
cows. However, other domestic animals are
raised in the U.S. and other countries for
their milk production. Some of these
animals are

a. goats.

b. sheep.
c. camels.

d. all of the above.
Food Science/Processing Occupations
28
An example of a cultured milk product is


a. yogurt.
b. cheese.
c.
buttermilk.
d. all of the above.
Food Science/Processing Occupations
29
Workplace Skills
COMMUNICATING ON THE JOB Communicate orally
with others. Use telephone etiquette. Interpre
t the use of body language. Prepare written
communication. Follow written directions. Ask
questions about task.
30
A company's most reliable tool of internal
communication is the a. company
bulletin board. b. community daily
newspaper. c. local gossip network.
d. company newsletter.
Workplace Skills Communicating on the Job
31
Workplace Skills
COMMUNICATING ON THE JOB Use telephone
etiquette.
You are conducting business on a phone with no
hold button and must leave the phone unattended
to obtain some information. The best thing to do
with the phone receiver is to

a.
hang up.
b. place the receiver between
the rests. c. lay
the phone receiver flat in the middle of the
desk. d. put the receiver next to
a radio that is playing music.
32
Workplace Skills
COMMUNICATING ON THE JOB Follow written
directions.
Tom is employed at a toy store and his task today
is to assemble a bicycle. He should


a. read the directions
completely before starting.
b. read Step 1 and start assembling the bike.
c. look at the diagram
and start assembling the bike.
d. read the directions if he encounters
difficulties.
33
S C E N A R I O S
From The VTECS Perspective
34
What are scenarios?

• A realistic work situation with
• pre-established criteria that measures an
  individuals achievement of career cluster
• foundation, pathway, or specialty
  knowledge/skills.

35
Why use scenarios?
36
How many buses does the army need to transport
1,128 soldiers, if each bus holds 36 soldiers?
37
This is from a national mathematics assessment
for 8th grade students. Almost one-third of the
8th graders answered the question 31 remainder
12
38
So, why use scenarios?
Use scenarios . . . . . . for instruction to
help students put what they learn into a
context. . . . for assessment to provide
evidence of learning that reflects
worthwhile content.
39
Three Principles for Developing Well Designed
Scenarios

• 1 Scenarios require learners to transfer
  knowledge and skill learned in school to real
  life work situations.

40
Three Principles for Developing Well Designed
Scenarios

• 2 Scenarios place learners in a simulated job
  role and setting.

41
Three Principles for Developing Well Designed
Scenarios

• 3 The measurement criteria is critical to the
  evaluation of the scenario and provides the basis
  for the scoring guide (rubric or checklist).

42
SCENARIODEVELOPMENT
43
Planning/Organizing

• Identify Knowledge/Skill
• Identify Business Contacts

44
Developing the Scenario

• Obtain real situation through
• face to face contact with
• business/industry representative
• Obtain employer expectations for
• successful completion of tasks in the
• scenario.
• (continued)

45

• Developing the Scenario
• (continued)

• Write workplace content and identify
  knowledge/skills, set-up requirements, academic
  and workplace skills.
• Determine performance elements measurement
  criteria
• Develop scoring guiderubric or performance
  checklist based on measurement criteria.

46
SCENARIO FORMAT

• A. Scenario Title
•  B. Foundation Knowledge/Skills
  Pathway Knowledge/Skills
• Specialty Knowledge/Skills
• C. Workplace Context/Situation

47
SCENARIO FORMAT (continued)

• D.  Time for Completion
• E. Academic Skills Required
• F. Workplace Skills

48
SCENARIO FORMAT (continued)

• G. Notes to the Teacher
• H. Performance Elements Measurement Criteria
• I. Scoring Guide
• (Rubric/Checklist

49
Weve learned that . . .
It is critical that the . . .

• workplace context,
• performance elements,
• measurement criteria, and
• scoring rubric or checklist

. . . Are aligned.
50
Weve learned that . . .

• Scenarios should allow enough
  to be implemented in different localities or
  situations without compromising the content and
  criteria.

flexibility
51
VTECS WEBSITE
www.vtecs.org
52
Building a BetterX! Test!
Questions?