Question setting

1
Question setting

• IEA Insight is a key support for IEA members.
• Through Insight you can tap into a range of
  materials in one place that can
• help you develop your own knowledge and skills
  in specific areas of assessment
• help you develop your teams knowledge and
  skills
• give you access to information that we hope you
  will find stimulating and informative.

2
Question setting
This short IEA Insight will give you a glimpse of
some key aspects of question setting More
detailed information on the subject is available
to IEA members in the members area of
www.ioea.org.uk This Insight will
cover Question types Some general points on good
practice Things to avoid
3
Question setting
Whether you are setting questions for assessment
in your own centre, class or tutor group or
involved in setting questions for public
examinations, there are still good and bad
approaches to the process.
4
Question setting
Compare two main characters from different works
of literature from different centuries. Good
question?
5
Question setting
Compare two main characters from different works
of literature from different centuries. There
is no evidence that Lady Macbeth liked fish
Tarka the Otter loved it.
Whereas Tarka was despatched with a spade, it
was a balcony wot done for Lady Macbeth.
6
Question setting
Poorly written questions have a negative impact
on the learners who are faced with them Poorly
written questions also create frustration amongst
assessors
7
Types of Question
Summative Assessment Multiple choice Short
answer Extended answer / essay Continuous
Assessment Tasks End of unit tests Extended essays
8
Types of Question
Knowing which type of question to use, when, is a
skill in itself
9
Multiple Choice Questions

• Benefits
• Objective marking requires no interpretation
  from marker
• Quick to answer
• Test a breadth of knowledge quickly
• Marking is fast and consistent
• Disadvantages
• Limited testing of skills
• Need very tight focus
• Longer to set than other question types
• Setting is a specialist skill

10
Multiple Choice Questions
Terminology Question stem Correct answer
key Choices options Incorrect answers
distractors
11
Multiple Choice Questions - Example
The capital city of Spain is (STEM) A Barcelona
(DISTRACTOR) B Madrid (KEY) C Valencia (DISTR
ACTOR) D Aragon (DISTRACTOR)
12
Multiple Choice Question Stems
Stems can be written in various ways Among
these are Completing a statement The capital
city of Spain is ... Logic and reasoning Which
of the following statements can be justified
using the information in the passage alone? Odd
one out Each of the following is a cause of
drought except ...
13
Negative Stems
Using the negative in a Stem can be
confusing There are three things wrong with
this question Which of the items in the
following list is not one of the major organs of
the human body? A Skin B Heart C Leg D Pancreas
14
Vague and Confusing Stems
Which of the items in the following list is not
one of the major organs of the human
body? A Skin B Heart C Leg D Pancreas Some
learners will focus on the word organ and
simply pick one out The word major has not
been defined The answer Leg is too obvious
15
Vague and Confusing Stems
This question stem is confusing Which of these
is not a good reason for the world economy to
slow down in the event of a major natural
disaster?
16
Vague and Confusing Stems
Which of these is not a good reason for the world
economy to slow down in the event of a major
natural disaster? It uses the negative The main
thrust of the question is separated from the
opening It is difficult to word a correct answer
and even harder to write distractors
17
Carefully Worded Odd One Out Questions
It is valid to test the ability to recognise
which items in a list stand out in some
way Using the format Each of the following is
... except can be a good way around the
confusing negative stem problem
18
True / False Questions
True / False questions are not true multiple
choice questions e.g. A basic principle of
physics is that energy can neither be created nor
be destroyed True or False?
19
True / False Questions
A better approach would be to ask A basic
principle is that energy A can be created but
not destroyed B can be destroyed but not
created C can be destroyed and created D cannot
be destroyed or created
20
True / False Questions
A basic principle is that energy A can be
created but not destroyed B can be destroyed but
not created C can be destroyed and
created D cannot be destroyed or created Even
here the correct answer is the only one to
contain the word cannot so it might need
further revision
21
Multiple Choice Questions Summary

• You have now seen some of the benefits of
  multiple choice questions.
• They can
• be objective
• be answered quickly
• test a breadth of knowledge
• be marked quickly and consistently

22
Short Answer Questions
SAQs can be marked By human markers - possibly
subject specialists By human markers who are not
specialists On screen On line Automatically -
if the list of possible outcomes is finite and
easily referenced
23
Short Answer Questions
Be clear in your demands Avoid ambiguity e.g.
Say how glaciers move SLOWLY And the old maths
classic
24
Short Answer Questions
are useful for developing points slightly
further than multiple questions would allow a
good way to explore a learners knowledge
quickly easily administered and marked - but not
always automatically are not Multiple Choice
Questions Essay titles If the point you wish to
test is ideally suited to another method then use
the other method
25
Extended Answer Questions
Traditional essay style questions Used to test
knowledge in depth Need to be carefully worded
to test what is actually required
26
Writing Extended Answer Questions
e.g. Give an account of Macbeth This pretends
to be an essay but is nothing more than a request
for a candidate to write out the plot. A low
order skill would actually be tested
27
Writing Extended Answer Questions
e.g. Macbeth is a good man who allows himself to
be tricked by the forces of evil. Discuss the
above statement with reference to at least three
scenes from the play. This is clearer in its
demands and gives an element of structure to the
task which was not present in the previous
question.
28
Structuring Extended Answer Questions

• Having said that ...
• Dont turn an Extended Answer Question into
  nothing more than a series of SAQs (Short Answer
  Questions)
• This can be the case if the question has
• too many parts
• e.g. Give two reasons why Britain sent troops to
  France in 1914 and three reasons why the USA
  entered the war and two outcomes of the treaty of
  Versailles.

29
Deriving Extended Answer Questions
The question is being asked in order for a
candidate to demonstrate knowledge and/or
skills The question should come naturally from a
programme of study and not be tacked on
artificially
30
Extended Answer Questions
can assess knowledge in depth probe into
skills should not lead to repetitive low order
tasks be unconnected to a programme of study
31
Showing Off
It might be interesting to set questions for the
local pub quiz which test esoteric
knowledge BUT Question setters should resist
the temptation to show the world how clever they
are by setting candidates ridiculously difficult
tasks If there is a clear form of words then use
it
32
Remember what you are testing
The question is being asked for a clear
reason It is in clear language It asks what it
seems to ask Deciphering the question should
not be the test
33
Teaching to the tests or testing what is taught?
Whilst the role of assessment remains very
important in school, college and workplace
education and training, we should all remember
that assessment is not an end in itself. Keep a
sensible balance between teaching / training and
the various forms of assessment. The IEA hopes
to engage members in wide ranging discussion
about the nature and the future of
assessment. You can find out more about question
setting in the members area of the IEA website -
www.ioea.org.uk