MGTO 324 Recruitment and Selections

1
MGTO 324 Recruitment and Selections

• Scale and Test Construction
• Kin Fai Ellick Wong Ph.D.
• Department of Management of Organizations
• Hong Kong University of Science Technology

2
Prologue

• In the last lesson, I have discussed the
  scientific elements in testing
• Today, we focus on how a test can be constructed
• In particular, you are expected to understand the
  following
• The concepts of item, scale, and test
• Different item formats
• How to write a set of good items
• Multiple-item scaling

3
Outline
4
Outline
5
Part I Basic Concepts

• What is a (psychological) test?
• Measurement device or technique
• To quantify behavior or aid in the understanding
  and prediction of behavior
• A set of items designed to measure
  characteristics of human beings that pertain to
  behavior

6
Part I Basic Concepts

• What is an item?
• A specific stimulus to which a person responds
  overtly
• What is the English word of ??
• You can manage well in interpersonal relationship
• Overt behaviors (Scientific Standard)
• Observable
• Measurable
• Can be replicated

1 2 3 4 5
Strongly disagree Strongly agree
7
Part I Basic Concepts

• What is a scale?
• The quantified scores obtained from a test
• The raw scores are related to some defined
  theoretical or empirical distribution
• The matching between the raw score and the
  theoretical meaning of that score
• E.g., 0oC freezing point 100oC boiling point
• The same theoretical meaning could be represented
  by different scales
• Temperature Degree Celsius vs. Degree Fahrenheit
• Length Meter vs. Feet Kilometer vs. Mile
• Weight lb vs. kg
• Wealth HK vs. US

8
Part I Basic Concepts

• What is a scale?
• Examples
• Thermometer
• Raw scores 2.4 cm
• Degree Celsius 100
• Theoretically (empirically) boiling point
• HKCEE
• Raw scores 87
• Grade A
• Theoretically (empirically) Excellent students
• IQ test
• Raw scores 2400
• IQ 130
• Theoretically (empirically) Gifted individual

9
Part I Basic Concepts

• Essential steps in scale and test construction
• Have a clear definition of what (i.e., the
  psychological construct) the test is supposed to
  measure
• E.g., Locus of control self-efficacy
• Generating a set of items that at least seems to
  capture the construct
• Determining the scale format
• Pilot tests to assess the face validity,
  reliability, construct validity, and criterion
  validity
• Deleting or revising less useful items
• Assessing reliability and validity again
• Revising again
• Assessing again
• Revising again
• Assessing again
• Revising again
• Assessing again
• Shorten the scale
• Assessing again.

10
Outline
11
Part II Item Formats
12
Part II Item Formats

• Dichotomous formats
• Offers two alternatives for each item
• True/False or Select a more appropriate
  statement
• You often spend more than three hours on typing
  every day
• Agree vs. Disagree
• Generally speaking, the salary for me accurately
  represents my contribution to the organization
• Agree vs. Disagree
• Which job, technical or administrative, do you
  prefer?
• Technical vs. Administrative
• Scores
• Simply count the number of items a person endorse
• Commonly used in both educational and personality
  tests

13
Part II Item Formats

• Dichotomous formats
• Some famous scales used dichotomous formats
• Locus of control (Rotter, 1966)
• the degree to which people believe they are
  masters of their own fates from the OB textbook
  (Robin, 2003)
• High in externality
• Less satisfied with jobs Higher absenteeism
  rates More alienated from the work Less
  involved in their jobs
• Choose one
• A. Many of the unhappy things in peoples lives
  are partly due to bad luck
• B. Peoples misfortunes result from the mistakes
  they make
• Choose one
• Heredity plays the major role in determining
  ones personality
• It is ones experiences in life which determine
  what one is like

14
Part II Item Formats

• Dichotomous formats
• Advantages
• Simple, easy to administer and score
• Absolute judgment people should declare one of
  the two alternatives
• Disadvantages
• Effect of memorizing materials
• Requires numerous items to produce reliable
  results (chance 50)

15
Part II Item Formats
16
Part II Item Formats

• Polytomous formats
• Offers more than two alternatives (e.g., usually
  4 to 5)
• One correct choice and other distractors
• Multiple-choice
• Scores
• Number of items correctly answered
• How many distractors?
• More distractors may not be better than less
  distractors (Sidick, Barrett, Doverspike, 1994)
• Less influenced memory effects (relative to
  dichotomous format)
• Mainly used in educational tests

17
Part II Item Formats
18
Part II Item Formats

• Likert-type formats
• Usually, people are required to express their
  degree of dimension on a statement
• I can manage well in interpersonal relationship
• Monotone items
• A higher score suggests higher agreement

1 2 3 4 5
Strongly disagree disagree neutral agree Strongly agree
19
Part II Item Formats

• Likert-type formats
• Number of alternatives
• Odd (5, 7, or 9)
• Even (6, or 10) avoid midpoint
• Popular in psychological tests
• It can be subject to various psychometric
  analyses, such as factor analysis
• For example, the General Self-efficacy Scale
• Self-efficacy
• The individuals belief that he or she is
  capable of performing a task from OB textbook
  (Robin, 2003)
• Higher self-efficacy
• Less likely to give up under difficult situations
• Usually perform better than

20
Part II Item Formats
21
Part II Item Formats

• Cumulative (Guttman) formats
• Items on the same dimension are set up in
  ascending order
• Subject with a particular attitude will agree
  with all items on one side of that position and
  disagree with other items
• Example Addition, long-division, and calculus
• Monotone items

22
Part II Item Formats

• Cumulative (Guttman) formats

23
Part II Item Formats

• Cumulative (Guttman) formats

24
Part II Item Formats
25
Part II Item Formats

• Cumulative (Guttman) formats
• Advantages
• A single score carries complete information about
  the response patterns
• Calculus OK division OK addition OK
• Marriage OK being neighbor OK
• When there is no random error
• Provides a test of the unidimensionality of what
  are to be tested
• Cumulative response pattern will not be obtained
  when the items do not measure only one dimension
• Disadvantages
• Problems resulted from random error
• Difficult to find domains that are unidimensional
• Less popular than Likert-type format

26
Outline
27
Part III Writing good items

• Define clearly
• Clearly define what you want to measure
• Check the face and content validity
• Is the items out of syllabus
• How many psychological factor I want to measure?
  List them all
• I want to develop a test that helps me hire
  employees who have strong self-learning tendency
• What do you want to measure from the test?
• What is self-learning tendency?
• Give a clear definition before moving to the next
  step

28
Part III Writing good items

• Clearly think about the item formats
• Think carefully what type of tests as well as
  what statistical analyses you want to use
• Some statistical analyses may not be appropriate
  for certain item formats
• Rank order is not appropriate for t test
• Polytomous responses may not be able to be
  analyzed by Factor Analysis
• Likert-type or dichotomous format seems to be a
  good choice as the default

29
Part III Writing good items

• Sources of items
• Discourse and text
• From brainstorming or informal conversation
• Asking others the meaning of self-learning
• Use qualitative method is a more systematic
  method
• Open-ended interview or question
• Focus group
• Content analysis
• Newspaper
• Classic literature
• The goal is to generate a pool of items that
  seems to measure what we want to measure
• E.g., When I have problems, Ill seek help from
  books prior to people.

30
Part III Writing good items

• Nature of items
• The meanings of items should be clear,
  straightforward, and can be easily understood
• Avoid exceptionally long items
• Avoid reading difficulty
• Dont use jargon.
• Social desirability
• I like self-learning
• Offensiveness
• Pay attention to the problems of sexism and
  racism
• Avoid double-barreled items
• seeking help from people is a better method than
  seeking help from books because people are more
  accessible.
• Reverse items
• Most knowledge could not be learned without the
  help from others
• I can learn almost all knowledge by myself

31
Part III Writing good items

• Generate an item pool
• You first need to generate a set of item pool
• The numbers of items in this pool are usually
  much more than the final version of the scale
• Do a preliminary test
• Select useful items
• Discard or revise other items (you will learn
  these skills later in this course)
• You may need several rounds of revision before
  the scale becomes reliable and valid

32
Part III Writing good items

• What should be included in a test apart from the
  basic items?
• Clear instruction for subjects
• Give examples whenever possible
• Questions for demographic information
• Age, gender, education level, etc.
• Declare that how the collected information will
  be used
• Is it confidential? The purpose of collecting the
  data who will assess the data, etc.

33
Outline
34
Part IV Multiple-item scaling

• True score theory
• Due to the matter of precision, measures may vary
  from time to time
• The true scores can hardly be obtained by only
  one measure
• Sometimes it may overestimate or underestimate
  the true score, the errors are assumed to be
  randomly distributed
• The true scores can be obtained by averaging
  multiple responses

35
Part IV Multiple-item scaling

• The concepts of multiple-item scaling
• I want to measure ones attitude toward whether
  secondary students can have romantic
  relationships
• Single item
• I encourage secondary students to develop
  romantic relationship
• Two items
• I encourage secondary students to develop
  romantic relationships
• We should respect the secondary students freedom
  of engaging in romantic relationships
• The issues of reliability and validity

36
Part IV Multiple-item scaling

• Advantages multiple-item scaling
• It allows testing the nature of a construct
• A set of items may capture more than one
  dimension
• Intelligence
• Memory span, verbal ability, and visual-spatial
  ability
• Items measuring same dimension or same construct
  should be highly inter-correlated
• Items measuring different dimensions are supposed
  to have relatively low inter-item correlation
• It allows us to check scale validity through
  Factor analysis
• Discuss again later
• Increase reliability and validity
• Remember the concept of true score theory
• Random errors may exert impacts on a particular
  single response
• Effects of random errors can be neutralized by
  measuring multiple responses