Quantitative Methods

1
Quantitative Methods

• Survey Development
• Prof. Paul Licker, Ph. D.

2
Agenda

• Survey Research
• Instrumentation
• Stages of Instrumentation
• Construct-tion
• Generation
• Pretesting
• Pilot Testing
• Item Analysis
• Reliability Assessment
• Validity Assessment

3
Survey Research

• Gathering data about a phenomenon using a
  statistical approach by observing directly or
  indirectly some subset of the set of phenomena
• Often uses individuals recollections or
  perceptions of events or their evaluations of the
  events

4
Survey Data Gathering Methodologies

• Personal Interviews
• Group Interviews
• Phone Interviews
• Mail-out Questionnaires
• Hand-out Questionnaires
• Clip-and-Mail Questionnaires
• Survey of Secondary Data

5
Survey Methodologies Reasoning

• The goal of our research is to describe or relate
  the behaviour, opinions, impressions, knowledge,
  etc. of people in situations.
• If we cannot directly observe them, their
  behaviour, etc., we must rely on their own
  (subjective) observations of themselves.
• Individuals and their actions and opinions are
  unique, however. No single self-observation can
  be generalised to everyone.

6
Survey Methodologies Reasoning-2

• In order to describe the general situation, we
  aggregate a series of measurements, observations
  or judgments about individual (subjective
  impressions of) events.
• The objective description is the average of all
  these subjective descriptions.
• We assume all biases average out and compensate
  for individual differences.

7
Strengths...andWeaknesses? ?

• Friendly, familiar, cheap
• Relatively quick
• Useful if survey questions are clear
• Ideal for Descriptive research
• Uses common skills

• Hard to control
• Samples are often very large
• Has hidden flaws that can be easily ignored
• Cannot easily be used to draw causal inferences
• Retrospective

8
Example Questionnaires

• Questions are written out
• Highly structured
• Sent out or handed out to a large group
• Response is on paper, to be mailed or handed back
• Anonymity, ease of reading and response, and
  clarity are the key qualities
• Can use the web or email for distribution

9
Questionnaire Procedure

• Create Sample of Questions
• Pretest Questions
• Derive Sampling Frame, Draw Sample
• Distribute Questionnaires
• Motivate Response
• Enable Response
• Collect Responses
• Code and Record Responses

10
1. Mail-Out Questionnaires

• Determine Sampling Frame
• Create Package, with response motivator
• Pretest all questions
• Expect 25 for public campaign, 50 for
  in-company campaign response
• You must explain non-responses

11
Mail-out Questionnaire Example

• A sample of 1300 members of the ASM was mailed a
  questionnaire concerning their careers in IS.
  The questionnaire was pretested on 20 ASM members
  and fellow academics. Only 256 replied on the
  first wave. A second wave of 1300 was sent out
  and an additional 162 replied. Of these 402 were
  actually usable. Eighteen responses came in
  after the research was written up.

12
2. Hand-Out Questionnaires

• Get company permission
• Use sampling frame
• Distribute through company, collect through
  company
• Expect at least 50 response rate
• Watch interaction with company situation in terms
  of date, time, season
• Confidentiality is a BIG concern

13
Hand-out Questionnaire Example

• Using departmental secretaries, ALL 503
  professional and managerial staff at a large
  financial company were given questionnaires on
  their perceptions of the role of IS in corporate
  success. A week later the secretaries attempted
  to collect the responses only 478 were collected
  and returned of these 353 were useful.

14
3. Mail-in Questionnaires

• Subjects come across the questionnaire and are
  self-motivated to complete it and return it (see
  also electronic variants)
• Suffers from strong self-selection problems only
  those who are motivated to return it do sothey
  may have an axe to grind.
• Not considered valid research
• Similar to radio talk shows.

15
Mail-in Questionnaire Example

• Readers of the Business Times were encouraged to
  fill out, clip, and return a questionnaire about
  their desires to leave South Africa. 74 of
  those who professional people who responded said
  they would leave if they could, which was widely
  reported and quoted. In fact, those who arent
  intending to leave would never clip the coupon,
  giving vastly inflated positive responses.

16
Examples of Question Format
5. N (4, 5, 6, 7-) Point Scale 6.
Grid 7. Filter If xxxx skip to 9 8.
Distribution of 100 A__ B__ C__ D__
Survey Questionnaire
1. Fill in the blank __________ 2. Multiple
Choice (one/more) First Third
Fifth Second Fourth Sixth 3. Rank
Order Item 1 ___ Item 2 ___
Item 3 ___ Item 4 ___ 4. This one is
___Yes ___ No
People will make errors!
No Low Mod Hi V.Hi

A B C D E F
Various choices for each item

Give clear di-rections!
17
Questionnaire StrengthsandWeaknesses

• Relatively inexpensive
• Can collect lots of data
• Easy to do
• Thought to be quick and cheap
• Management is relatively simple
• Data need little interpretation
• No noverbals?????

• Questions must be ironclad
• Who is responding
• What about non-response?
• Followup is hard
• Can turn out to be expensive
• May require artistry, presentation important

18
Problems to Watch Out For

• Low response rates must be explained
• Incomplete response must be handled
• Lack of Pretesting
• Bad questions (ambiguous, meaningless)
• Open-ended questions interpretation
• Self-selection
• Who is responding

19
Electronic Variants of Questionnaire Surveys

• Web (requires programming, artistry)
• Discussion Groups (may annoy lots of people)
• Electronic Mail (can destroy anonymity)
• Can gather information quickly over a wide
  geographical area
• Suffers from self-selection
• Has a halo effect ( or -) on IS issues
• Sampling frame is a problem
• Easy to do very badly after a lot of effort

20
Questionnaire Diagnostic-1

• A mail-out questionnaire is sent to 250 customers
  of a bank to find out expectations concerning
  E-commerce. The questionnaire contains 142
  fill-in-the-blanks questions about finance,
  technology, banking, and customer expectations of
  possible E-commerce ventures. Can you expect
  some problems?

21
Questionnaire Diagnostic-2

• Almost 1000 questionnaires are handed out to
  microcomputer users (obtained from Computer
  Services) through secretaries in a government
  department for research examining user-IT
  relations. The six-page questionnaires are to be
  picked up in one week by the secretaries. Only
  151 of the questionnaires are returned. Will
  there be any problems in interpretation of the
  data?

22
Questionnaire Diagnostic-3

• Six companies are simultaneously switching to SAP
  and you want to get impressions from IT staff
  concerning the switch, so you send out two
  questionnaires (during the switch and 12 mo.
  after) by mail to all IT staff. Mostly
  open-ended questions, the survey has a response
  rate of 4. You determine that respondants dont
  differ demographically from non-respondants.
  Whats the problem here?

23
Instrumentation

• The development of valid and reliable instruments
  to measure a phenomenon.
• Instrument consists of items.
• Items are stimuli to respondents
• Items must correspond individually or in sets to
  constructs
• Respondents must be able to give valid
  responses in a reliable way.
• Instrument should be efficient.

24
Stages of Instrumentation

• Specify domain of constructs
• Generate sample of items
• Pretest and Pilot test for readability and
  content validity
• Purify instrument, remove extraneous items
• Assess validity (four types)
• Assess reliability or internal consistency

25
Constructs
An instrument measures constructs The instrument
suffers from the limitations of ALL instruments,
namely Jiggle (accuracy) Refinement
(precision) Failure (external reliability) Consi
stency (internal reliability) Ease of use, etc.
26
Instrumentation
Instrumentation is the task of building an
instrument The instrument must be appropriate for
the domain of the construct The instrument must
be valid and reliable The instrument must be
efficient and productive The instrument must be
cost-effective
27
Instrument Use Process
What the Instrument says
The instrument is a lens through which to view
the phenomenon
Problem areas
use
Instrument
quality
Phenomenon (Human Behaviour or Experience)
locus
28
Instruments Measure Proxies
Theory
Concept/ Construct
World of Ideas and Concepts World of Reality and
Measurement
Actual instantiation of the theoretical concept
A variable that is highly correlated (generally
causally) with the proxy
Proxy
Surrogate
29
Measurement Challenges
1. Conceptualising precisely to construct2.
Instantiating (operationalising step 1)
constructs3. Measuring proxies or determining
available surrogates that can be measured
reliably4. Demonstrating to others satisfaction
that steps 1 to 3 can be performed validly
Proxy
Surrogate
Valid, reliable measurements
30
Interpretation Challenges
1. Determining that the reverse correspondence
among proxies, surrogates and constructs is
logically valid.2. Using that correspondence,
interpret corresponding relationships among
constructs (i.e., concepts) in the theory3.
Reason whether or not the implied relationships
among the constructs supports or denies support
to your theory
Theory
Concept/ Construct
Valid, reliable measurements
31
Types of Validity
Construct Is the instrument actually measuring
the construct or events or is it its out- comes
actually an artifact of the instrument
Predictive Can the instrument dis- tinguish
different cases (such as with control variables)?
Discriminant Do the items making up the
variables correlate strongly within factors?
Content/Face Does the instrument adequately
cover the content of the con- structs? Are the
items repre- sentative of the content? Is
the instrument constructed sen- sibly?
Convergent Do the items converge on the
constructs?
32
Types of Reliability
External Instrument can be applied
successful- ly in the physical domain
desired Instrument is stable in its usage from
use to use (intersubject)
Internal All elements of a scale
consistently measure the desired construct within
the instrument (within subject)
33
Construct-tion

• The goal is to construct an instrument consisting
  of variables that are internally reliable, valid
  reflections of the constructs and practical to
  administer (in the sense that valid responses can
  be collected at affordable cost)
• Primary tools are argument, debate, pretesting,
  pilot testing, correlational analysis

34
Generation of Items

• Literature survey
• Supervisor
• Your own intuitiion
• Create a large pool, dont worry about initial
  size
• State items in comparable way, if building scales

35
PreTesting

• Use a panel of judges, experts in the field,
  people familiar with the culture of informants
  (respondents)
• Can use other doctoral students, lecturers,
  industrial sponsors, etc.

36
Pilot Testing

• Test for readability
• Test for respondability (reliability)
• Purpose is to create a smaller set of items by
  eliminating those that pose problems.
• Such problems include jargon, form and format,
  grammar, ambiguity, multidimensionality (two or
  more questions in one), cultural no-nos, language
  level, assumed intelligence, etc.)

37
Readability Procedures
1. Read the instrument yourself. Can it be
read? 2. Have your supervisor and committee read
the instrument 3. Pilot the instrument on a
captive group first. Do a protocol analysis on
this group. Speak with them as they hear each
question and go over its possible meanings. Can
they begin to answer the questions? 4. Pilot the
instrument on a representative small group of
respondents.
38
Item Analysis (Purification)

• Make sure that the factors apparent in the
  instrument are in fact pure and do not include
  extraneous factors that are merely artifacts of
  the questions themselves.
• Get rid of garbage items
• Use item-scale correlation to calculate
  Cronbachs Alpha and eliminate those whose
  elimination will not lower the Alpha.

39
Assessing Reliability

• Reliability is of two types
• Item-scale (internal consistency)
• Test-retest (external consistency)
• Overall alpha of 0.7 or 0.8, even 0.9 should be
  found.

40
Assessing Validity

• Factor analysis
• Do items load purely on one factor only?
• Do any items fail to load on any factors?
• Rule of thumb is to use factor loading of 0.5 as
  criterion

41
The End Product
Internally consistent
An exhaustive series of variables
The Instrument
Mutually Exclusive
42
The End Product
That reflect the underlying constructs of your
theory
The Instument