INFO4990: IT Research Methods Experimental Research (Joseph Davis) 9/5/2005

1
INFO4990 IT Research MethodsExperimental
Research(Joseph Davis) 9/5/2005

• Outline
• Strategy of experimental research (Platt)
• What are experiments and how to do experimental
  research,
• Different kinds of experimental designs.

2
Experimental Research Strategy(Platt)

• Systematic and precise method of scientific
  thinking
• Accumulative method of inductive inference
• Can contribute to rapid scientific progress

3
(No Transcript)
4
Steps in Experimental Research

• Devise alternative hypotheses (Existing theory)
• Devise crucial experiments with alternative
  possible outcomes, each of which exclude one or
  more possible hypotheses, (Experiment)
• Conduct the experiment, get a clean result,
  (Outcome)
• Back to step 1, making sub-hypotheses, or
  sequential hypotheses to refine the
  possibilities, exclusion and induction (exclusion
  and building the inductive (logical) tree)

5
Origins of the experimental approach

• Francis Bacon (interconnecting theory and
  experiment) building the conditional inductive
  tree, consecutive inductive inferences)
• Karl Popper, falsificationism, there is no such
  thing as proof in science, science advances
  through a series of disproofs or falsification.
  Assertions in science have to be falsifiable,
• .. it must be possible for all empirical
  scientific systems to refuted by experience.
• Fisher s work in the 1930s and 40s in the area
  of statistical inference.

6
Experimental Research Methods

• What is an experiment?
• Manipulation of one or more variables by the
  experimenter to determine the effect of this
  manipulation on another variable.
• Carefully designed and executed plan for data
  collection and analysis to test specific
  hypotheses.
• Examples of hypotheses?
• Well-designed experiments can permit causal
  inferences to be made.

7
Experiment Features

• Typically held at the discretion of the
  researcher.
• Ability to use various controls to isolate
  sources of variation.
• Ability to explore cause-effect relationships.

8
Phases in Experimental Studies

• Formulation of one or more hypotheses.
• Usually deductions or derivations from
  theoretical explanations (of the behavioural
  phenomenon) or strong hunches/speculations.
• Translation of the hypotheses into a set of
  treatment conditions and appropriate experimental
  design.
• Conduct the experiment, collect the data
• Statistical analysis of the data, interpretation
  of the results and writing up.

9
Experimental Design

• Independent Variables
• The variable(s) manipulated in the experiment.
• (also called manipulated variable, treatment
  variable or factor).
• Typically nominal (categorical) variable.
• Dependent Variable(s)
• Measure(s) that capture (performance) of the
  phenomenon

10
Control or Nuisance Variables

• Undesired sources of variation in an experiment
  that affect the dependent variable measurement,
• Typically of three types
• -organismic
• -environmental
• -experimental task.

11
Approaches to control the nuisance variable

• Random assignment of subjects to treatment
  groups,
• Holding the (pre-identified) nuisance variable
  constant for all subjects,
• Statistical control using Analysis of Covariance
  (ANCOVA).

12
Sources of Bias

• Experimenter cues
• Demand characteristics
• Evaluation apprehension
• Hawthorne Effect
• Negativistic subject

13
Experiments - Advantages

• Possibility of a variety of manipulative and
  statistical controls,
• Random assignment of subjects greater precision
  and higher confidence in specifying and testing
  causal relationships,
• Manipulation Checks possible.
• May help identify issues and problems previously
  unrecognised.

14
Experiments - Disadvantages

• Problems associated with lab settings,
• Some phenomenon cannot be studied under
  controlled conditions,
• Limitations imposed by moral concerns.

15
Procedures

• Getting Ethics Committee Approval
• Cover Story description and purpose
• Recruiting participants
• Sample selection
• Reference to criterion population
• Remuneration and motivation
• Training the participants
• Preparing the setting
• Controlled manipulation of independent
  variable(s)
• Manipulation checks
• Precise measurement of dependent variable(s).

16
Reviewing a Paper that used Experimental Research
Method

• Theoretical Background
• Key Hypotheses
• Experimental Design
• - Treatment (Independent) Variable(s)
• - Dependent Variables
• - Nuisance Effects
• - Manipulation checks
• Results robustness and value
• Alternative Research Design how would you do it
  differently?

17
Types of experimental Designs

• Completely Randomised Designs (CR-p) with one
  treatment.
• (single treatment with p levels)
• Randomised Block Designs (RB-p)
• Isolating the effects of nuisance variables by a
  blocking procedure whereby subjects who are
  relatively homogeneous w.r.t to the nuisance
  variable are assigned to the same block
• Completely Randomised Factorial Designs (two or
  more treatments evaluated simultaneously
• Repeated Designs

18
Quasi-experimental Designs

• Used when it is impossible or difficult to
  perform true, controlled experiments,
• Particularly in organisational settings
• Essentially compromise designs
• Used when time ,cost, and practicality are
  critical.
• One shot design
• One group pretest-posttest design
• Static Group Design

19
Observational Methods
20
Survey Methods

• Typically questionnaire surveys
• Strengths
• Quantitative data
• Transparency and some level of objectivity
• Succinct and easily understood
• Comparability/reproducibility
• Can deal with complex problems/issues

21
Survey Methods

• Surveys
• Interviewer completion
• Respondent completion
• Types of surveys
• Household
• Telephone
• Mail
• Customer
• Captive Group
• Organisation
• Web-based

22
Survey Methods

• Issues
• Response rates
• Biases and errors

23
Survey Methods

• Types of Questions
• Open ended and pre-coded
• Measurement of Attitudes and opinions
• Likert Scales
• Attitude Statements
• Semantic Differential

24
Survey Methods

• Questionnaire Construction
• Validity/Reliability issues
• Pilot Testing
• Sample selection
• Coding the data
• Statistical Analysis.