Quantitative Research Methods

1
Quantitative Research Methods

• Session 1
• What on earth are we trying to do here?
• Helen Petrie

2
Preamble

• The course is not just about how to do
  statistical tests and which ones to choose
• Will introduce a much broader approach to
  quantitative research methods
• Why do we need this?

3
Examples of research disasters/problems

• Rats in mazes example left or right turns?
• Creating technology for a problem people dont
  have (very prevalent in assistive technology)
• Explaining a difference that doesnt exist?
  (worst case execution time analysis - WCET)

4
and an example I will draw on

• Do people have difficulty navigating websites?
• What characteristics of the navigational
  facilities alleviate these problems?
• What inconsistencies in these facilities
  contribute to these difficulties?

5
A starting point

• What is the science in computer science about?
• What does science mean to you?
• Write down what makes computer science a science

6
What should make it a science

• The use of the scientific method
• the collection of data through observation and
  experimentation, and the formulation and testing
  of hypotheses (Websters Dictionary)
• Is this generally what we are doing in computer
  science?

7
Why has the scientific method got to do with
Quantitative Research Methods?

• If you are interested in QNT, then you must be
  interested in undertaking something related to
  the scientific method (SM), although maybe you
  dont realize this
• Considering the whole philosophy and practicality
  of the SM should help you to do it more
  accurately, thoroughly and elegantly

8
First pass at defining the scientific method

• Science is a structure built upon facts
  (Davies, 1968)
• Empiricism
• Observe (lots of) facts --gt build a theory to
  explain them
• Galileo often cited here - dropping objects off
  Leaning Tower of Pisa (repeated on the moon)

9
What are the facts???

• Is this man about to step onto the surface of the
  moon or a secret NASA film set?
• Are the shadows correct, where are the stars in
  the sky, did the flag flutter, where is the blast
  crater, etc etc
• The observations/facts can be hotly disputed
• Excellent discussion on wikipedia of the moon
  landing conspiracy including application of the
  scientific method to the debate

10
The facts depend on many things

• Could get into a whole discussion of whether you
  see red the way I see red
• Apart from people who have colour vision
  deficiencies (and there are a lot of them), as
  long as we agree to call a particular range of
  the spectrum red, lets not stress this one

11
Facts for experts, facts for novices

• More interestingly, what an expert /scientist
  sees may differ from what a novice sees
• e.g. X-rays, K mesons, finger print whorls etc
  etc

12
Scientific method v2

• What you observe and how you describe it is
  shaped (I wont say determined) by your
  theoretical framework
• So
• Observe facts within a framework -gt further
  develop theory via induction -gt make predictions
  via deduction

13
Small example

• A number of descriptive/relational studies show
  that people have difficulty navigating websites
  when the navigational bars are inconsistent in
  their location through a website
• (by induction)
• People need consistency in navigational
  mechanisms
• (by deduction)
• People will have more difficulty and find a
  website less acceptable if the navigation is
  inconsistent

14
Logical induction

• (not the same as mathematical induction)
• Particular -gt general/universal
• All lectures I have attended are boring.
• Therefore all lectures are boring.
• Problem of generalisability (highly important to
  the SM)
• I have only attended lectures by some lecturers
  (a sample out of the population of lecturers) so
  my logic may be flawed

15
Flawed logic

• Notice that I deliberately did that when I gave
  my example on navigation
• Studies show that people have difficulty
  navigating websites when the navigation bars are
  inconsistent in their LOCATION
• Therefore people need consistency in NAVIGATIONAL
  MECHANISMS
• Maybe its only LOCATION thats important

16
Strong theories

• However, Im making a stronger theory here,
  that is easier to falsify
• First lets look at the next step of the example
• People will have more difficulty and find a
  website less acceptable if the navigation is
  inconsistent
• Deduction general to specific
• Nitty-gritty of the SM, making a good,
  falsifiable test of this
• Turning a theoretical hypothesis into a testable
  hypothesis

17
Three types of empirical research

• Descriptive studies
• carefully mapping out the situation (in effect,
  describing the facts)
• Observing behaviour, ethnographic research
• Generally not enough of this in the social
  sciences (because they are so busy testing their
  theories), so we lack information about how
  people behave (Carrolls psychology of tasks)
• Are we developing software that people dont
  need?

18
Three types of studies

• Relational/correlational studies
• Looking for relationships between things, even if
  we dont have a theory to explain them
• fishing expedition research - looking for what
  affects what, trying to find the components for a
  theory

19
Three types of research

• What Rosnow and Rosenthal (gurus of research
  methods in psychology) call experimental research
• but Id rather call causal research - as its not
  always really experimental
• where you try and pin down the nature of the
  relationships, the theory behind the
  observations/facts
• Test a hypothesis
• Usually requires a series of studies, not just
  one experiment

20
Creating a testable hypothesis

• We tend to start from a general, vague question
• Need to turn this into something specific and
  appropriate
• Often have two things we need to specify
• Independent variable (the aspect of the
  environment that we are interested in)
• Dependent variable (the behaviour that we are
  interested in)
• (variable something that changes, takes
  different values, that we can alter or measure)

21
Operational definitions

• Called operationalizing the hypothesis - turning
  the vague/theoretical concepts into operational
  definitions
• Independent variable - the nature of the
  navigation on the website
• Dependent variable - the difficulty that people
  have

22
Operational definitions II

• Not necessarily one particular operationalization
  of a variable that is the best
• May well need multi-operationalism (i.e.
  different operational definitions for variables)
  in the same, or different studies, before one is
  confident that one has understood a particular
  phenomenon
• In HCI and related areas I think we do not do
  this enough - one study, one measure and we move
  on in psychology one finds many studies on the
  same phenomenon with slight variations published
• Non-psychologists see this as obsession, but its
  good science

23
Operational definitions

• Navigational consistency
• changes in navigational bars and elements of
  those bars location, font colour, background
  colour, font type, exact wording, background
  decoration, grouping
• changes in in-text navigation initial colour,
  underlining, visited colour
• So in this one very small aspect of web design,
  there are many variables
• One of the problems we have is isolating exactly
  what is causing the problem (true experimental
  design helps here - tomorrow)

24
Operational definitions

• Difficulty that people have
• Objective measures - time taken to complete
  tasks, errors made
• But need to consider two aspects
• will the size of the difference be noticeable?
  equipment power calculations
• Might I get ceiling or floor effects (i.e.
  everyone can do the task error free/everyone
  finds something incredibly difficult)
• Subjective measures - ask people to rate how
  acceptable a website is (and what exactly are
  you going to ask people to rate?)

25
Operational hypothesis

• People will take longer to complete tasks, make
  more errors, and give lower ratings of
  acceptability on a website with a navigation bar
  that varies in its location from screen to screen
  in comparison to one in which the navigation bar
  appears in a consistent position on all screens
• I have multi-operationalized the dependent
  variable, but have a narrow, single operational
  definition for the independent variable -
  tomorrow you will see why

26
H0 vs H1

• I have stated the alternate hypothesis - that
  there will be a difference (known as H1)
• I have stated it as a directional alternate
  hypothesis - that Im predicting that one
  condition (level of the independent variable,
  arrangement of the world) will produced higher
  task times and errors, lower acceptability
  ratings
• Sometimes one is predicting a difference, but
  cannot predict which direction it will take (a
  non-directional alternate hypothesis) - this
  makes a lot of difference in the statistical
  tests one conducts, and Im sure Paul will take
  that up in his part of the course

27
H0 vs H1

• The null hypothesis is the prediction that there
  will not be any difference - that navigational
  consistency will not have any effect on
  times/errors/acceptability ratings
• In doing your statistical tests, you are actually
  trying to reject the null hypothesis

28
Fallacy of rejecting H0

• If you do reject H0, you still might not have
  identified exactly what in the situation that is
  causing the difference
• A problem much discussed in research methods, the
  fallacy of rejecting H0
• Paul Meehl, one of my heros, argued that unless
  you do a very tight experiment, your chances of
  falling into this fallacy is about 50 - so you
  might as well toss a coin
• Penguin research video

29
Do you need an operational definitions/hypotheses?

• A question that Im often asked by students - do
  I need hypotheses for my research?
• Depends a lot on whether you are doing
  descriptive/relational/causal research
• If causal - absolutely
• If the others, it certainly helps to set out what
  are your variables (theoretical/operational
  definitions), the phenomenon/question/hypothesis
  you are investigating
• Might not be able to formalize it to a precise H1

30
Allows you to make a simulation of what you will
find

• Really useful to mock up the data you will
  produce in a study, the levels of the independent
  and dependent variables, the numbers etc
• Will it be statistically analysable (may need to
  consult a statistician, but much easier for them
  to advise you)
• Will it really answer your hypothesis/question?

31
Examples

• Rats just whether they turned left or right did
  not produce the right kind of data that
  discriminated enough between the conditions to
  answer the question posed (this was obvious to a
  person with some statistical training)
• Navigation is the question really about the best
  location for the bar or whether the bar is
  consistently in the same position (perhaps you
  need to answer the first before the second - a
  very common outcome of planning variables and
  hypotheses

32
Reading for this session

• Chalmers, A.F. (1999). What is this thing called
  science? 3rd Edition. Open University Press.
• Rosnow, R.L. and Rosenthal, R. (2005). Beginning
  behavioral research a conceptual primer. 5th
  Edition. Pearson Prentice Hall.
• Rosenthal, R. and Rosnow, R.L. (1991).
  Essentials of behavioral research. 2nd Edition.
  McGraw Hill.