M.Sc. Computing Research Methods 06-Experimental Research

1
M.Sc. ComputingResearch Methods06-Experimental
Research

• Adrian J Pullin
• FML208
• pullina_at_hope.ac.uk
• hopelive.hope.ac.uk/imc/staff/pullina

2
Plagiarism Warning

• The majority of the material in this presentation
  would constitute plagiarism if it were submitted
  for assessment

3
What is Science?

• Science is best defined as a careful,
  disciplined, logical search for knowledge about
  any and all aspects of the universe, obtained by
  examination of the best available evidence and
  always subject to correction and improvement upon
  discovery of better evidence. What's left is
  magic. And it doesn't work. Randi, James,
  n.d.
• James Randi has an international reputation as a
  magician and escape artist, but today he is best
  known as the world's most tireless investigator
  and de-mystifier of paranormal and
  pseudoscientific claims.

4
Originality (1)

• Setting down a major piece of new information in
  writing for the first time
• Continuing previous original work
• Carrying out original work designed by supervisor
• Providing single original technique, observation
  or result in otherwise unoriginal but competent
  research
• Having many original ideas, methods or
  interpretations performed by others under your
  direction
• Originality in testing someone elses idea
• Empirical work that has not been done before
• Making a synthesis that has not been made before
• Using already known material but with new
  interpretation

5
Originality (2)

• Doing something in your country that has
  previously been done in other countries
• Corollary comparing something in your country
  with the same thing in another country AP
• Taking a particular technique and applying it in
  a new area
• Bringing new evidence to bear on an old issue
• Being cross-disciplinary and using different
  methodologies
• Looking at areas that people in that discipline
  have not looked at before
• Adding to knowledge in a way that has not been
  done before
• Phillips and Pugh, 2000

6
What is Scientific Method?

• Observe some aspect of the universe
• Invent a tentative description, called a
  hypothesis, that is consistent with what you have
  observed
• Use the hypothesis to make predictions
• Test those predictions by experiments or further
  observations and modify the hypothesis in the
  light of your results
• Repeat steps 3 and 4 until there are no
  discrepancies between theory and experiment
  and/or observation
• Wudka, Jose 1998

7
What is a Theory?

• When consistency is obtained the hypothesis
  becomes a theory and provides a coherent set of
  propositions which explain a class of phenomena.
• A theory is then a framework within which
  observations are explained and predictions are
  made
• Wudka, Jose 1998

8
What Is Scientific Method? (Diag)
Wudka, Jose 1998
9
Steps in the Scientific Method
Dolphin L, 1992
10
Scientific Method Text Version

• Working from previous knowledge
• Formulate research question
• State hypothesis
• Test hypothesis
• Design experiment
• Conduct experiment
• Analyse data
• (Iterate)
• Publish results
• Peer review
• After Kenyon, P, n.d.

11
Triangulation

• Note triangulation on previous slides
• Concept is credible because a range of sources
  state the same thing
• In this case, a massive range, of which this is
  only a very small sample

12
Why Scientific Method?

• Unprejudiced
• Dont have to believe researcher
• Redo experiment and determine if results are true
  or false
• Conclusions will hold irrespective of state of
  mind, religious persuasion, state of
  consciousness of investigator and/or subject
• Faith (belief that does not rest on logical proof
  or material evidence) does not determine whether
  theory is adopted
• Theory is accepted not on prestige or convincing
  powers of proponent, but on results obtained
  through observations and/or experiments which
  anyone can reproduce
• (? Do you really believe this? - AP)
• Results obtained using the scientific method are
  repeatable
• Most experiments and observations are repeated
  many times
• If original claims not verified, origin of
  discrepancies is studied
• Wudka, Jose 1998

13
Caveat

• Getting the same results is not the same as
  drawing the same conclusions

14
Experimental ResearchHolah, M 2005 next 10
slides

• Most scientific of all methods
• Method of choice
• At least in some areas
• Provides controlled situations
• An experiment is a study of cause and effect
• Involves deliberate manipulation of one variable,
  while trying to keep all other variables constant

15
Laboratory Experiments (1)

• Try to keep all aspects of the situation constant
  except one
• The one we are looking at
• Aspect we vary is called the independent variable
  (IV)
• Change this in a very precise way
• The aspect we measure, the dependent variable
  (DV), changes as a result
• If our thesis is correct
• DV depends on IV

16
Laboratory Experiments (2)

• All other variables which might affect the
  results and therefore give us a false set of
  results are called confounding variables
• Or random variables
• E.g. 
• Differences in instructions given by an
  experimenter
• Could be overcome by standardising instructions
• Differences between participants, e.g. in their
  age
• Eliminated as a variable by using a single age
  group

17
Laboratory Experiments (3)

• Change one variable (the IV)
• Measuring another (the  DV)
• Control all others, as far as possible
• Experimental method allows us to draw conclusions
  with far more certainty than any non-experimental
  method
• If IV is only thing that is changed then it must
  be responsible for any change in DV

18
Example (Say HCI)

• Divide participants into 2 groups
• Experimental group and control group
• Introduce a change for experimental group and not
  control group
• A control group is a group for whom the
  experimenter does not change IV
• Experimental and control groups must be matched
  on all important characteristics
• Age, sex, experience etc.

19
Advantages of Laboratory Experiments

• Only means by which cause and effect can be
  established
• IV is thought of as cause and (DV) as effect
• Allows for precise control of variables
• Control group enables experimenter to isolate one
  key variable (IV) and observe its effect on some
  other variable (DV)  
• Experiments can be replicated
• Cannot generalise from one set of results
• Repeating experiment, with same results obtained,
  gives more confidence that theory being tested is
  valid
• Experimental method consists of standardised
  procedures and measures which allow it to be
  easily repeated
• Experiments yield quantitative data
• Can be analysed using inferential statistical
  tests
• Permit statements about how likely the results
  are to have occurred through chance

20
Limitations of Laboratory Experiments (1)

• Artificiality experiment is not typical of real
  world situations
• Most experiments done in laboratories
• Strange, contrived environments
• People asked to perform unusual or bizarre tasks
• Unusual equipment set-up
• Bias George Miller (1962) estimated 90 of U.S.
  experiments used college students
• Accessible and cheap
• Results still tend to be generalised to
  population as a whole

21
Limitations of Laboratory Experiments (2)

• Not possible to completely control all variables
• Variables at work of which the experimenter is
  unaware
• Ethics
• Experiments with people nearly always involve
  deceiving participants to some extent
• Educational experiments are still experimenting
  on children
• (Aside need approval from head teacher/parents
  and possibly a CRB check)
• Normative data
• Grouping people together and trying to cancel out
  individual differences to look at a group norm is
  limited in how much it can tell us because it
  ignores what is special about people

22
Field Experiment

• Experiments in a more natural setting
• The field
• IV still deliberately manipulated by researcher
• Not possible to have such tight control over
  variables in the field
• Advantage of being far less artificial than the
  laboratory

23
Natural Experiment

• Take advantage of natural situation to carry out
  investigation
• Circumstances cannot be manipulate
• E.g. school may decide to switch to a new CBL
  package
• Effects compared with similar school using
  different package 
• Not true experiment
• Unable to manipulate or control variables
• Quasi-experiment
• Is possible to compare two groups
• Equivalent of experimental and control group
• Participants are unaware that they are taking
  part in an investigation
• Not as artificial as a laboratory setting

24
Scientific Method Infallible?

• Error? - Electrochemically Induced Nuclear Fusion
  of Deuterium
• Martin Fleischmann, Department of Chemistry, The
  Universityof Southampton, Hants. S09
  5NH,ENGLANDStanley Pons, Department of
  Chemistry, University of Utah, Salt Lake city, UT
  84112 USA
• Submitted to Journal of Electroanalytical
  Chemistry March 11,1989 in final form March 20,
  1989
• http//www.totse.com/en/fringe/fringe_science/fusi
  on.html
• http//encyclopedia.thefreedictionary.com/coldfus
  ion
• Hoax? - Piltdown Man
• http//www.clarku.edu/piltdown/Prefacegratitude.
  html
• http//www.talkorigins.org/faqs/piltdown.html
• Deliberate fraud? - molecular-scale transistors
• http//www.socialismtoday.org/70/science.html
• http//seattlepi.nwsource.com/business/88624_bell2
  61.shtml

25
Scientific Method Protection

• http//www.sfn.org/NL/1997/March-April/ethics_poli
  cy.html
• http//www.bps.org.uk/publications/journals/princi
  ples-of-publishing.cfm
• http//www.liv.ac.uk/research/work/conduct.htm

26
And Finally

• "There is not the slightest indication that
  nuclear energy will ever be obtainable. It would
  mean that the atom would have to be shattered at
  will." Albert Einstein, 1932

27
ReferencesHarvard format

• Wudka, Jose 1998 The Scientific Method
  http//phyun5.ucr.edu/wudka/Physics7/Notes_www/n
  ode5.htmlSECTION02120000000000000000 accessed
  24/02/05
• Holah, M 2005, The Experimental Methods,
  http//www.holah.karoo.net/experimental_method.ht
  m Accessed 22/02/05
• This is a site aimed at Psychology AS students,
  but the material on experimental method is
  excellent
• Dopohin L 1991 Steps in the scientific method,
  http//www.ldolphin.org/SciMeth2.html accessed
  24/02/05
• Phillips, E and Pugh, DS, 2000 How to Get a
  PhD A Handbook for Students and Their
  Supervisors, Open University Press
• Kenyon P n.d. The Scientific Method
  http//salmon.psy.plym.ac.uk/year1/scimeth3.gif
  accessed 24/02/05

28
See Also

• Web search on scientific method
• http//en.wikipedia.org/wiki/Scientific_method
• http//koning.ecsu.ctstateu.edu/Plants_Human/scime
  th.html
• Randi, James n.d. http//www.randi.org/
• But also
• http//www.dharma-haven.org/science/myth-of-scient
  ific-method.htm
• Note these are very sloppy references
• Do them properly in your dissertation