Sampling, Probability

1
Sampling, Probability Experimental Design Dr
Heather WharradWebsite www.nottingham.ac.uknt
zhjw
2
Learning Outcomes

• Review the main features of experimental design
• Briefly consider sampling and probability in
  relation to experiments
• Identify different experimental designs
• Explore the issues raised by the experimental
  approach in research

3
Experimental Research

• 'I am just experimenting with this new recipe
• In experimental evaluation
• - carrying out an experiment is not the ad hoc
  activity suggested above, the term 'experiment'
  has a very precise meaning.

4
Nature of Problem

• When asking many different questions about
  practice, researchers need a growing range of
  methodological approaches. Such methods should
  accord with the nature of the question or problem
  to be addressed
• Treece Treece (1977)

5
PROBLEM

• always multi-factorial
• A
• B C
• D E F
• G H
• I
• The more complex or varied the pattern of
  interaction the less likely it is that an
  experiment can be designed (Wilson-Barnett,
  1991)

6
Define the Question

• Be specific
• How do A, B C relate to D, E F X
• How does A relate to C ?

7
Variables

• Factors that are being investigated are called
  variables
• The independent variable is manipulated by the
  researcher (intervention)
• The dependent variable is measured by the
  researcher (outcomes)
• In experimental evaluation the effect of the
  independent variable on the dependent variable is
  being investigated

8
3 principles of experimental design

• 1.
• In experimental evaluation the researcher
• -systematically varies the independent variable
  and measures the response in the dependent
  variable
• Essential components of the experimental design
  are..controlled comparisons and evaluations of
  manipulated change(Wilson-Barnett, 1991)

9
Choosing the DV and IV - are the measures valid?
Are they reliable?

• Valid - do they measure what they are supposed
  to?
• Reliable -do they give the same result no matter
  who uses them?

10

• Internal Validity
• are the changes in the DV only due to the
  intervention (IV) and not due to other factors?
• External validity
• can the results be generalised to the wider
  population?

11
3 principles of experimental design

• 2.
• In experimental evaluation the researcher
• eliminates the influence of variables other than
  the independent and dependent variables

12
Confounding Variables

• Factors which might influence the dependent
  variable but are not the main focus of the study
• Extraneous variables

13
3 principles of experimental design

• 3.
• In experimental evaluation the researcher
• randomly selects and allocates the subjects to a
  control group (no treatment or standard
  treatment) and experimental group (treatment)

14
Randomisation

• Each study unit has an equal chance of being or
  not being in the experimental group
• eradicates potential for researcher bias
• evenly distribute known and unknown confounding
  variables between the groups

15
Scientific rigour versus human nature

• Silverman 1980 (in Oakley pg 140) Effect of
  artificial light on occurrence of retrolental
  fibroplasia in babies
• Assignment to light or no-light was made on
  the basis of blue and white marbles in a box. One
  day, I noted that our head nurse reached into the
  box for a marble and then replaced it because it
  wasnt the colour that corresponded to her belief
  about the best treatment for her babies. I became
  convinced that we had to shift to sealed
  envelopes, as used in the British streptomycin
  trial. When the first sealed envelope was drawn,
  the resident physician held it up to the light to
  see the assignment inside! I took the envelopes
  home and my wife and I wrapped each
  assignment-sticker in black paper and resealed
  the envelopes.

16
Pre-treatment effects of randomisationWardle et
al (1996) Randomised placebo controlled trial of
effect on mood of lowering cholesterol
concentration. BMJ, 313, 75-78
17
Exercise 1

• What are the independent and dependent variables
  in the studies outlined in the 3 extracts?
• What confounding variables might have influenced
  the results?
• What have the authors done to address this
  problem?

18
Experimental hierarchy
19
Does drug X have any effect on diastolic blood
pressure compared to drug Y (standard treatment) ?

• X (n10) Y(n10)
• 1 80 110
• 2 65 65
• 3 75 90
• 4 70 85
• 5 60 90
• 6 85 70
• 7 90 90
• 8 100 80
• 9 70 60
• 10 65 55
• MEAN 76.0 (SD 12.6) 79.5 (SD 16.9)
• Null hypothesis?

20
Hypothesis

• Supposition about the data
• Is there a difference between data sets A B?
• Null hypothesis - there is no difference between
  data sets A B
• ACCEPT ? REJECT
• Significance Testing

21
Are the means significantly different?????

• 1. Null hypothesis there is no difference
  between drug X and drug Y on BP
• 2. Carry out a paired t-test
• t value is a measure of the relationship
  between
• difference between the means
• variation in the samples
• 3. Use degrees of freedom (measure of sample
  size) and t-value to determine probability (p)
  that the difference between the means is due to
  chance
• 4. For drugs X Y - t1.24 pgt0.05
• Accept or reject the Null hypothesis??

22
Statistical test of the assumption of no
difference

• What is the probability of observing a difference
  as large, or larger as that observed if there
  really is no underlying difference between
  treatments?

23
p-values can only be between 0 and 1
1
0

• p0.999 certain
• p0.75 very likely 3 in 4
• p0.5 fairly likely 1 in 2
• p0.05 fairly unlikely 1 in 20
• p0.001 very unlikely 1 in 1000

24
Probability

• By convention, a significance level of 5
  (p0.05) is considered to be acceptable
• 5 risk that the null hypothesis is true
• put the other way round, there is a 95
  probability that any observed difference is not
  the result of chance.

25
Significance level
26
Problems

• Variation can obscure the real facts or
  relationships
• biological variation
• experimental imprecision
• other variables unrelated to the question

27
Sampling error

• There is no difference between a treatment X and
  control drug Y
• In reality mean Xmean Y
• By chance one can show a difference

28
The 2 possible errors of hypothesis testing
Decision made to reject Null hypothesis
29
Designs

• Pre-test - Post-test
• Subjects Data Treatment Data
• Control Group Pre standard Post
• Exptal Group Pre new Post
• Post-test only
• Subjects Treatment Data
• Control Group standard Post
• Exptal Group new Post

30
Designs

• Within subject
• same subjects are used for the experimental group
  and the control group
• Between subject
• different subjects are used for the control group
  and the experimental group
• subjects are randomly allocated to the groups

31
Exercise 2

• What is the control group in the 3 extracts
• What design is used in each extract ie pre-post
  test, within or between subjects?
• Is the choice of control group and design
  appropriate?
• What are the advantages and disadvantages of a
  within subject and a between subject design?

32
Within-subject - strengths limitations

• Reduce influence of different subject
  characteristics - no need to randomise
• Greater risk of maturation, fatigue and
  mortality (drop-out) effects

33
Between-subject - strengths limitations

• Baseline characteristics may differ between
  groups - randomisation and large sample sizes
  reduces this effect
• ethical issues - withholding a potentially
  beneficial treatment from a control group

34
Experimental research has high internal validity
if..

• Control of confounding variables
• Reliable instruments
• Appropriate choice of independent and dependent
  variables
• Random selection to groups
• Standardisation of protocol
• Minimising factors such as maturation, history
  and mortality.

35

• Experiments tend to have low external
  validity...
• controlled conditions - alters physiological
  and responses and behaviour in humans
• reductionist approach ie selecting a small number
  of variables oversimplifies the true picture
• Hawthorne effect
• external validity improved by replicating
  experiments and carrying out experiments
  double-blind

36
Woolf et al (1990) Hierarchy of Evidence

• I well designed randomised controlled trials
• II-1 well designed controlled trial without
  randomisation quasi experiments
• II-2 well designed cohort (prospective) study,
  preferably from more than one centre
• II-3 well designed case control (retrospective)
  study, preferably from several centres
• III large differences from comparisons between
  time and/or places with or without intervention
• IV opinions of respected authorities based on
  clinical experience descriptive studies and
  reports of expert committees

37
Exercise 3

• What are the threats to the external validity?
• can the results be generalised to the wider
  population?
• What are the threats to the internal validity?
• are the changes in the DV only due to the
  intervention (IV) and not due to other factors?

38
Conclusions

• When considering experimental research
• How complex is the pattern of interaction of the
  variables relating to the problem to be
  investigated
• What are the independent and dependent variables
  going to be? How are these measured?
• What other factors might influence the
  experiment?
• How are the subjects selected?
• What is the power of the study
• What design is used?
• How are the data analysed?

39
ConclusionsCompared to other research methods

• Strengths
• relationships between variables investigated very
  precisely
• eg cause and effect
• effectiveness of an intervention
• high internal validity

• Weaknesses
• less generalisability
• difficult measuring complex emotions eg stress,
  pain
• rigours of exptal design may cause ethical issues
  and conflict with professional codes