Study Design Basics

1
Study Design Basics - The Devil is in the Details
-
Paul Zorn Regional Conservation Biologist Ontario
Service Centre, Parks Canada
2
Objectives

• Study design
• What is it?
• Why is it important?
• When to use it?
• How to recognize it?

01
3
Topics to Be Covered

• Types of Study Designs
• Classes of Variables
• Target Population vs. Sampled
• Population vs. Sample
• Pseudoreplication
• Randomization, Replication
• and Control
• Sampling Strategies

• Statistics
• 2-tailed versus 1-tailed Tests
• Scale and Autocorrelation
• Power Analysis, Effect Size
• and Sample Size Estimation
• Preliminary Sequential
• Sampling
• Sampling Protocols

02
4
Study Design What is it (generally) and why is it
important?
03
5
What is it?

• Study design is simply the process one
  undertakes
• to develop a scientific study.
• Scientific study objective, controlled,
  precise.

The most critical stage of implementation and
completing a monitoring study is not data
collection, presentation or interpretation, but
rather design. Careful design will
increase effectiveness, reduce costs and lead to
improved interpretation. (Jones, K.B. 1986. The
Inventory and Monitoring Process. pp. 1-10 in
A.Y. Cooperrider, R.J. Boyd and H.R. Stuart
(eds.). Inventory and Monitoring of Wildlife
Habitat.)
04
6
Types of Designs
Quasi-Experiments
Observational Studies
Experiments

• Observing the
• system of interest
• under controlled
• circumstances.
• Randomization,
• replication.

• Studies where
• strict controls
• randomization
• are not possible
• or feasible.
• Inference is
• compromised.

• No attempt is
• made to
• manipulate
• the system of
• interest.
• No a priori
• hypothesis is
• made.

05
7
Why is it Important?

• Without careful attention to design, at best
  youll
• loose your investment in time money through
• inconclusive results. At worst youll act on
  bad
• information and do more harm than good.

• Careful design can

• Control for bias in
• your data.
• Improve precision
• accuracy.
• Address biological
• statistical
• assumptions.

• Reduce the risk
• of error.
• Provide better
• quality information.
• Allow for effective
• adaptive
• management.

06
8
Study Design What is it (specifically) and what
are its common elements?
07
9
Topics to Be Covered

• Types of Study Designs
• Classes of Variables
• Target Population vs. Sampled
• Population vs. Sample
• Pseudoreplication
• Randomization, Replication
• and Control
• Sampling Strategies

• Statistics
• 2-tailed versus 1-tailed Tests
• Scale and Autocorrelation
• Power Analysis, Effect Size
• and Sample Size Estimation
• Preliminary Sequential
• Sampling
• Sampling Protocols

08
10
Classes of Variables
09
11
Target Sampled Population vs. Sample
Statistical vs. Biological Populations
Target Population
Sampled Population
Sample

• The total collection
• of units about which
• you want to make
• an inference.
• Measuring all units
• in your target pop.
• is a census.

• Often there are
• elements of the
• target population
• that cant be
• measured, so a
• subset is measured.
• This is the sampled
• population.

• The collection of
• units (replicates)
• within the sampled
• population that are
• actually measured.

10
12
Which is Which? - An Example
Is salamander abundance different between
deciduous coniferous forests?

• 2 forest types
• 4 forest patches (2 decid., 2 conif.)
• 6 plots (3 per type)
• 100 total salamanders (68 decid., 32 conif.)

What is the target population, sampled
population sample size?
11
13
Pseudo- replication

• A serious and fairly common mistake.
• Occurs when the sample size is incorrectly
  estimated.
• Usually occurs when the number of values that
  make up
• a sample are used as a studys sample size.
• From the previous slide, using 100
  (salamanders)
• instead of 6 (plots) as the sample size is an
• example of pseudoreplication.
• Pseudoreplication can be disastrous for study
  design
• and power analysis.

12
14
Topics to Be Covered

• Types of Study Designs
• Classes of Variables
• Target Population vs. Sampled
• Population vs. Sample
• Pseudoreplication
• Randomization, Replication
• and Control
• Sampling Strategies

• Statistics
• 2-tailed versus 1-tailed Tests
• Scale and Autocorrelation
• Power Analysis, Effect Size
• and Sample Size Estimation
• Preliminary Sequential
• Sampling
• Sampling Protocols

13
15
Randomization

• Most people remember that random samples are
  important
• but theyre not sure why.
• Random with respect to what?
• For many monitoring studies random sampling is
  NOT
• the optimal study design.

14
16
With Respect to What?

• Often we want to specifically sample across a
  full range
• of variation or gradient (non random).
• We still want to sample randomly, however, with
  respect
• to the detection probability of the items of
  interest. Failure
• to do so introduces bias error.

15
17
Replication

• Once we have identified the target and sample
  populations
• and the sets of conditions across which we
  want to sample
• we then want to replicate the samples in each
  set.
• Replication minimizes the probability that your
  result is based
• on chance events. How many replicates you need
  can be
• addressed through Power Analysis.
• Replication can occur across space and time
  (e.g., monitoring).

16
18
Experimental Control

• The purpose of most sampling strategies is to
  control or
• standardize variables that are related to the
  items of interest.
• Experimental controls often take the form of
  blocking or
• stratification (e.g., stratified random
  sampling).
• The application of experimental controls will
  reduce error
• and increase precision.
• If auxiliary (supplementary) data on controlled
  and disturbing
• variables is collected then their effects can
  often be determined
• through analysis (e.g., partial correlation,
  analysis of residuals).

17
19
Sampling Strategies
There are several other strategies. The above
tend to be the most common and will be discussed.
18
20
Random Sampling

• The simplest sampling
• strategy.
• Samples are selected
• without respect for the
• underlying structure.
• Used when the study area
• is homogeneous and
• blocking does not reduce
• variance.
• Also used when sample
• sizes are large so that
• the range of conditions
• will likely have enough
• replicates.

19
21
Systematic Sampling

• The underlying structure
• of the target population is
• unknown.
• There is assumed to be some
• disturbing variable(s) but
• not enough is known to
• create blocks (strata).
• Samples are systematically
• and evenly spread across
• the study area (usually with
• a random start) in an effort
• to replicate the unknown
• structure.

20
22
Stratified Sampling

• The study area is blocked
• according to some controlled
• variable(s).
• Samples are taken randomly
• or systematically from each
• block.
• The number of samples per
• block (strata) should be
• equal or very close (balanced
• design).
• Strata can be defined by one
• variable or a combination of
• several variables.
• Reduces the sampling error
• increases precision.

21
23
Two Stage Sampling

• A nested design where you
• take a sample (stage 1) and
• then subsample the original
• sample (stage 2).
• Useful when you are able to
• draw very large and unwieldy
• samples (e.g., benthic
• invertebrates).
• Possible to have multi-stage
• sampling.
• Useful for large scale studies.

22
24
Repeated Measures

• A repeated measures design occurs when samples
  are drawn
• in a consistent order without randomization.
• The most common case is monitoring where samples
  are ordered
• with respect to time (e.g., annual surveys of
  long term monitoring
• plots).
• The original design of repeated measure studies
  can be of any
• type.
• Observations in a repeated measures design may
  not be
• independent (autocorrelated) in which case
  your effective sample
• size may be smaller than you think (a form of
  pseudoreplication).
• Autocorrelation to be discussed later.

23
25
Topics to Be Covered

• Types of Study Designs
• Classes of Variables
• Target Population vs. Sampled
• Population vs. Sample
• Pseudoreplication
• Randomization, Replication
• and Control
• Sampling Strategies

• Statistics
• 2-tailed versus 1-tailed Tests
• Scale and Autocorrelation
• Power Analysis, Effect Size
• and Sample Size Estimation
• Preliminary Sequential
• Sampling
• Sampling Protocols

24
26
Statistics

• Statistics is a vital part of any study design
  as it is
• usually the tool used to answer the question.
• The type of statistical method to be used is
  guided by your
• question and the design of your study.
• Statistics is a separate course in itself.
• It is the data assumptions inherent in
  statistical methods
• that drives the development of so many
  sampling strategies.
• Knowledge of statistics is required to address
  sample size
• estimates and determine the cost of
  science-related programs.

?
25
27
What To Know About Statistics

• Sometimes getting results out of stats can seem
  a bit like
• pulling a rabbit out of a hat. Stats can be a
  bit of an art form.
• Some things to watch out for as a manager

• Effect sizes (magnitude of relationship) is more
  important
• than P values.

• Type II error is often more important than Type
  I error but
• rarely reported.

• Confidence levels (e.g., 95) are arbitrary and
  not one size
• fits all.

• Managers are often specifically interested in
  1-tailed not
• 2-tailed distributions.

26
28
How Many Tails?

• Using the incorrect null hypothesis in a
• study is quite common. 2-tailed distributions
  are often tested
• when it should be a 1-tailed distribution.
• For management purposes (e.g., active
  management, impact
• assessments, ecological restoration) we are
  often interested
• in a 1-tailed distribution.
• An upside with testing 1-tailed distributions is
  that you only
• need around half the sample size to achieve
  the same power
• as an equivalent 2-tailed test. Therefore,
  half the cost.

27
29
Topics to Be Covered

• Types of Study Designs
• Classes of Variables
• Target Population vs. Sampled
• Population vs. Sample
• Pseudoreplication
• Randomization, Replication
• and Control
• Sampling Strategies

• Statistics
• 2-tailed versus 1-tailed Tests
• Scale and Autocorrelation
• Power Analysis, Effect Size
• and Sample Size Estimation
• Preliminary Sequential
• Sampling
• Sampling Protocols

28
30
Scale

• The issue of scale has been referred to as the
  central
• problem in ecology.
• There is no single correct scale at which to
  study something
• and the results you get are dependent on the
  scale you choose.
• A linear change in scale does not necessarily
  mean a linear
• change in the measured value. Scale dependent
  patterns can
• be complex and non-linear.
• How can you select the appropriate scale(s) for
  a study?

29
31
Grain Extent
Spatial Scale
Temporal Scale

• In ecology scale is measured in
• terms of grain and extent.

30
32

• Choose an extent based on the expected range
• of variation (from expert opinion, literature
  review,
• and/or preliminary sampling), project
  logistics
• (e.g., access, cost) and management interest.

Extent of A Study Area
extent too small
31
33
Grain of A Study Area

• Measures taken close together in space or time
• (small grain) tend to be more correlated with
• eachother than measures taken far apart
• (not independent).

AUTOCORRELATION
32
34
Autocorrelation

• Autocorrelation is a
• form of
• pseudoreplication
• and causes you to
• think that you have
• more independent
• samples than you
• really do.
• (degrees of freedom
• are reduced)
• Autocorrelation will
• over estimate the
• statistical
• significance of your
• result (P value) or
• give you a result
• that is simply wrong.

Temporal Autocorrelation Time Series Analysis ACF
Plots
Spatial Autocorrelation Spatial
Statistics Morans I
33
35
Spacing Samples
34
36
Reducing Autocorrelation (Or when to Ignore It)
Separate or aggregate your samples in space
and/or time by the lag distance where
autocorrelation becomes zero.
1month
When to not worry about Autocorrelation If your
analysis shows a VERY strong relationship (e.g.,
Plt0.01) then loosing some degrees of freedom will
not likely reverse your conclusion. In this
case, autocorrelation wont bother you very much.
50m
35
37
Topics to Be Covered

• Types of Study Designs
• Classes of Variables
• Target Population vs. Sampled
• Population vs. Sample
• Pseudoreplication
• Randomization, Replication
• and Control
• Sampling Strategies

• Statistics
• 2-tailed versus 1-tailed Tests
• Scale and Autocorrelation
• Power Analysis, Effect Size
• and Sample Size Estimation
• Preliminary Sequential
• Sampling
• Sampling Protocols

36
38
Power Analysis, Effect Size Sample
Size Estimation
Power analysis quantifies the relationships
between the following

• Effect Size The smallest detectable change.

• Variation The amount of noise in the data.

• Confidence Probability of not making a false
  alarm (Type 1 error).

• Power Probability of not missing a significant
  change when it happens
• (Type 2 error).

• Sample Size The amount of data you have.

• Using Power Analysis, if you have estimates for
  any 4 of these things,
• you can solve for the 5th thing.

37
39
The Power Curve
effective and efficient
effective but not efficient
not effective or efficient
38
40
Preliminary Sequential Sampling

• Where do you get estimates for power analysis
  parameters?

• Preliminary Sampling (i.e., pilot projects)
  occurs before the formal
• start of a study. This stage allows for
  testing of methods, evaluating
• effort, and for collecting power analysis
  parameters to pin down
• the strength of the proposed study.

• Sequential Sampling (i.e., ongoing monitoring)
  occurs when data are
• analyzed during the data collection phase of a
  project. Sequential
• sampling, analysis, and evaluation makes the
  manager more familiar
• with the study and its data and promotes
  Adaptive Management.

Data collected from preliminary sequential
sampling can often be used in the overall study.
People who skip this step often end up wasting
money, not saving it.
39
41
Topics to Be Covered

• Types of Study Designs
• Classes of Variables
• Target Population vs. Sampled
• Population vs. Sample
• Pseudoreplication
• Randomization, Replication
• and Control
• Sampling Strategies

• Statistics
• 2-tailed versus 1-tailed Tests
• Scale and Autocorrelation
• Power Analysis, Effect Size
• and Sample Size Estimation
• Preliminary Sequential
• Sampling
• Sampling Protocols

40
42
Sampling Protocols
Permanent plots, line transects, point counts,
mark-recapture, etc.

• How to choose? Specifics of all the types of
  sampling protocols
• is a course in itself.

• Dont reinvent the wheel. Be familiar with
  previous related
• studies and use tried true protocols.

• How big should my plots be? How long my
  transects?
• How long the radius in my point counts?
• There is usually a trade off between size
  number.
• Generally speaking, choose the smallest size
  needed to
• accommodate the size and pattern of the item
  of interest
• and then replicate as much as possible.

41
43
Topics to Be Covered

• Types of Study Designs
• Classes of Variables
• Target Population vs. Sampled
• Population vs. Sample
• Pseudoreplication
• Randomization, Replication
• and Control
• Sampling Strategies

• Statistics
• 2-tailed versus 1-tailed Tests
• Scale and Autocorrelation
• Power Analysis, Effect Size
• and Sample Size Estimation
• Preliminary Sequential
• Sampling
• Sampling Protocols

42
44
Study Design When to use it?
43
45
Money is Always Tight

• Study design should be carefully considered
  before
• any investment in monitoring is made.
• No amount of statistical hocus pocus will turn
  data collected from a
• bad design into useful information.
• Not all designs are complex and not all topics
  covered today will
• always be relevant, but some always will be
  (e.g., scale, disturbing
• variables - bias).
• Most study design problems groups face are
  common across
• the country. If you have a problem chances are
  someone has already
• thought about it.

44
46
Study Design How to recognize it?
45
47
1. Is there a clear, concise question?
What To Look For
2. Can the proposed study conceptually
answer the question?

• There isnt 1 type of
• study design
• that always works best.
• The focus today was on
• the principles that form
• the basis for good design.
• These principles are
• transferable and something
• to be considered for any
• study.
• When I review a parks
• study design for the first
• time the things I immediately
• look for are

3. Is there a list of potential disturbing
variables that can bias the study and is
there a plan to deal with them?
4. Is the scale of the study appropriate?
5. Are they sampling across a range of
variation (including where the effect is
absent)?
6. Are there replicates sampled for
combinations of space, time and/or control
variable(s)?
7. Is there a stated plan on how the data is
to be analyzed once its collected?
46
48
The Big Leagues Of Study Design
end