Designing Scientific Experiments

1
Designing Scientific Experiments

• Dr. Gail P. Taylor
• MBRS-RISE Coordinator
• UT San Antonio

08/2006
2
References

• CRITICAL THINKING, THE SCIENTIFIC METHOD, AND
  PAGE 25 OF GILBERT Dany S. Adams, Department of
  Biology, Smith College, Northampton, MA 01063
  http//www.sdbonline.org/SDBEduca/dany_adams/criti
  cal_thinking.htmlblurb
• Validity http//carbon.cudenver.edu/lsherry/rem
  /validity.html
• At the Bench, A Laboratory Navigator updated
  edition. Kathy Barker, Cold Spring Harbor Press,
  2005

3
Scientific Method

• Observe phenomenon conceive ideas
• Make predictions/develop a hypothesis
• Devise a test/formulate experiment
• Carry out experiments
• Draw conclusions from results
• Reject or support hypothesis

4
Types of Experiments

• Science does not generally deal with facts, but
  rather with evidence
• Each experiment weakens or strengthens a
  hypothesis
• All evidence is not equal
• Try to discern cause and effect!

5
Planning Experiments I

• What ideas have you come up with?
• Why is your idea important?
• Have other people tested this idea before?
• http//www.pubmed.org
• What type of background information is available?
• Define question/Develop hypothesis (and null
  hypothesis)

6
Determining Causality

• Causality can be difficult to prove
• Different experimental designs impact differently
• Correlative Evidence (weak evidence) Found
  together in time or space
• Loss of Function (stronger evidence)
• Blocked a phenomenon
• Gain of Function (strongest evidence)
• Initiation of event leads to second event
  (additional function)

7
Example Protein X may be involved in Cellular
Aggregation

• Show it
• Correlative evidence (time and space)
• Antibody used to detect
• Found in particular microorganism when
  aggregating (and not when free living)
• Found in area where cells are contacting one
  another during aggregation
• No causality nothing beyond inference about
  function
• Clumping could cause the protein expression
• Clumping and protein expression could be induced
  by same causative agent
• Could be completely coincidental

8
Example Protein X may be Necessary for Cell
Aggregation

• Block it
• Loss of Function - What does its loss do to
  clumping?
• Antibody to protein used to block it from
  functioning.
• Or knock out gene
• Clumping no longer takes place
• Need controls-
• Clumping specifically and only being inhibited
• cells not dying
• May support real clumping agent to function
• Therefore it is necessary for clumping

9
Example Protein X may be Sufficient for
Cellular Aggregation

• Move it
• Gain of Function
• In organism that does not normally clump
• Artificially introduce required protein
• Or artificially turn it on at all times
  (constituitively express)
• Aggregation now takes place
• Therefore is sufficient to induce clumping

10
Progression to Necessary and Sufficient

• Often you will see this progression through
  Biological scientific papers
• What is it?
• Yes, its there
• Yes, its in the right place
• Its loss produces this response
• Its addition produces this response

11
Planning Experiments II

• Consider statistical methodologies during
  planning stages
• Look in prior papers for ideas about statistics.
• Statistical analysis will generally discern that
  likelihood that a result occurred by chance
• Consult mentor or statistician for confirmation
• Compare 1 treatment and control t-test
• Decide on p (Probability value) p lt 0.05 or 0.01
• Compare many treatment groups ANOVA
• Many more

12
Planning Experiments III

• Variables
• Independent (manipulated)
• Dependent (outcome)
• of samples (minimum 2, 3 better)
• repetitions (minimum 2x)

13
Internal Validity

• Cause and Effect- Did the experimental treatment,
  and only the experimental treatment, cause the
  effect!
• Controls (Be Careful!!!)
• Prevent additional variables from sneaking into
  your experiment
• Must control for
• Selection Anything that makes treatment and
  control groups different at beginning (random
  assignment)
• History What different things may happen
  between expt. And control groups between initial
  treatment and measurement
• Maturation Natural changes in subjects (aging)
• Instrumentation All tests/equipment/reagents
  must stay same throughout experiment
• Testing incoming may teach the subject
• Mortality Subjects may leave or die
  (contamination)
• Regression If initial test scores were high, on
  average, will naturally move towards mean

14
External Validity

• The extent to which the findings of the study can
  be applied, reproduced, or generalized to another
  setting or systems. i.e., techniques to ensure
  that these groups correspond to general
  population
• Unrepresentative Sample Sample members not
  representative of general population.
• Clear Description of the Treatment or Protocol
  (replicability)
• Hawthorne Effect Subjects know that they are
  being studied and it influences behavior
• Novelty Effect Particularly in humansenjoy
  experiment, then possibly dont.
• Pretest Sensitization If the pretest is part of
  the treatment, it will obviously affect the
  results or findings.
• History and Treatment Interaction something else
  happened which influenced results, for all
  participants
• Measurement of the Dependent Variable Treatment
  and data collection must be the same every time!

15
Types of Controls

• Experimental
• Standards/calibration
• Animal/Cell selection/care
• Positive controls
• See what a positive response looks like and that
  it can be obtained. (positively expressing
  cells)
• Negative controls
• Shows what a zero response looks like
• Treatment controls
• All groups treated identically except for indep.
  Variable
• If two treatments combined, show individual
• All time points must be covered
• Multiple samples

16
Keeping it Simple

• Your mentor wants to look at the time course
  effects of a possible cancer suppressor on
  proliferation and mRNA expression in six breast
  cancer cell lines. Wants to look at 0, 12, 24,
  36, 48, 72, 96h

17
The beauty of Small experiments.

• Mega Experiment
• Ex 6 types of cells, 7 time points, treated and
  untreated (2), in triplicate (3).
• 6x7x2x3 252 plates
• Plan Strategically and Break it down
• 1 cell line, treated and untreated, duplicate, 7
  time points 28. Or postpone duplicates.

18
Results from Small Experiments

• Low possibility for confusion
• Reasonable workload
• Reasonable use of resources
• Ability to assess as progress
• Easy to interpret
• Can change directions on fly
• Easy to create discrete graphs

19
Chasing the Big Problem

• For a Publication
• Need a Big Picture of what you are pursuing
  tell a good story
• Start with correlation
• Get additional information
• Knock it out/Add it back/Overexpress
• Slight modifications, depending on field

20
How to do Experiment Obtain Protocol

• Instructions for carrying out a particular
  technique
• If followed, will produce desired results
• Best if its a proven protocol
• Designing your own is time-consuming
• Obtain from another investigator
• In lab, best
• A book of protocols, from web, from kit
• Will need fine-tuning for your local
  circumstances
• Methods section from published papers (least
  reliable)

21
Review Protocol

• Read and do dry run-through
• May find logic gaps
• May find references to common procedures you do
  not know

22
Personalize Protocol

• Rewrite (keeping same steps, etc) to make more
  sense to you.
• Add notes about own equipment required

23
Fully Prepare before Experiment

• Buy all required materials
• Radioisotopes
• Make all solutions and buffers
• Reserve machine time if needed

24
Follow Protocol exactly, first time through

• If it doesnt work, you can assume its you.
• Do again. Not work?
• Can get help from person who provided

25
Modify Protocol

• Once protocol is working, modify.
• Make notations of changes
• Rewrite for next run though
• Good if type into computercan record changes and
  re-print.

26
During Experiment

• Record which media, temperature or date-sensitive
  agents you used
• Record any procedural deviation
• Dropped something
• Delayed
• Calibration questions
• Put in lab notebook in a timely fashion

27
Interpreting Results I

• Did the Experiment work?
• Examine procedural (markers, cells lived)
• Examine positive control (yes, antibody working)
• Examine negative control (No, did not have
  everything come up positive)

28
Interpreting Results II

• What were the results?
• Compared to controls, did you see effect?
• Graph your data
• How big was effect?
• Did effect vary over time?

29
Interpreting Results III

• What does the experiment mean?
• Do the results make sense?
• Was the result what you expected?
• Can you explain spurious results?
• What additional controls may you need?

30
Interpreting Results IV

• What do other investigators think?
• Talk to lab members
• Discuss results with someone versed in technique
• Run through background papers again
• Repeat results

31
Interpreting Results V

• Are the results repeatable?
• Do experiment again
• Add any additional controls

32
Agh! It didnt work!

1. Check notes.
2. Redo the experiment
3. Focus on individual parts of expt.
4. and controls
5. Do partial expt. to insure its fixed
6. When youve done alltry again several times
7. If external protocol, may want to switch

33
Switching Projects

• Never can reproduce data
• Project has little support from PI
• Direction of project has changed
• Not technically possible to do experiments well
• Project too difficult or involved