INF 397C Introduction to Research in Library and Information Science Fall, 2003 Day 5

1
INF 397CIntroduction to Research in Library and
Information ScienceFall, 2003Day 5
2
The Scientific Method
3

• More than anything else, scientists are
  skeptical.
• P. 28 Scientific skepticism is a gullible
  publics defense against charlatans and others
  who would sell them ineffective medicines and
  cures, impossible schemes to get rich, and
  supernatural explanations for natural phenomena.

4
Research Methods

• S, Z, Z, Chapters 1, 2, 3, 7, 8
• Researchers are . . .
• like detectives gather evidence, develop a
  theory.
• Like judges decide if evidence meets scientific
  standards.
• Like juries decide if evidence is beyond a
  reasonable doubt.

5
Science . . .

• . . . Is a cumulative affair. Current research
  builds on previous research.
• Scientific Method
• Empirical (acquires new knowledge via direct
  observation and experimentation)
• Systematic, controlled observations.
• Unbiased, objective.
• Operational definitions.
• Valid, reliable, testable, critical, skeptical.

6
CONTROL

• . . . Is the essential ingredient of science,
  distinguishing it from nonscientific procedures.
• The scientist, the experimenter, manipulates the
  Independent Variable (IV treatment at least
  two levels experimental and control
  conditions) and controls other variables.

7
More control

• After manipulating the IV (because the
  experimenter is independent he/she decides what
  to do) . . .
• He/she measures the effect on the Dependent
  Variable (what is measured it depends on the
  IV).

8
Key Distinction

• IV vs. Individual Differences variable
• The scientist MANIPULATES an IV, but SELECTS an
  Individual Differences variable (or subject
  variable).
• Cant manipulate a subject variable.
• Select a sample. Have half of em get a
  divorce.

9
Operational Definitions

• Explains a concept solely in terms of the
  operations used to produce and measure it.
• Bad Smart people.
• Good People with an IQ over 120.
• Bad People with long index fingers.
• Good People with index fingers at least 7.2
  cm.
• Bad Ugly guys.
• Good Guys rated as ugly by at least 50 of
  the respondents.

10
Validity and Reliability

• Validity the truthfulness of a measure. Are
  you really measuring what you claim to measure?
  The validity of a measure is supported to the
  extent that people do as well on it as they do on
  independent measures that are presumed to measure
  the same concept.
• Reliability a measures consistency.
• A measure can be reliable without being valid,
  but not vice versa.

11
Theory and Hypothesis

• Theory a logically organized set of
  propositions (claims, statements, assertions)
  that serves to define events (concepts), describe
  relationships among these events, and explain
  their occurrence.
• Theories organize our knowledge and guide our
  research
• Hypothesis A tentative explanation.
• A scientific hypothesis is TESTABLE.

12
Goals of Scientific Method

• Description
• Nomothetic approach establish broad
  generalizations and general laws that apply to a
  diverse population
• Versus idiographic approach interested in the
  individual, their uniqueness (e.g., case studies)
• Prediction
• Correlational study when scores on one variable
  can be used to predict scores on a second
  variable. (Doesnt necessarily tell you why.)
• Understanding cont. on next page
• Creating change
• Applied research

13
Understanding

• Three important conditions for making a causal
  inference
• Covariation of events. (IV changes, and the DV
  changes.)
• A time-order relationship. (First the scientist
  changes the IV then theres a change in the
  DV.)
• The elimination of plausible alternative causes.

14
Confounding

• When two potentially effective IVs are allowed to
  covary simultaneously.
• Poor control!

15
Intervening Variables

• Link the IV and the DV, and are used to explain
  why they are connected.
• Heres an interesting question WHY did the
  authors put this HERE in the chapter?
• Because intervening variables are important in
  theories.

16
A bit more about theories

• Good theories provide precision of prediction
• The rule of parsimony is followed
• The simplest alternative explanations are
  accepted
• A good scientific theory passes the most rigorous
  tests
• Testing will be more informative when you try to
  DISPROVE (falsify) a theory

17
Populations and Samples

• Population the set of all cases of interest
• Sample Subset of all the population that we
  choose to study.

Population Parameters
Sample Statistics
18
Ch. 3 -- Ethics

• Read the chapter.
• Understand informed consent, p. 57 a persons
  expressed willingness to participate in a
  research project, based on a clear understanding
  of the nature of the research, the consequences
  of declining, and other factors that might
  influence the decision.
• Odd quote, p. 69 Debriefing should be informal
  and indirect.
• Know that UT has an IRB http//www.utexas.edu/re
  search/rsc/humanresearch/

19
Ch. 7 Independent Groups Design

• Description and Prediction are crucial to the
  scientific study of behavior, but theyre not
  sufficient for understanding the causes. We need
  to know WHY.
• Best way to answer this question is with the
  experimental method.
• The special strength of the experimental method
  is that it is especially effective for
  establishing cause-and-effect relationships.

20
Good Paragraph

• P. 196, para. 2 Discusses how experimental
  methods and descriptive methods arent all THAT
  different well, theyre different, but related.
  And often used together.

21
Good page P. 197

• Why we conduct experiments
• If results of an experiment (a well-run
  experiment!) are consistent with theory, we say
  weve supported the theory. (NOT that it is
  right.)
• Otherwise, we modify the theory.
• Testing hypotheses and revising theories based on
  the outcomes of experiments the long process of
  science.

22
Logic of Experimental Research

• Researchers manipulate an independent variable in
  an experiment to observe the effect on behavior,
  as assessed by the dependent variable.

23
Independent Groups Design

• Each group represents a different condition as
  defined by the independent variable.

24
Random . . .

• Random Selection vs. Random Assignment
• Random Selection every member of the population
  has an equal chance of being selected for the
  sample.
• Random Assignment every member of the sample
  has an equal chance of being placed in the
  experimental group or the control group.
• Random assignment allows for individual
  differences among test participants to be
  averaged out.

25
Lets step back a minute

• An experiment is personkinds way of asking
  nature a question.
• I want to know if one variable (factor, event,
  thing) has an effect on another variable does
  the IV influence the DV?
• I manipulate some variables (IVs), control other
  variables, and count on random selection to wash
  out the effects of all the rest of the variables.

26
Block Randomization

• Another way to wash-out error variance.
• Assign subjects to blocks of subjects, and have
  whole blocks see certain conditions.
• (Very squirrelly description in the book.)

27
Challenges to Internal Validity

• Testing intact groups. (Why is the group a
  group? Might be some systematic differences.)
• Extraneous variables. (Balance em.) (E.g.,
  experimenter).
• Subject loss
• Mechanical loss, OK.
• Select loss, not OK.
• Demand characteristics (cues and other info
  participants pick up on) use a placebo, and
  double-blind procedure
• Experimenter effects use double-blind procedure

28
Role of Data Analysis in Exps.

• Primary goal of data analysis is to determine if
  our observations support a claim about behavior.
  Is that difference really different?
• We want to draw conclusions about populations,
  not just the sample.
• Two ways stat and replication.

29
Two methods of making inferences

• Null hypothesis testing
• Assume IV has no effect on DV differences we
  obtain are just by chance (error variance)
• If the difference is unlikely enough to happen by
  chance (and enough tends to be p lt .05), then
  we say theres a true difference.
• Confidence intervals
• We compute a confidence interval for the true
  population mean, from sample data. (95 level,
  usually.)
• If two groups confidence intervals dont
  overlap, we say (we INFER) theres a true
  difference.

30
What data cant tell us

• Proper use of inferential statistics is NOT the
  whole answer.
• Scientist could have done a trivial experiment.
• Also, study could have been confounded.
• Also, could by chance find this difference.
  (Type I and Type II errors hit this for real in
  week 5.)

31
This is HUGE.

• When we get a NONsignificant difference, or when
  the confidence intervals DO overlap, we do NOT
  say that we ACCEPT the null hypothesis.
• Hinton, p. 37 On this evidence I accept the
  null hypothesis and say that we have not found
  evidence to support Peters view of hothousing.
• We just cannot reject it at this time.
• We have insufficient evidence to infer an effect
  of the IV on the DV.

32
Notice

• Many things influence how easy or hard it is to
  discover a difference.
• How big the real difference is.
• How much variability there is in the population
  distribution(s).
• How much error variance there is.
• Lets talk about variance.

33
Sources of variance

• Systematic vs. Error
• Real differences
• Error variance
• What would happen to the standard deviation if
  our measurement apparatus was a little
  inconsistent?
• There are OTHER sources of error variance, and
  the whole point of experimental design is to try
  to minimize em.
• Get this The more error variance, the harder
  for real differences to shine through.

34
One way to reduce the error variance

• Matched groups design
• If theres some variable that you think MIGHT
  cause some variance,
• Pre-test subjects on some matching test that
  equates the groups on a dimension that is
  relevant to the outcome of the experiment. (Must
  have a good matching test.)
• Then assign matched groups. This way the groups
  will be similar on this one important variable.
• STILL use random assignment WITHIN the groups.
• Good when there are a small number of possible
  test subjects.

35
Another design

• Natural Groups design
• Based on subject (or individual differences)
  variables.
• Selected, not manipulated.
• Remember This will give us description, and
  prediction, but not understanding (cause and
  effect).

36
Weve been talking about . . .

• Making two groups comparable, so that the ONLY
  systematic difference is the IV.
• CONTROL some variables.
• Match on some.
• Use random selection to wash out the effects of
  the others.
• What would be the best possible match for one
  subject, or one group of subjects?

37
Themselves!

• When each test subject is his/her own control,
  then thats called a
• Repeated measures design, or a
• Within-subjects design.
• (And the random groups design is called a
  between subjects design.)

38
Repeated Measures

• If each subject serves as his/her own control,
  then we dont have to worry about individual
  differences, across experimental and control
  conditions.
• EXCEPT for newly introduced sources of variance
  order effects
• Practice effects
• Fatigue effects

39
Counterbalancing

• ABBA
• Used to overcome order effects.
• Assumes practice/fatigue effects are linear.
• Some incomplete counterbalancing ideas are
  offered in the text.

40
Which method when?

• Some questions DO lend themselves to repeated
  measures (within-subjects) design
• Can people read faster in condition A or
  condition B?
• Is memorability improved if words are grouped in
  this way or that?
• Some questions do NOT lend themselves to repeated
  measures design
• Do these instructions help people solve a
  particular puzzle?
• Does this drug reduce cholesterol?

41
Hinton typo

• P. 62, para. 1 . . . population standard
  deviation, µ, divided by . . . .

42
Some questions wed like to ask Nature
43
Midterm

• Emphasize
• How to lie with statistics concepts
• To know a fly concepts
• SZZ Ch. 1, 2, 7, 8
• Hinton Ch. 1, 2, 3, 4, 5
• De-emphasize
• SZZ Ch. 3
• Other readings
• Totally ignore for now
• SZZ Ch. 14
• Hinton Ch. 6, 7, 8