Evaluating Research Claims

1
Evaluating Research Claims
2
Evaluating TV Ad Claims

• An ad on TV says,
• USE Cocoa Creme. Youll have youthful skin in
  10 minutes. All natural ingredients.
• You say, Yeah, sure! Youthful skin.
• I bet your face falls off in 10 minutes.

3
Evaluating TV Ad Claims

• Another ad tells you to,
• Use Motor Magician. Your engine will purr like
  a kitten. And youll get 80 miles per gallon!
• You say, I dont want a motor that sounds like
  a cat. 80 miles per gallon? HA! Baloney.

4
Evaluating TV Ad Claims

• In other words.
• You arent fooled by ads promising what is very
  unlikely, and that give no EVIDENCE to support
  the inflated promises.

5
Evaluating Medical Claims

• The same goes with medicine.
• Lets say your child is ill with an infection.
• The physician says, Heres a prescription for
  Megabiotic.
• You ask, Will it work, Doctor?
• Physician. Oh, yes. It will fix her right
  up.
• You. Does it have bad side effects?
• Physician. Oh, no. No side effects.
• You. How do you know, Doctor?

6
Evaluating Medical Claims

• How do you know, Doctor?
• That is THE question.
• How do you know?
• What makes you so sure?
• Do you have trustworthy evidence to back up your
  claim?

7
Evaluating Medical Claims

• Will you let your child take the prescribed
  medication if these (below) are the physicians
  answers?....
• Well, I and my doctor friends think it works
  very well.
• In a study of five persons, everyone got well.
• In a study of 1000 persons, many said they felt
  much better.
• You wouldnt be satisfied by these answers!
• In fact, youd tell the physician, See you
  later!

8
Evaluating Medical Claims

• What DO you want the physician to tell you?
• How about these?...
• Megabiotic was tested in 10 experimental studies.
• The research was not conducted by the drug
  company, but by independent researchers who had
  nothing to gain whether Megabiotic was effective
  or not effective.
• Each study lasted from one to five years.
• Each study had at least 1000 persons in it.
• 
  More

9
Evaluating Medical Claims

• Each study sample had a mix of different kinds of
  persons in it younger/older, more ill/less ill,
  male/female.
• Each study compared the effects of Megabiotic
  with other medications (alternative treatments)
  and with no treatments (control group).
• Each study measured the exact amount of infection
  before treatment, each week during treatment, and
  at the end of the course of treatment. The
  measurement was objective---the amount of
  infection in the blood.
• Side effects were also measured, such as blood
  pressure, skin rash, and kidney function.
• More

10
Evaluating Medical Claims

• All of the studies found that persons who
  received Megabiotic had NO serious side effects.
  
• Persons who received Megabiotic showed directly
  observable (in the blood) reductions in infection
  each week until, by the end of one year, 95 of
  the persons were cured.
• Now would you let your child take the medication?

11
Evaluating Medical Claims

• Probably.
• Why?
• Why is the second set of answers from the doctor
  convincing?
• Think about it.

12
Evaluating Medical Claims

• The second answers are more convincing because
• The second answers are more believable.
• The second answers give you solid REASONS to
  believe that the medication is likely to cure
  your childs infection and not cause harm.
• The physician gave you evidence that is hard to
  dismiss as just baloney.
• 
  More

13
Evaluating Medical Claims

• You cant say, It wont work with my child,
  because the research on Megabiotic was tested
  with thousands of persons---many of whom were
  just like your child.
• You cant say, Maybe persons got well by
  themselves. It wasnt Megabiotic, because
  persons in the control group (who got no
  Megabiotic) did NOT get well.
• You cant say, They didnt measure improvement,
  because in fact they measured what can be SEEN
  (the amount of infection in the blood)---an
  objective measure of improvement, not an opinion.
• 
  More

14
Evaluating Medical Claims

• You cant say, The effects are probably
  temporary persons will get sick again, because
  some of the studies lasted five years, and no one
  got sick again with the infection.
• You cant say, The research was rigged to get
  good results. For example, researchers dropped
  from the study all persons who were NOT getting
  well. You cant say this because the drug
  company (that wanted the drug to work) did not DO
  the research.

15
Summary So Far

• Lets summarize.
• What makes research believable?
• What makes the conclusions of research
• The (drug, treatment, or method) is
  effective. Therefore, its a good idea to use
  it.
• so trustworthy that you can be confident in
  your decision to use it?

16
Summary So Far

• Here is what makes research believable and the
  conclusions trustworthy.
• The effect of the thing being tested (program,
  materials, treatment, or method) is defined and
  measured objectively.
• The thing being tested is tested against
  alternatives (to see which works better) and
  against no treatment (to see if the thing works
  at all---that is, improvement is much greater
  than when there is NO treatment).
• The research is done many times (replicated), to
  ensure that results are not chance, not a fluke.
• The research is done for a long period of time
  (longitudinal), to see if the effects are
  long-lasting.
• More

17
Summary So Far

• The research is done with samples of persons who
  have the same features as persons in the larger
  population who might later receive the program,
  treatment, or method. representative sample
• The researchers do not have a stake in the
  outcomes. independent research

18
Lets Apply This to Education

• Just as you feel responsible for the care of
  persons you love, you also feel responsible for
  the children you teach.
• You dont want students to be taught with
  curriculum materials or with teaching methods
  that dont work and that cause harm---for
  example, teaching students the wrong way to read
  or to solve math problems.
• Therefore, you want to ask tough questions about
  research done on curriculum materials and
  teaching methods that are advertised, or that a
  school district wants you to use, or that your
  colleagues say are terrific.
• What tough questions should you ask about
  research?

19
Questions About Purpose. Selling?

• What tough questions should you ask about
  research?
• Lets say you are reading a research report, or
  a description of a new program on a website, or
  an advertisement for curriculum materials.
• Does the ad, report, or program description sound
  as if the authors are selling an idea or a
  productto persuade you that they are right that
  they have the answer that they can fix schools
  and raise achievement?
• Just (believe us, join us, buy our
  materials).
• If selling is their purpose, this is NOT good.
  Why?
• You can be almost certain that the only data
  (evidence) the writers will show are data that
  support their claims. They will not show data
  that do NOT support their claims. This is
  called dishonest.

20
Purpose Selling

• In other words, you should be highly suspicious
  of their claims about effectiveness.
• Our method works.
• Our ideas are sound.
• Use our ideas and program.
• These claims do not come from objective
  scientific research. They come from persons and
  groups who are making up a story (a sales pitch)
  that will get you to buy.
• If they had solid research evidence that their
  ideas, materials, or procedures were effective,
  their writing would not sound like a sales pitch.
  

21
Purpose Reporting

• However, the purpose of a journal article or a
  program description may simply be to REPORT the
  findings of research. For example, the authors
  are telling you what they learned---not what they
  want you to believe or to do.
• For instance, authors may report which one of
  three methods of peer tutoring produces the
  highest achievement.
• Or what percentage of fourth graders are
  proficient in reading on grade level after a new
  reading program was used.
• Simple reporting is good. If authors are not
  trying to sell you an idea or materials or
  programs, then they are probably going to report
  honestly what they found they have nothing to
  gain simply by passing on information.

22
Questions About the Research Strategy

• There are different KINDS of research questions
  for instance, there are questions.
• About persons opinions. How do teachers feel
  about.?
• About what persons do. How do students respond
  when teachers.?
• About factors that cause or predict changes.
  Does timely correction of errors reduce those
  same errors happening in the future?
• About what is effective. Which program (A, B,
  or C) is more effective in raising the reading
  achievement of diverse learners?

23
Questions About the Research Strategy

• There are different kinds of research questions,
  and there are different kinds of research
  strategies---or WAYS of trying to answer the
  questions. For instance, there are
• Surveys.
• Observation.
• Experimentation.

24
Questions About the Research Strategy

• Heres a big idea
• Certain kinds of research questions require
  certain research strategies,
• or else the research will NOT give evidence
  (data, findings) that you can trust.
• Do you have high blood pressure?
• Which is the better research strategy to get an
  answer you will trust?
• Do a survey of the opinions of 20 of your
  friends.
• Use a blood pressure cuff (every day for a week,
  at different times of day) to get exact
  measurement.
• Blood pressure is NOT an opinion-kind-of-thing!
  
• It is an objective thing. PRESSURE. Therefore,
  it requires an OBJECTIVE measurement---with the
  cuff.

25
Questions About the Research Strategy

• Here are different kinds of research questions
  and the proper research strategies.
• What are teachers opinions about a new math
  program that they are using?
• How well do students think they read three months
  after theyve been taught with a new reading
  program?
• How satisfied are teachers with the supervision
  and professional development they receive?
• What are the evaluations of parents on the
  helpfulness of their childrens teachers?
• These are opinion questions.

26
Is the Research Strategy Appropriate for the
Research Questions?

• These are opinion questions.
• The researcher wants information on how
  teachers, students, and parents SEE things.
• Therefore, a useful strategy would be SURVEY
  research.
• Example of survey research ?

27
Is the Research Strategy Appropriate for the
Research Questions? Survey Research

• Consider the question on parents satisfaction.
• The researcher selects a sample of families.
• This sample is representative of the whole
  population of families of children at schools
  for example, small and large families two-parent
  and single-parent families families with one
  child and families with more children poor
  families and affluent families.
• Then the researcher either uses a questionnaire
  (that parents fill out) or uses interviews
  (face-to-face) in which the researcher asks the
  same kinds of questions as on the questionnaire
  or gives questionnaires to all families and also
  interviews a smaller sample to obtain more
  detailed information.

28
Is the Research Strategy Appropriate for the
Research Questions? Survey Research

• The researcher can examine (analyze) the survey
• information (data) from the questionnaires or
• interviews, and can then determine
• What kinds of help parents feel is adequate.
• What kinds of help parents feel is inadequate.
• What percentage of parents are satisfied vs. not
  satisfied with teachers helpfulness.
• These findings could assist school administrators
  to
• improve family-school relations.

29
Is the Research Strategy Appropriate for the
Research Questions?

• Here is a different kind of research question.
• What are the different ways that teachers respond
  or try to handle students disruptive behavior?
  What are the effects of each way?
• When students are doing peer tutoring, do
  students learn to play the role of tutor and the
  role of tutee? If so, what are the behaviors in
  these roles?
• When a class is organized into learning centers
  (e.g., reading, math, science), and students are
  allowed to come and go as they wish, what do
  students actually DO in the centers?

30
Is the Research Strategy Appropriate for the
Research Questions?

• These are questions about what persons
  DO---about observable activity---NOT about how
  persons see things---their opinions.
• Therefore, a useful research strategy is
  OBSERVATIONAL research.
• Example of observation research?

31
Is the Research Strategy Appropriate for the
Research Questions? Observational Research

• Consider the question of how teachers respond to
  disruptive
• behavior.
• The researcher selects a sample of classrooms.
• Standing out of the way, the researcher takes
  running notes (like a sportscaster) on
  teacher-student interaction.
• After several weeks, the researcher examines the
  notes and
• a. Identifies different kinds of disruptive
  behavior e.g., making noises, talking out of
  turn, noncompliance with teacher requests.
• b. Identifies how the teacher typically
  responds to each kind of disruptive behavior
  e.g., telling a student to stop, threatening to
  send the student to the office, ignoring the
  disruptive behavior but praising students when
  they are engaging in desirable behavior.
• c. Determining what generally happens when
  teachers respond in different ways e.g.,
  students do the disruptive behavior more often
  vs. less often.

32
Is the Research Strategy Appropriate for the
Research Questions?

• Here is yet another kind of research question.
• Is peer tutoring more effective (does it yield
  higher achievement) for some subjects than other
  subjects?
• Are teachers opinions about their new math
  programs affected by whether a math program
  provides written procedures for teachers to
  follow vs. does not provide written procedures
  for teachers to follow?
• More

33
Is the Research Strategy Appropriate for the
Research Questions?

• Under what conditions (e.g., size of school,
  recent levels of student achievement, quality of
  teaching staff) is it better (there is faster
  change, less opposition) when school reform is
  rapid and extensive vs. slow and in small steps?
  In other words, what factors predict which kind
  of school reform works better.
• Which math program yields the most math
  proficiency in students overall and students by
  subgroups, such as ethnicity and social class?
• These are questions about causation.

34
Is the Research Strategy Appropriate for the
Research Questions?

• These are questions about causation. The idea
• is that
• Certain factors ? Predict or produce
• or variables changes in other
  factors
  or variables

35
Is the Research Strategy Appropriate for the
Research Questions?

• Here are examples of questions about which
  factors or
• variables predict or produce (cause) effects in
  other variables.
• Which features of a math program (Teaching
  procedures? Practice? How tasks are sequenced?)
  have the largest effects on (cause the largest
  changes in) learning?
• Which features of a school affect (cause or
  inhibit) the success of efforts to reform the
  school curriculum?
• How does subject matter affect (cause or inhibit)
  learning through peer tutoring?
• Do written procedures or the absence of written
  procedures in a math program affect (cause)
  teachers opinions of the program?

36
Is the Research Strategy Appropriate for the
Research Questions?

• This kind of question.
• Which factors cause what changes?
• Which curriculum, program, or procedure is more
  effective?....
• .requires the strategy of experimental
• research.

37
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• The essential feature of the strategy of
  experimental
• research is that you
• Compare two or more situations (e.g., schools)
  that are as similar as you can make them (e.g.,
  of students who are poor, size of school, quality
  of teaching staff), but these schools are
  different in the few factors (e.g., reading
  materials) whose effects you are trying to find
  out, or test.
• If the only main way that the schools differ is
  the
• reading materials used, and if one school has
  consistently
• higher reading achievement than the others, then
  it is
• very likely that the reading materials make the
  difference.
• What ELSE could it be? After
  all. ?

38
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• The factor that makes the difference in reading
  achievement cant be class size---because this is
  the same across all the schools.
• The factor that makes the difference in reading
  achievement cant be teacher quality---because
  this is the same across all the schools.
• The factor that makes the difference in reading
  achievement cant be the percentage of students
  who are poor---because this is the same across
  all the schools.
• Therefore, the factor that makes the difference
  in reading achievement must be the ONE major way
  the schools are different----reading curriculum.

39
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• However, not all experiments give information
  that you can
• trust. It depends on how the experiment is
  carried out. For
• example
• 1. Maybe students in schools with the highest
  reading achievement were already reading at a
  higher level. Therefore, these students dont
  have to learn very much in order to come out
  first. In other words, maybe students
  pre-skills (and not the curriculum materials)
  account for higher achievement.
• To rule out this possibility, an experiment
  should have pre-
• test measures of the variables whose change we
  are interested
• in. For instance, we should measure reading
  skills in all of the
• schools BEFORE new reading materials are
  introduced, to
• see if some classes are ahead of other classes.

40
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• So, if an article says that a program or
• method produced more change or is more
• effective than another program or method, ask
• if the researchers did pre-tests to show that
• students all started at the same level. If they
• did not do pre-tests, then you cannot trust the
• claim about effectiveness. The group with the
• highest achievement may have started at a
• higher level.

41
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• 2. How do you know how much students learned?
  The only way to find out is to do post-tests
  (e.g., at the end of the semester and year) to
  see where students reading achievement ended up.
  And you have to see how much students changed
  (learned, achieved) between the pre-test (reading
  proficiency at the start) and the post-test
  (reading achievement at the end). And you need
  to report how much difference there is between
  the pre-test and post-test scores.
• If researchers do NOT do post-tests, or do not
  REPORT the
• size of differences between the pre-test and
  post-test, then you
• dont know HOW MUCH students learned as a result
  of the
• curriculum that was used between the pre-test and
  the post-
• test. Therefore, you cant trust claims that one
  curriculum
• produced more achievement than the others.

42
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• 3. Were measures objective?
• You dont rely on a physicians subjective
  opinion about your
• health.
• You look fine to me.
• Maybe the physician is right.
• But what if the physicians opinion is wrong?
• Just so, you want objective evidence of (in this
  experiment)
• reading achievement. It is nice to know how much
  teachers
• think their students learned, but you cant judge
  
• effectiveness based on opinion. Reading is a
  real thing.
• It can be measured objectively---for example, how
  many
• words students read correctly per minute.

43
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• So, if researchers do NOT use objective measures
  of (for example) reading and math achievement
  (e.g., counting correct words students read or
  problems students solve per minute), but instead
  use the subjective opinions of teachers
  (Students improved a great deal.) then you
  cannot trust claims about effectiveness----because
  the researchers did not give evidence of
  effectiveness. They only gave evidence of how
  teachers see things. But that was not the
  question.

44
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• 4. Were instruments and measurement methods
  tested for validity (they measure what they are
  supposed to measure) and reliability (different
  persons measuring the same thing would come up
  with about the same scores)?
• Validity. Do you think a good measure of
  reading proficiency is whether students can name
  the parts of a book and can turn the pages?
• If your school principal is selecting a new
  reading program, is that the information you want
  the principal to use?
• Students taught with Weird Phonics learned to
  turn the pages. So, Ill use that program with
  200 students in my school.
• I dont think so.

45
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• Turning pages is not what READING means.
• You want to know if a program teaches students
  to (1) say the sounds that go with the letters
  (2) accurately and quickly read single words (3)
  accurately and quickly read passages (4) learn
  many new vocabulary words each week (5) answer
  both simple and hard comprehension questions
  about what they read.

46
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• So, if researchers use instruments and measures
  that do not measure what they OUGHT to measure,
  then the findings have nothing to do with the
  research question.
• The question is whether a curriculum teaches
  students to read. But the findings have to do
  with turning pages. Invalid measure.
• The question is whether a certain teaching
  procedure increases math achievement. But the
  findings have to do with how much students
  enjoyed being taught with the method. Invalid
  measure.
• The question is whether peer tutoring is
  effective at helping students to spell better.
  But the findings have to do with changes in
  self-esteem. Invalid measure.

47
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• 5. Is the sample size, composition (kinds of
  participants), and selection/allocation of
  participants to groups appropriate for the type
  of study and for how the findings might be used?
  
• Poor.
• The researcher claims that a teaching method is
  effective. However, the method was used in only
  one class, or in one school. There is no way to
  tell if it would be effective anywhere else.
• Poor.
• The researcher claims that a method is not
  effective. However, it was used in only one
  class, or in one school. There is no way to tell
  if it would have been effective elsewhere.
• More

48
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• Poor.
• The researcher claims that an instructional
  procedure or curriculum is more effective than
  another. However, the comparison groups (one
  class used one curriculum the other class used a
  different curriculum) were not created by random
  assignment of children to the groups, or by
  matching the groups to make sure they had, for
  example, the same percentage of boys and girls.
• Therefore, the groups may have been different in
  other ways besides the curriculum e.g., more
  children in one group got reading instruction at
  home, and this (not the curriculum) caused
  differences in achievement.
  More

49
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• So, if research involves comparing (for example)
• achievement from one group/sample (class,
• school) to another, but the researchers did NOT
• make sure (through random assignment of
• students to the samples, or through matching)
  that
• the groups have the same characteristics, then
  you
• cant trust that differences in achievement are
• caused by the curriculum or teaching method.

50
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• Here is one kind of experimental research that
  you
• DO want to see. Level-1 research.
• The researcher conducts a pilot study to see if a
  method
• seems to work well enough that it ought to be
  given a more
• valid test. The researcher uses a small sample
  (one class or
• school). It is not a representative sample of
  the school
• population, but that is alright. Do you want to
  use a method
• that may not work on a lot of kids? The
  researcher finds that
• the new method is followed by significant student
  
• achievement. But since the sample is small and
  not
• representative of the larger school population,
  and since
• the research was done only once, the researcher
  cautions that
• the findings are very tentative and cannot be
  generalized
• anywhere else. Further research is needed.
• This is called honesty.

51
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• Here is another kind of experimental research
  that you DO
• want to see. Level-2 research.
• The researcher repeats research on the same
  method in
• different schools. Each new study is called a
  replication. The
• sample of schools is representative of the
  population of
• schools---small/large, wealthy/poor, rural/urban.
  
• The researcher claims that data from most of the
  studies
• show that the method is effective in most of the
  schools, but
• the researcher cannot figure out which factors
  hinder its
• effectiveness or what additional factors may be
  needed to
• make it work better. The researcher advises
  caution in using
• the method. This is honest and morally
  responsible.

52
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• Here is yet another kind of experimental research
  
• that you DO want to see. Level-3 research.
• The researcher conducts a study in several school
  
• districts at once. The sample of districts is
• representative of the larger population of
  schools and
• school districts. Three new reading curricula are
  
• tested against each other. The three curricula
  are
• randomly assigned to the districts. Objective and
  
• validated measures of essential reading skills
  are
• made before the new curricula are introduced
  (pre-
• test), every two weeks during the use of the
  curricula
• (progress assessment). and at the end of the year
  
• (post test, or outcome assessment.
  More

53
Is the Research Strategy Appropriate for the
Research Questions? Experiment

• At the end of the year, researchers examine the
  amount of change in students skills from
  pre-test to post-test. They find that curriculum
  A produced significantly more achievement (and
  objectively high achievement) in all of the
  reading skills (compared to achievement produced
  by curricula B and C) in 8 out of the 10 schools
  in which it was used. The researchers therefore
  claim that curriculum A is more likely than the
  other curricula to produce high reading
  achievement, and that it can be used with at
  least modest confidence. However, they say that
  more research is needed to determine the school
  conditions in which curriculum A works best.
• Would you use curriculum A?