Experimental Measures and NonExperimental Quantitative Research Design

1
Experimental Measures and Non-Experimental
Quantitative Research Design

• Katie Rommel-Esham
• Education 504

2
Paper-and Pencil Tests

• A situation in which a standard set of questions
  is presented to each subject.
• The focus is on some cognitive task what the
  person knows (achievement), is able to learn
  (ability or aptitude), chooses or selects
  (interests, attitudes, or values), or is able to
  do (skills).
• All tests measure current performance.

3
Standardized Tests

• Provide uniform procedures for administration and
  scoring, which may include timing, a script,
  specified testing conditions, materials allowed,
  whether or not students questions may be
  answered
• Scoring is usually objective
• Most have been given to some norming group
• Most are prepared commercially which helps to
  provide some degree of insurance regarding norms,
  reliability, and validity
• May not be specific enough to provide as
  sensitive a measure as is needed.

4
Criterion-Referenced Interpretation

• Individuals scores are compared to a
  predetermined standard of performance (the
  criterion), not with the scores of others
• Score is usually expressed as a percentage or
  pass/fail
• Focuses on what individuals are able to do
• Most result in highly skewed distribution
• Good for diagnosis

5
Achievement Tests

• Measure present knowledge and skills of a sample
  of relevant content
• Have restricted coverage, and are generally tied
  closely to school subjects
• Emphasis is on recent learning
• Types include diagnostic, survey battery, single
  subject achievement
• May be either norm- or criterion-referenced,
  though most are norm-referenced

6
Standards-Based Tests

• Standardized achievement tests with
  criterion-referenced interpretations
• Based on established standards
• Influenced by what students learn in school as
  well as by what they learn at home and in the
  community
• Students are generally judged as proficient or
  non-proficient
• Generally high stakes tests (for promotion,
  graduation, accreditation, for example)

7
Norm-Referenced Interpretation

• Scores indicate how an individual compares with
  the norming population (i.e. individuals scores
  are compared with the scores of others)
• Interpretation relies on relative standing with
  less emphasis on absolute amount of knowledge or
  skill
• Instrument needs to be able to make distinctions
  among individuals

8
Norm-Referenced Interpretation

• Extremely easy and extremely difficult items are
  not included, though items tend toward the
  difficult end of the spectrum
• Score is generally expressed as a standard score
  or grade equivalent
• Need to look very carefully at the norming sample
  when interpreting results
• If you are studying gifted students and comparing
  them to a national norming sample, a ceiling
  effect may result

9
Standard Scores

• Most common is the z-score
• where xi is an individual score, is the mean,
  and s is the standard deviation

10
z- Score

• The z-score provides an indication of how far a
  given score is from the mean in standard
  deviation units
• It allows comparisons across non-equivalent
  testing situations
• Scores below the mean will have a negative z
  -score, while those above the mean will have a
  positive z -score
• A z-score is not a score that would be recorded
  in a grade book, however it provides useful
  information

11
Aptitude Tests

• Used to predict future performance on a criterion
  prior to instruction, placement, or training
• The test itself, not individual items, is
  considered to be predictive
• The terms intelligence test and ability test
  are often used interchangeably with aptitude

12
Measures of Noncognitive Traits

• Personality, Attitude, Value, and Interest
  Inventories

13
Questionnaires

• Most widely used technique to gather information
  from subjects
• They are economical, anonymous, and consistent
• Subjects respond to something written

14
Guidelines for Questionnaires

• Items should be clear
• Questions should be limited to a single idea or
  concept (not double barrelled)
• Respondents should be competent to answer (refers
  to timing, area of expertise, etc)
• Questions should be relevant
• Items should be short and simple
• There should be no negative items
• There should be no biased items

15
Types of Items

• Open Form
• Subjects write in whatever response they choose
• Exert little control over the responses
• Closed Form
• Subjects choose from among pre-determined
  responses
• May limit accuracy and variability of responses

16
Scaled Items - Likert Scale

• Question or statement followed by a scale of
  potential responses
• May be tailored to fit the nature of the
  question, but may also be misleading
• Science is very important
• _____ _____ _____ _____
  _____
• strongly agree agree neutral
  disagree strongly disagree

17
Scaled Items - Semantic Differential

• Variation of the Likert Scale
• Uses adjective pairs, each of which is used as an
  anchor for a continuum
• Math
• Like ___ ___ ___ ___ ___ ___ Dislike
• Easy ___ ___ ___ ___ ___ ___ Tough

18
Ranked Items

• Allows for respondents to prioritize
• For example, a respondent might indicate that
  five items are very important, which is of
  limited usefulness without a rank ordering of the
  items

19
Checklist

• May provide respondents with a number of items
  from which to choose
• May also be used to ask students to reply yes/no,
  or choose the category to which they belong
• For categorical responses, respondents can be
  placed in exactly one category

20
Problems

• Response set includes selecting all positive
  responses or all negative responses (regardless
  of content), guessing, and sacrificing speed for
  accuracy
• Faking includes responding in socially desirable
  ways
• Reliability is generally lower than in cognitive
  tests
• Construct validity is difficult to establish
• Because there are generally not right answers,
  the nature of the comparison group is
  particularly important

21
Observations

• Rely on a researcher seeing and hearing things
  and recording these observations rather than
  relying on subjects self-report responses to
  questions or statements
• In quantitative research, the observer acts as a
  complete observer
• Observations may be high inference or low
  inference

22
Observer Effects

• Observer Bias due to preconceived notions of the
  observer
• Contamination result of observers knowledge of
  the purpose of the study
• Halo Effect based on initial impressions

23
Interviews

• Interviews are essentially vocal questionnaires

24
Advantages of Interviews

• They are flexible and adaptable, and may be used
  with those who are not capable of completing a
  questionnaire (those who are illiterate or too
  young to read and write, for example)
• Responses may be followed up, clarified
• Result in a higher response rate than
  questionnaires

25
Disadvantages of Interviews

• Potential for subjectivity and bias
• Costly
• Time consuming
• Lack of anonymity
• Respondent may be uncomfortable and not relay
  accurate information
• Interviewer may ask leading questions
• Sample may be smaller (resources)

26
Interview Questions

• Structured Would you say the program has been
  highly effective, somewhat effective or not
  effective at all?
• Semistructured What has been the most effective
  aspect of your teacher development program?
• Unstructured Tell me about your mentoring
  program.
• Leading Given the expense of the new reading
  program, should we make the adoption this year?

27
Interviewer Effects

• Bias
• Contamination
• Halo Effect

28
Unobtrusive Measures

• A type of measure in which participants are asked
  or required to do nothing out of the ordinary
• Provide data that are uninfluenced by an
  awareness of the subjects that they are
  participants
• Includes things like physical traces, including
  worn floors, books, or computers also documents,
  letters, and reports