Part 2: Quantitative Methods

1
Part 2 Quantitative Methods

• October 2, 2006

2
(No Transcript)
3
Sampling
4
Target vs. Accessible Population

• High School Sports Officials
• Students enrolled in 5th grade in NM schools
• Albuquerque Residents
• The U.S. Electorate

• National Association of Sports Officials
  Membership
• New Mexico 5th graders
• Albuquerque Phone Book
• Registered Voters

5
size of
sample
Leap
Simple
Random
Sampling
volunteers
Population
Validity
Systematic
Sampling
Random
Sampling
Stratified
Sampling
Processes
Procedures
Sampling
Error
Cluster
Sampling
Pros
Selection
Convenience
Biased
and
and
Sampling
Sampling
Cons
Assignment
6
The Inferential Leap
7
Simple Random Sampling

• 1 092725 012157 827052 297980 625608 964134
• 2 104460 007903 484595 868313 274221 367181
• 3 676071 388003 266711 323324 044463 762803
• 4 881878 862385 203886 261061 096674 811548
• 5 534500 336348 086585 241740 581286 008435
• 6 094276 615776 242112 985859 075388 082003

1. Andrea 2. Tina 3. Paul 4. Wilbur 5. Sandra
6. Kathy 7. Jim 8. George 9. Emir 10. Becky
11. Sharon 12. Gladys 13. Jose 14. Bill 15. Sue
16. Erica 17. Aaron 18. Fred 19. Pam 20. Roger
8
Stratified Random Sampling

• 1 092725 012157 827052 297980 625608 964134
• 2 104460 007903 484595 868313 274221 367181
• 3 676071 388003 266711 323324 044405 762803
• 4 881878 862308 203886 261061 096674 811548
• 5 534500 336348 086585 241740 581286 008435
• 6 094276 615776 242112 985859 075388 082003

6. Paul 7. Wilbur 8. Jim 9. George 10. Emir
1. Andrea 2. Tina 3. Kathy 4. Sandra 5. Becky
6. Sharon 7. Gladys 8. Sue 9. Erica 10. Pam
1. Jose 2. Bill 3. Aaron 4. Fred 5. Roger
9
Systematic Sampling

• Say you have a target population that has a
  100,000 members.
• And
• A list is available.
• You need 1,000 cases for your sample.
• 100,000/1000 100.
• Select a random number from table.
• Then select every 100th case.

10
Cluster Sampling

• Naturally occurring groups.
• State, district, school, classroom, student.
• Randomly sample from one level then survey,
  interview, etc.
• Multistage sampling.
• Randomly select from one level.
• Then randomly select within that level.

11
Convenience Sampling

• Why convenient?
• Sample that is located near the researcher
• Connections with administrator or staff
• Researcher is familiar with the setting
• Data is already available
• Shortcomings of convenience samples?

12
Volunteers in Sampling

• How might volunteers differ?
• Children having parental permission
• More academically competent
• More popular with peers
• More physically attractive
• Less likely to smoke or use drugs
• More likely to be white
• More likely to come from two-parent household
• More likely to be involved in extracurricular
  activities
• Less likely to be socially withdrawn
• Less likely to be aggressive

13
Size of the Sample

• Bigger is (usually) better.
• Unless?
• How big is big? Power analysis.
• Practical issues.
• Attrition.
• Reliability.
• Cost/ benefit.

14
Correlation Instrumentation

• Reliability and Validity

15
Correlation Coefficients

• Pearson product-moment correlation
• The relationship between two variables of degree.
• Positive As one variable increases (or
  decreases) so does the other.
• Negative As one variable increases the other
  decreases.
• Magnitude or strength of relationship
• -1.00 to 1.00
• Correlation does not equate to causation

16
Positive Correlation
17
Negative Correlation
18
No Correlation
19
Correlations

• Thickness of scatter plot determines strength of
  correlation, not slope of line.
• For example see
• http//noppa5.pc.helsinki.fi/koe/corr/cor7.html
• Remember correlation does not equate to causation.

20
Negative Correlation
21
Operationism vs. Essentialism

• According to Stanovich
• What are they?
• How do they differ?

22

• Essentialist
• Like to argue about the meaning of our terms
• What does the theoretical concept really mean?
• Must have a complete and unambiguous
  understanding of the language involved.
• Operationism
• Link concepts to observable events that can be
  measured.
• Concepts in science related to a set of
  operations.
• Several slightly different tasks and behavioral
  events are used to converge on a concept.

23
Validity and Reliability

• Validity is an important consideration in the
  choice of an instrument to be used in a research
  investigation
• It should measure what it is supposed to measure
• Researchers want instruments that will allow them
  to make warranted conclusions about the
  characteristics of the subjects they study
• Reliability is another important consideration,
  since researchers want consistent results from
  instrumentation
• Consistency gives researchers confidence that the
  results actually represent the achievement of the
  individuals involved

24
Reliability

• Test-retest reliability
• Inter-rater reliability
• Parallel forms reliability
• Internal consistency (a.k.a. Cronbachs alpha)

25
Validity

• Face
• Does it appear to measure what it purports to
  measure?
• Content
• Do the items cover the domain?
• Construct
• Does it measure the unobservable attribute that
  it purports to measure?

26
Validity

• Criterion
• Predictive
• Concurrent
• Consequential

27
Types of validity (cont.)
Here the instrument samples some and only of the
construct
28
Types of validity
Here the instrument samples all and more of the
construct
29
The construct
Here the instrument fails to sample ANY of the
construct
The instrument
30
The construct
Here the instrument samples some but not all of
the construct
The instrument
31
Perfection!
32
Reliability and Validity
33
In groups of 3 to 4

• Sampling
• What is the target population?
• What sampling procedure was used?
• Do you think the sample is representative?
• Why or why not?
• Measurement
• What types of reliability and validity evidence
  are provided?
• What else would you like to know?

34
Ways to Classify Instruments

• Who Provides the Information?
• Themselves Self-report data
• Directly or indirectly from the subjects of the
  study
• From informants (people who are knowledgeable
  about the subjects and provide this information)

35
Types of Researcher-completed Instruments

• Rating scales
• Interview schedules
• Tally sheets
• Flowcharts

• Performance checklists
• Observation forms

36
Excerpt from a Behavior Rating Scale for Teachers
Instructions For each of the behaviors
listed below, circle the appropriate number,
using the following key 5 Excellent, 4
Above Average, 3 Average, 2 Below Average, 1
Poor. A. Explains course material
clearly. 1 2 3 4 5 B. Establishes rapport with
students. 1 2 3 4 5 C. Asks high-level
questions. 1 2 3 4 5 D. Varies class
activities. 1 2 3 4 5
37
Excerpt from a Graphic Rating Scale
Instructions Indicate the quality of the
students participation in the following class
activities by placing an X anywhere along each
line. Always Frequently Occasionally Seldom Neve
r 1. Listens to teachers instructions.
Always Frequently
Occasionally Seldom
Never 2. Listens to the opinions of other
students. Always
Frequently Occasionally
Seldom Never 3. Offers own opinions
in class discussions.
38
Sample Observation Form
39
Discussion Analysis Tally Sheet
40
Performance Checklist Noting Student Actions
41
Types of Subject-completed Instruments

• Questionnaires
• Self-checklists
• Attitude scales
• Personality inventories

• Achievement/aptitude tests
• Performance tests
• Projective devices

42
Example of a Self-Checklist
43
Example of Items from a Likert Scale
44
Example of the Semantic Differential
45
Pictorial Attitude Scale for Use with Young
Children
46
Sample Items from a Personality Inventory
47
Sample Items from an Achievement Test
48
Sample Item from an Aptitude Test
49
Sample Items from an Intelligence Test
50
Item Formats

• Questions used in a subject-completed instrument
  can take many forms but are classified as either
  selection or supply items.
• Examples of selection items are
• True-false items
• Matching items
• Multiple choice items
• Interpretive exercises
• Examples of supply items are
• Short answer items
• Essay questions

51
Unobtrusive Measures

• Many instruments require the cooperation of the
  respondent in one way or another.
• An intrusion into an ongoing activity could be
  involved which causes a form of negativity within
  the respondent.
• To eliminate this, researchers use unobtrusive
  measures, data collection procedure that involve
  no intrusion into the naturally occurring course
  of events.
• In most cases, no instrument is used, however,
  good record keeping is necessary.
• They are valuable as supplements to the use of
  interviews and questionnaires, often providing a
  useful way to corroborate what more traditional
  data sources reveal.

52
Norm-Referenced vs. Criterion-Referenced
Instruments

• All derived scores give meaning to individual
  scores by comparing them to the scores of a
  group.
• The group used to determine derived scores is
  called the norm group and the instruments that
  provide such scores are referred to as
  norm-referenced instruments.
• An alternative to the use of achievement or
  performance instruments is to use a
  criterion-referenced test.
• This is based on a specific goal or target
  (criterion) for each learner to achieve.
• The difference between the two tests is that the
  criterion referenced tests focus more directly on
  instruction.