Conducting Scientifically-Based Research in Teaching with Technology, Part II

1
Conducting Scientifically-Based Research in
Teaching with Technology, Part II

• SITE Annual Meeting Symposium
• Atlanta, Georgia
• Gerald Knezek Rhonda Christensen
• University of North Texas
• Charlotte Owens Dale Magoun
• University of Louisiana at Monroe
• March 2, 2004

2
Time Table for PART II

400 420 Issues in Scientifically-Based Research, Fitting the KIDS Project into the template
420 435 Groups work in roundtables to fit own ideas into template, facilitators float
435 450 Presentation, group discussion of 3-4 plans
450 500 Questions/Comments/Feedback
500 End Session
3
Scientifically-Based Research(Whose Definition?)

• Methodology Issues
• Randomization Issues
• Instrumentation Issues
• Determination of Impact
• Analysis/Interpretation Issues

4
Issues of Methodology

• Quantitative
• Currently in favor, heavy on analysis methodology
• Qualitative
• Rich analysis, takes longer
• Mixed Methods
• Seeing process in operation often necessary to
  find out why in education
• Theory Building vs. Theory Testing
• Exploratory/Data Mining vs. Hypothesis Testing

5
Issues of Randomization

• Random assignment (currently emphasized)
• For internal validity (fidelity of experiment)
• Start with large group
• Randomly assign 1/2 treatment, 1/2 control
• (Versus)
• Random sampling
• Drawing from larger population
• For generalizability to larger population
• External validity (Trust that this would work
  elsewhere)
• Also very important

6
Issues of Instrumentation

• Much emphasis on standardized outcome measures as
  ultimate (valid) criteria
• Less attention to reliability/accuracy of
  legislated tests and measures
• Little attention to how/where/when (or numerous
  other holes in) the data gathered
• Mistrust of teacher self appraisal/reflection

7
Analysis of Impact

• Hypothesis testing
• Is the impact real (not due to chance?)
• P .05, .01, or none (vs. confidence intervals?)
• Effect size as a yard stick of impact
• Report p - level and ES

8
What is Effect Size?

• it is convenient to use the phrase effect
  size to mean the degree to which the phenomenon
  is present in the population, or the degree to
  which the null hypothesis is false. (Cohen,
  1977, p. 9)
• Effect size is a standardized measure of the
  strength (degree of impact) of a discriminating
  feature or intervention.

9
How to Interpret Effect Size

• Cohens d (1965, 1977, 1988) vs. other
• Small (.2), medium (.5) vs. large (.8)
• .2 IQ difference between twins vs. non-twins
• height difference between 15 and 16 yr. old
  girls
• .5 large enough to be visible to naked eye
• height difference between 14 and 18 yr. Old girls
• .8 IQ difference between college freshman and
  Ph.D.s
• height difference between 13 and 18 yr. Old girls
• Compare to other common effect sizes
• As a quick rule of thumb, an effect size of 0.30
  or greater is considered to be important in
  studies of educational programs. (NCREL)
• For example .1 is one month learning (NCREL)

SRI International. http//www.ncrel.org/tech/claim
s/measure.html
10
Effect Sizes are Known for Many Interventions
(Ex Dedes Optimal Areas of Information
Acquisition, 1990)

• 1. Learners construct knowledge rather than
  passively ingest information
• Acceleration (Study 1) 1.00
• Acceleration (Study 2) .57
• Individualized Instruction (Study 1) .32
• Individualized Instruction (Study 2)
• A. Curriculum compacting .83
• B. Credit for prior learning .5
• 2. Sophisticated information-gathering tools are
  used to stimulate the learner to focus on testing
  hypotheses rather than on plotting data
• Higher Order Questions(Study 1) .34
• Cognitive Processing (Study 3) .69
• 3. There is collaborative interaction with peers,
  similar to team-based approaches underlying
  today's science (Note that in constructivist
  methodology the teacher is considered a peer)
• Reinforcement (Study 1) 1.17
• One to One Mentoring (Study 2) .57
• Social Skills (Study 3) .47
• Hancock, R. J. (2003). The Expanded Will, Skill,
  Tool Model A Step Toward Developing Technology
  Tools That Work. Paper presented to EdMedia 2003,
  Honolulu, Hawaii.

11
Issues of Analysis/Interpretation

• Much attention to single correct procedure
• T-test of differences vs. Analysis of Covariance
• Power estimates for hierarchically nested data
• Little recognition of value of multiple views of
  data
• Nonparametric techniques for small samples
• Too much emphasis on accept/reject null and too
  little on strength of effect (ES/APA)
• Tendency to use no data to make decisions rather
  than rely on less than perfect information

12
Its all About Confidence

• As shown in Figure 1, three of the measures 95
  confidence intervals are roughly 3/4 of a
  confidence interval band above that is, no more
  than 1/4 of the 95 confidence interval range
  overlaps from the upper to the lower group.
  Differences in this range are as a rule-of-thumb
  meaningful according to emerging APA
  guidelines, and roughly comparable to a p .05
  level of significance (Cumming, 2003). The effect
  size for the combined upper three versus the
  lower two is approximately ((3.093.052.95)/3)
  ((2.322.41)/2/ ((1.341.331.401.001.05)/5)
  (3.03 2.37) / 1.22 .66 / 1.22 .54,
  considerably larger than the .30 cutoff beyond
  which technology interventions are considered
  meaningful (Bialo Sivin-Kachala, 1996).
  Teachers rated the ARTS to the Delta class as
  much more useful for promoting interest in music
  and creating a positive effect on students
  overall education experience that for improving
  reading and math skills.

13
Weve Long Been in the Credible Evidence Business

• Research based on dissertation criteria
• Quantitative to tell us what is happening
  Qualitative to tell us why it is happening

14
(No Transcript)
15
KIDS Project Technology Innovation Challenge
Grant

• KIDS - Key Instructional Design Strategies
• 9.2 million 1999-2004 TICG
• Replication of successful model to 50 rural
  school districts
• Major Project Components
• Technology Integration Professional Development
• Technology-enhanced Reading Instruction
• External Evaluator Univ. of North Texas
• Research Agenda added to Project Evaluation

16
(No Transcript)
17
1999 - 2002 Findings

• High integration teachers make a critical
  difference for students without computers at
  home.
• Elementary school girls have equal or higher
  attitudes toward computers than boys.
• Technology skills of high school students are
  often higher than their teachers.
• Rural teachers and students fall between Allen
  ISD and Laredo on baseline measures of technology
  proficiency.
• Technology-infused reading activities accounted
  for approximately 10 of reading achievement
  gains.

18
Teacher Stage of Adoption vs. Home Access
2001(SITE 2003 Research Award)
19
Trends in Computer Enjoyment - 2001 sample (Girls
Computers, NECC 2003)
20
2002-2003 Experimental Design

• 7 randomly selected control districts
• Compared with 18 treatment districts
• Interventions
• Summer Institute (Eisenhower Model)
• Tools to integrate into the classroom
• New technology-enhanced reading program

21
Technology Self Efficacy Gain(Pre-Post, 40-hour
Summer Inst.)
22
Technology Self Efficacy Gain(Pre-Post, 40-hour
Summer Inst.)Technology Proficiency
Self-Assessment by Ropp, 1999
23
Is this good?

• In keeping with American Psychological
  Association reporting guidelines (APA, 2001, p.
  25), and in accordance with standards established
  by other scholarly sources, the indications of
  this generalized approach to reporting gains in
  terms of standard deviation units are that the
  summer 2002 professional development institute
  had a moderately large effect on the technology
  integration skills of teachers (Cohen, 1969) and
  resulted in gains well beyond the .3 benchmark
  commonly regarded as indicative of educational
  significance (Bialo, 1996).

24
2002-2003 Student Achievement Findings

• Treatment gained more than controls (plt.05)
• Reading accuracy - Grade 1 and 2
• Reading comprehension - Grade 2
• Average Effect Size .33 (range .23 - .89)
• Many reading programs were used in typical
  classrooms and some were very effective.
• Students of teachers attending Institute gained
  more (2 times ES).

25
Grade 1 vs. 2 Reading Gains
26
Research Design Template

• Evaluation Planned/Required
• Annual report to Dept. of Ed, 5-year summative
• Research Question 1
• Is the KIDS Summer Inst. effective in promoting
  technology integration among teachers?
• Research Question 2
• Is there a positive impact of the KIDS
  technology-based reading program on student
  achievement?

27
Research Design Template (cont.)

• Dependent Variable(s)
• Gains in Level of Technology Integration
  (teachers)
• Reading Achievement Gains (Grade 1-3 Students)
• Independent Variables
• Teachers Before vs. after training
• Students Treatment vs. Control
  classrooms/schools/districts
• Randomization Possible/Control Group
• Yes, 7 districts randomly selected from pool of
  150 matching treatment group membership criteria

28
(No Transcript)
29
Instrumentation

• For Teachers
• Several instruments for attitudes, skills, and
  level of technology integration capability
• Reliabilities range from .78 to .95 for typical
  teachers
• For Students
• Texas Primary Reading Inventory (TPRI)
• Used by more than 90 of Texas districts in K-2
• Reliability reported as high by creators
• Story discontinuity and 2nd grade ceiling effect
  reported by Knezek, Christensen Dunn-Rankin
  (2003).

30
Data Analysis

• T-tests of treatment vs. control gain scores
• Reading accuracy
• Reading comprehension
• Analyis of covariance not carried out due to
  violations of assumptions
• Effect Size computations carried out using
  Cohens d (in spite of paired data)

31
Outlet for Findings

• SITE 2004 -)
• PT3 Leadership Institute Presentations
• Electronic Newsletter/Web Site
• 4th Annual Book on Project Findings nearing
  completion

32
Significance/Implications of Findings

• Technology - enhanced reading can be effective in
  promoting higher achievement in 1st and 2nd grade
  students
• The average effect size is educationally
  meaningful, on the order of an additional 3
  months of achievement gain over 1 school year.
• The model has demonstrated strong evidence of
  effectiveness

33
Has KIDS Established Strong Evidence of
Effectiveness?
34
For Additional Information

• View KIDS Project findings at http//iittl.unt.edu
  
• Contact Gknezek_at_tenet.edu or
• RhondaC_at_tenet.edu