Theory and Methods in OST Research

1
Theory and Methods in OST Research
Denise Huang
American Educational Research AssociationAnnual
Meeting AERA San Diego, CA - April 14, 2006
2
Experimental Qualitative Evaluation
Purpose New knowledge, Truth To uncover and understand what lies behind any phenomenon Mission achievement, product delivery
Outcome Generalizable conclusions Emergence of theory Specific decisions
Value Explanatory and predictive power Rich descriptive data Insiders point of view Determining worth and social utility
Impetus Curiosity and ignorance Understanding Needs and goals
Conceptual Basis Cause and effect relationships Experiences as a whole, in context, interactive Means ends processes
Key event Hypothesis testing Observations of multiple, shifting axioms Assessing attainment of an objective
Classic paradigms Experimental method Correlational method Grounded theory approach System approach Objective approach
Discipline Control manipulation of variance Iterations Program planning and management
Criteria Internal and external validity Fit, understanding, generality, and control Isomorphism and credibility
Functional type Pure and applied True experimental Quasi-experimental Grounded theory Ethnography, Phenomenon approach Case studies, etc. Formative summative Process product
3
What do all these mean?

• The gold standard for research design and
  analysis involve experimental randomization.
• Satisfactory research findings required
  generalization and replication.
• The afterschool population has distinct
  characteristics
• Ethical issues
• Self-selective group
• Difficulty in obtaining comparison group
  (consent forms)
• Accuracy of data and availability of
  longitudinal records
• High transience
• Qualitative strategies are vital tools in finding
  out the intricate rich data about a program and
  help to explain, elaborate, and triangulate
  quantitative findings
• Matching study questions with design and
  appropriate methodology

4
Most afterschool studies are evaluation research

• The systematic application of social research
  procedures for assessing the conceptualization,
  design, implementation, and utility of social
  intervention programs (Strauss Corbin, 1990).
• Focus on finding and explaining educational
  /social effects or outcomes from the
  intervention, and
• Devising instructional strategies that will
  improve the outcomes.
• Formative and summative evaluations
• Process and impact evaluations

5
Different types of evaluation research

• Process Evaluations
• Formative studies
• Evaluations that assesses the conduct of the
  program during the initial design and testing
  stages with the intent to improve the program
• Program monitoring
• Systematic examination of program coverage and
  delivery- (target population, fidelity,
  efficiency)
• Identifying successful implementation strategies
  for program diffusion

6
Impact Evaluations

• Assesses the changes in the well-being of
  individuals that can be attributed to a
  particular intervention, such as a project,
  program or policy (Strauss Corbin, 1990).
• Summative evaluation
• Summative evaluation provides information on the
  product's efficacy ( it's ability to do what it
  was designed to do)
• By looking at the intervention group, the
  evaluator can examine the learning materials and
  learning process together with the outcomes--
  hence the name Summative Evaluation.
• Impact Evaluation
• Impact evaluation involves constructing a
  counterfactual
• Random selection and isolation from interventions
  are seldom practicable and sometimes ethically
  difficult to defend.
• Quasi-experimental method is often used.

7
Which theoretical perspective informed your work,
and why did you select this theory?

• Goal setting theory
• Social cognitive theory
• The contextual understanding of the social
  ecological model

8
The role that theory play in research

• Theory is an invention aimed at organizing and
  explaining specific aspects of the environment
  (Pedhazur Schmelkin, 1991)
• Theory provides the researcher with a selective
  point of view-an orientation
• Helps to determine what variable is relevant and
  which are not relevant
• Being a way of seeing, a theory is also a way of
  not seeing.

9
How did this theory shape your research methods?

• Mixed methods
• Process/outcome
• SEM
• HLM

10
Examples Formative, Process, Summative

• The ASL/KidzLit Formative Study
• ASL is a reading and writing project that was
  based on the theoretical foundation of the
  literature-based instructions (social cognitive
  theory)
• How do the LAs BEST teachers evaluate the ASL
  project in terms of training, materials, and
  support?
• To what extent have staff capabilities been
  enhanced after the ASL training?
• If the staff members teach during the regular
  school day and in the after-school program, to
  what extent have they carried the ASL strategies
  into their regular school day teaching?
• What changes in student attitudes might be
  associated with ASL? Are there any preliminary
  indicators that participation in ASL is affecting
  students?

11
Teacher survey I-formative evaluation

• For the effectiveness of the ASL training, the
  evaluation based on ASLs four core components
• Care
• Talk
• Read
• Connect
• To evaluate the ASL project as a whole, teachers
  perspectives were examined
• ASL materials
• ASL training
• Support for the ASL project
• For an affective/motivational outcome the
  following indicators were examined
• Students enjoyment
• Teacher attitude

12
The student survey

• Based on the social cognitive theory that
  learning is a social phenomenon and social
  collaboration is essential in the classroom
  climate, we investigate on students
• Enjoyment of their reading class
• Perception of the social support in the
  classroom
• Perception of their relation with their teachers.
• Based on the motivational theory, we look at the
  students
• Attitudes towards reading and writing and
• Reading and writing self-efficacy.
• And students perception of the classroom
  climate
• The autonomy orientation of their classroom
  climate and
• The emphasis on effort allocation.

13
Qualitative study

• Exploring the intellectual, social, and
  organizational capitals at LAs BEST

14
Quantitative outcome evaluations

• Keeping kids in school study
• 4 cohorts of students (all LAs BEST participants
  and a stratified random sample of
  non-participants)
• 3 participation level ( 1,2,3 years)
• 6-9th grade in 1998-1999 followed through to
  2002-2003
• Chi-square analysis
• Cox survival analysis

15
Comparison of dropout rates for LAs BEST vs.
LAUSD non-participants (3 Years).
16
The Afterschool Hours-SEM and HLM

• Examining the Relationship between Afterschool
  Staff-Based Social Capital and Student Engagement
  in LAs BEST
• What are LAs BEST staff perceptions of
  collective staff efficacy, teamwork, and
  communication, and the quality of their
  relationships with students?
• What are student perceptions of their
  relationships with LAs BEST staff? To what
  extent do they value education and have high
  aspirations for their futures? What are their
  reported levels of student engagement in LAs
  BEST and the day school?  
• How are staff-student relationships, teamwork and
  communication, and collective staff efficacy at
  the site level, related to student perceptions of
  their relationships with staff?  
• What is the association between student
  perceptions of their relationship with staff,
  their value of education, future aspirations, and
  engagement in the afterschool program and day
  school?

17
Hierarchical Linear Modeling

• Examine the relationship between the social
  capital predictors measured by the staff surveys
  (i.e., staff-student relationships, collective
  staff efficacy, and communication and teamwork)
  and student perceptions of social capital (i.e.
  staff-student relationship) as measured by the
  student surveys.
• The demographic variables of gender, grade level,
  and languages spoken were also included in each
  model to control for individual student
  differences.
• A total of 2,270 students and 395 staff from 50
  school sites were included in the HLM analysis.

18
Structural Equation Modeling
19
DOJ study-using propensity matching
cost-benefit analysis

• Quasi-experimental design
• Longitudinal sampling of academic and juvenile
  crime data
• Advanced multilevel propensity scores methods to
  establish study samples
• Hierarchical growth modeling and survival
  analysis (multilevel discrete-time hazard)
• Students were followed from 1994-1995-2002-2003
  school years
• Cost-benefit analysis

20
How did your method(s) contribute to our
knowledge of OST, and what were the limitations
of the method?

• Better comparison samplepropensity matching
• The importance of dosage
• Importance of goal setting
• Importance of employing a continuous improvement
  model

21
Why results are inconsistent?

• Linking results to program content
• __Design__________________________________________
  _
• Linking results to implementation
  procedures(program fidelity)
• __Treatment_______________________________________
  _
• Linking results to program attendance
• __Dosage__________________________________________
  _

22
What does that theory mean for what you would do
with children and youth during the out-of-school
time and inform program design?

• Importance of having
• A theory of change
• A logic model
• A continuous improvement strategy

23
Why create a logic model?

• They are powerful tools for designing, planning,
  implementing, and evaluating OST programs
• They set up a plan of how a program is expected
  to work
• Provide a map of how to achieve goals (cause and
  effect-program process and outcomes)
• Stimulate clear thinking, preparation, and
  organization

24
Example Theoretical-based logic model designed
for the Afterschool Partnership study


Process

Content
Process
Structure
25
Example Outcome based logic model
Indicators
Design Process Immediate
Outcome Expected Outcome Long-term
Outcome
Linkage with School
Achievement
Professional Development
26
Using data for continuous improvement
Data Based decision Making Process
Theory and Research-based Programming
Analysis of Results
Student Engagement Student Retention
Student Improvement
Plan of Action
Periodic Assessment
Specification of Monitoring
Continuous Adjustment
27
Looking Forward

• Gap in knowledge as we are working to develop the
  21st century skills
• Technology divides
• The contextual environmental
• Social capital and human networking system

28
dhuang_at_cse.ucla.edu