Evaluating Creative Teaching Practices and Collaborative Learning Environments

1
Evaluating Creative Teaching Practices and
Collaborative Learning Environments
Ann Mastergeorge and Ingrid Roberson
UCLA Graduate School of Education Information
StudiesCenter for the Study of
EvaluationNational Center for Research on
Evaluation, Standards, and Student Testing
Annual AERA Convention April 1, 2002 New Orleans,
LA
2
Overview

• The Construct of Engagement in School Reform
• Methodological Issues in Measuring Engagement in
  School Reform
• Discussion and Implications of Engagement for
  Classroom Practice

3
Introductionto Creative Learning Communities

• Supports school reform in schools across the
  country serving at-risk student populations
• Promotes change centered on creative teaching
  practices and collaborative learning environments
• Emphasizes the enhancement in students joy of
  learning in engaged learning environments

4
The Construct of Engagementin School Reform

• Procedural Engagement
• Focuses on student on-task behavior in learning
  activities
• Emphasizes student concentration and completion
  of learning activities
• Substantive Engagement
• Focuses on the emotional quality of students
  involvement in learning activities
• Emphasizes students enjoyment of learning
  activities since it is self-motivating and
  self-reinforcing

5
Developing A Dynamic Model of Engagement

• Substantive Engagement depends on both student
  and teacher engagement in classroom learning
  activities
• Includes the social dimension of learning
• Requires a high degree of reciprocity
• Learning Environments and Teaching Practices
  construct opportunities for students to
  experience engagement
• Communicates support and care
• Allows for student choice and control
• Models enthusiasm for learning

6
 Creative Learning Community
Learning Communities
Teaching Communities
Learning links
Teaching links
Shifts in attitude
Shifts in attitude
Classroom Collaboration
Student Engagement
Teacher Engagement
7
Methodological Issues in Measuring Engagement in
School Reform

• Multiple Methods of Evaluation
• Utilizes quantitative and qualitative approaches
• Implementation Process of School Reform
• Impact of School Reform on Engagement
• Employs a theory of action
• Provides focal points for the evaluation
• Suggests a sequence in which various effects may
  be expected

8
Leadership Goals
Anchor in curriculum, instruction assessment
Attend to cultural and structural change
Facilitate change
Connect to student learning
Build a shared vision
School Environment
Mobilize change
Attend to changing attitudes, beliefs, values
Provide materials
Provide Financial Support
Build a community
Provide Technical Support
Embed in school culture
Professional Development Accountability
Align with project goals
Provide the type and timely information
for project adjustments
Promote and provide time for on-going
professional inquiry
Provide opportunities for teacher collaboration
Develop measures aligned with creative teaching
practices
Creative Teaching Practice
Embed in classroom practice
Shift teacher attitudes, beliefs, and
expectations
Align with assessments (standardized and
performance)
Align with standards
Link to curriculum
Identify innovative practices
Embed in school culture
Engaged Student Learning
Promote the joy of learning through
exploration, collaboration, and
interpretation Improve student attitudes towards
learning
9
Measuring the Implementation Process

• Features Five Dimensions of School Reform
  Framework
• Goals
• Leadership
• School Environment (Culture and Structures)
• Professional Development
• Accountability
• Focuses on the implementation of creative
  teaching practices
• Teacher Survey
• Teacher Interview
• Classroom Practice Observation Protocol

10
Teacher Survey
11

Teacher Interview
12
Measuring the Impact on Engagement

• Features both Procedural and Substantive
  Engagement Theory
• Focuses on Student Engagement
• Teacher Survey
• Teacher Interview
• Student Survey
• Classroom Practice Observation Protocol

13
Teacher Survey
14

Teacher Interview
15
Student Survey
16
Classroom Practice Observation Protocol

• Sensitive to initial changes in teaching
  practices
• Appropriate for innovative and creative teaching
  practices
• Captures students attention to and enjoyment of
  learning activities

17
Dimensions of Classroom Practice

• Challenge of Lesson Activity
• Implementation of Lesson Activity
• Procedural Engagement
• Substantive Engagement
• Quality of Instructional Discussion
• Quality of Instructional Feedback

18
(No Transcript)
19
(No Transcript)
20
Challenge of Lesson Activity Exemplar
Teacher Lets think about all of the different
conditions. Student My experiment needs more
water. Teacher I heard someone else say too
cold so cold or warm. Student Ours is darker
and theirs is lighter amount of light
. Teacher Can you think of anything
else? Student I think we should put less water
and less worms.. with the worms they will just
dry, will shrivel and do nothing and the water
will just get everything damp. Teacher What do
you think is the most important thing to make
the leaves decompose? Student Dry, the
leaves crumble up. Its like they turn into
dirt. Teacher Maybe one thing we can do this
week, half put water and the other half put no
water and keep everything else the same.
(CO.10.08.YR2)
21
(No Transcript)
22
(No Transcript)
23
Implementation of Lesson Activity Exemplar
After reviewing both the proof reading process
and scoring rubric (both clearly written on chart
paper), teacher runs through following
example. Dear Curator I really love the ring
of the bell. I also thought the water wheel
was cool. Finally, the machines were interesting
to see. Thank You, Ms. X Teacher Alright, so
this letter Based on that, where would you
put them if you were the teacher? Student 2,
though it does have an introduction. Student 2,
it doesnt give them a sense and is supposed to
be in paragraphs. Teacher Anybody agree or
disagree and why? Student It looks like a
letter but it doesnt sound like a letter
(CO.01.03.YR2)
24
Procedural Engagement
1
2
3
4
NA
10-29 of the
30-59 of the
60-79 of the
80-100of the
students appear
students appear
students appear
students appear
Not
to be procedurally
to be procedurally
to be procedurally
to be procedurally
appli-
engaged.
engaged.
engaged.
engaged.
cable
to this
Example
Example
Example
Example
activity
Few students
Some students
Most students are
Almost all students are
conduct their light
conduct their science
on-task, conducting
on-task, trying to get the
bulb experiments
experiment but about
their experiment.
light bulb to work, using
while the rest of the
half of the class
They write their
different metal wires.
students wander
wanders around the
science notes,
They note their
around the room,
room or sits at their
detailing their steps
observations in their
chatting with other
desks chatting with
and results. Most
science journal. Most
students about
other students about
students complete the
students complete both the
unrelated topics or
unrelated topics or
experiment and most
experiment and their
play with each
stares out into space
complete or nearly
writing-up of the science
other.
day dreaming,
complete their
experiment in their
science notes.
journal.
Approximately what percent of procedurally
engaged?
25
Procedural Engagement Exemplar
All students are on-task, attentive to the
teacher. Some students raise their hands to
answer teacher questions. Teacher Is this a
sentence? Student Yes. Teacher So what does
it have to have at the beginning? Student A
capital letter. Teacher What does it have to
have at the end? Student A question
mark. Teacher What do I have to do to show
these words came out of my mouth? Three
students raise their hand...come up to the
board X. Student goes to board and puts
quotation marks around the sentence. (CO.02.01.YR2
)
26
Substantive Engagement
1
2
3
4

NA
60-79 of the
80-100 of the
10-29 of the
30-59 of the
students appear
students appear
students appear
students appear
Not
to be substantively
to be substantively
to be substantively
to be substantively
appli-
engaged.
engaged.
engaged.
engaged.
cable
to this
Example
Example
Example
Example
activity
Most student are actively
Almost all student are actively
Most students are
About half of the
engaged. Some students
engaged. Many students shout,
either not doing the
student are not
talk excitedly with their
"It works!, I did it! or Its alive!"
experiment, playing
actively engaged,
neighbors saying, "Did
when their light bulb works. Some
or talking with other
even if they are
you see that?, That is
students smile, their eyes
students or if
conducting the
pretty neat Did yours do
widening, as their light bulb goes
conducting the
experiment, their
that?" Other students,
on. Other students still trying
experiment, they are
attention is focused
smile to themselves as
review their notes or ask another
rushing through it,
elsewhere, listening or
their light bulbs lights up,
student, Hey, can you help me?
being careless,
talking to other
while others still trying
Near the end, students can be
paying more attention
students, playing with
scrunch up their faces in
heard talking with each other
to unrelated talk.
their pencils, looking
deep concentration.
about which metal worked better
outside the window.
as a conducter.
Approximately what percent of students are
substantively engaged?
27
Substantive Engagement Exemplar
The students are all listening. They are
riveted, looking at the teacher and the book
intently. The teacher engages them in a
discussion with 90 of the students
participating. Students hands shoot up in the
air waving back and forth frantically with
wiggling fingers, going "oh, oh. Teacher What
woke him up? Student The sun. Teacher X, what
wakes you up? Student My mom. Student
Sometimes my dog comes in and licks me. Student
Sometimes my brother. Student My moms
boyfriend hits me with a pillow. Student My dad
gets me up. Teacher So our caterpillar comes
out of the egg and how did he feel? He was
what? Students HUNGRY!!!
(CO.02.03.YR2)
28
(No Transcript)
29
(No Transcript)
30
Quality of Instructional Discussion Exemplar
At one table, there are three rather lively boys
and as each of them works on drawing two parallel
lines the following discussion ensues Student
2 Thats parallel, referring to the littlest
ones two parallel lines. Student 4 Can I
make it parallel? Student 2 That is parallel.
It doesnt matter the size of the other line,
if they dont intersect, then theyre
parallel. Student 1 If they dont intersect,
theyre parallel. The little one is still
unconvinced by the two boys, looking at his board
with a confused expression. Student 2
continues to teach the other student the concept
of parallel lines, emphasizing the word intersect
and then upon seeing the continued state of
confusion lets out a loud sigh and starts again.
He explains that the length of the line segment
or how close they are doesnt matter as long as
they dont intersect.
(CO.05.03.YR2)
31
(No Transcript)
32
(No Transcript)
33
Quality of Instructional Feedback Exemplar
Teacher Did you measure first? If youre going
to do me, put my name in, and then youll have to
do an estimate. Both students quickly go to
their papers to write down their estimates on a
small piece of paper. Teacher Now this is in
meters, upon seeing one student write down 5.
The student nods knowingly but continues to
write down the number 5. Teacher Do you think
Im 5 meters high? He looks at her
questioningly, not understanding. So she takes
the yardstick and starts to raise it slowly.
Teacher One meter. Raises it a meter. Two
meters. Raises it again. Three
meters. Student Oh, two! He vigorously
erases the number five and writes down the number
two. (CO.06.01.YR2)
34
Comparison of All and Matched Mean Ratings
35
Comparison of Year 1 and Year 2 mean ratings by
domain across all case study schools
36
Comparison of Year 1 and Year 2 matched mean
ratings by domain across all case study schools
37
Discussion and Implications for Classroom Practice

• Documenting Shifts in Classroom Practice from
  Year 1 to Year 2
• Issues in Accountability and Sustainability of
  School Reform
• Re-focusing, Re-thinking, and Re-tooling Measures
  of Classroom Practice

38
Documenting Shifts in Classroom Practice

• Overall, small effect on Teacher Practice (Effect
  Size of .33) and Student Engagement (Effect Size
  of .201)
• Between Schools, difference in degree of
  implementation of creative practices
• Professional development linked to promoted
  teaching practices
• Assessments to monitor progress

39
Issues in Accountability and Sustainability in
School Reform

• Combining and creating assessments that are
  accessible for progress monitoring
• Improvements
• Inquiry
• Creating and sustaining supportive
  infrastructures to sustain school reform
• Leadership
• Community Culture

40
Re-focusing, Re-thinking, and Re-tooling
Measures of Classroom Practice

• Re-focus efforts to include realistic
  expectations for reform implementation
• Re-think the product and outcome model of
  classroom measures measures should reflect
  process in action
• Re-tool measures to reflect conceptual and
  procedural realities of classroom practice

41
Paper will be posted on the UCLA CSE/CRESST
website www.cse.ucla.edu 

• Ann Mastergeorge
• Phone [phone]
• E-mail ammastergeorge_at_ucdavis.edu
•    
• Ingrid Roberson
• Phone [phone]
• E-mail roberson_at_cse.ucla.edu