Evaluating how our students think and learn: Research results from the trenches'

1
Evaluating how our students think and learn
Research results from the trenches.

• David Steer Ph.D.
• The University of Akron
• Akron, Ohio
• Research funded by NSF CCLI program and the Ohio
  Board of Reagents

2
What would convince you to use a resource,
technique or curriculum developed by someone else?
3
Agenda

• Method for course redesign.
• Classroom assessment techniques.
• Build a research program into your curriculum
  development.
• Show you some results.

4
What is a personal response system?
Components of a PRS
Power supply
Display
Receiver
Software
Transmitters
available from www.educue.com
Lets give it a test drive . . .
5
Using a personal response system

• How many students do you teach in your largest
  introductory classes?
• 1) 1 - 10
• 2) 11- 25
• 3) 26 - 40
• 4) 41- 100
• 5) 100

6
How do you identify at risk students in your
intro science courses?

• GPA or other known predictor.
• Using a variety of assessments prior to the first
  exam.
• Based on the first exam.
• None of the above.

7
Department Setting and Population

• Teach 2500 students/yr
• Sections of 40 to 160
• Majority non-science majors
• Many education majors
• Large number of at risk students
• Historical attrition 30
• Historical failure rate 25 (F)

8
Observation Conventional instruction does
little to overcome misconceptions, even for our
brightest students. Student understanding must
be challenged to overturn misconceptions.
9
What we did
Began Discussing Student Performance
Created IBL resources
Grant from AS OBR


Grant from NSF CCLI
Complete restructure of all introductory ES
courses and evaluation of IBL techniques
10
Which one of these goals is the most important
in general education courses?
1. Higher Order Thinking develop analytical,
problem solving skills, creative thinking 2.
Basic Academic Success improve listening,
reading, writing, speaking, math skills 3.
Discipline-Specific Knowledge learn facts,
concepts, understand importance of discipline 4.
Liberal Arts, Academic Values develop an
appreciation of cultures, social issues,
ethics 5. Work and Career Preparation develop
leadership, time management skills 6. Personal
Development develop commitment to values,
improve self-confidence
11
What are our teaching/learning goals?
Teaching Goals Inventory Angelo Cross,
Classroom Assessment Techniques (1993)
3.09/3.05
2.83/2.86
2.50/2.27
2.41/2.28
2.29/2.12
2.02/2.16
Four-Year College/University n 951
Community Colleges n 1873
TGI is available on-line at http//www.uiowa.edu/
centeach/tgi/index.html
12
What is the highest level of learning that
students routinely demonstrate in your general
education classes?
memorization and recall understanding using
knowledge taking apart information reorganizing
information making judgements
Content
1. Knowledge 2. Comprehension 3. Application 4.
Analysis 5. Synthesis 6. Evaluation
Higher-order thinking
13
Authentic Assessment? Are our assessment methods
meeting our teaching and learning goals?
14
Teaching and learning goals were ordered using
Blooms Taxonomy
memorization and recall understanding using
knowledge taking apart information reorganizing
information making judgements
Text
1. Knowledge 2. Comprehension 3. Application 4.
Analysis 5. Synthesis 6. Evaluation
In class rapid assessments
In (and out) of class exercises
15
Comprehension Example using Conceptests
How would the amount of incoming solar
radiation change during summer in Ohio
if the tilt of Earth's axis increased
from 23.5
to 30 degrees?
1.
Most of my students would get this.

2.
My students would do OK on this.

3.
Most would miss this.

4.
I would not ask such a question.
16
Applications Example using Diagram Analysis
youngest
oldest
17
Application Have students draw the diagram from
a list of events.
Student drawn diagrams
18
Analysis Example using Venn Diagrams
Venn diagrams can be used to identify the
characteristics of classification systems

• Analyze the key components of complex sets of
  geological features
• Example Use descriptions of specific volcanoes
  to identify characteristics of stratovolcanoes
  and shield volcanoes

19
Synthesis Example using Concept Mapping
A concept map is a graphical representation of a
student's knowledge about a topic.

• Two principal components
• Key terms or concepts
• Directional links (arrows) and linking phrases
  that connect the terms

20
Evaluation Example using Conceptests
Liquid hazardous waste is
disposed off by pumping it
down injection wells.
Which well location would
be the most suitable to
use for an injection well?
Think-pair-share Predict how students would
answer this.
21
Teaching and learning goals were ordered using
Blooms Taxonomy
memorization and recall understanding using
knowledge taking apart information reorganizing
information making judgements
Text
1. Knowledge 2. Comprehension 3. Application 4.
Analysis 5. Synthesis 6. Evaluation
Concept tests
Venn Diagrams
Concept Maps
Evaluation Rubrics
Open-ended questions can be used for all
categories.
22
How can we tell if these techniques are useful?
23
Group Assessment of Logical Thinking
12 question instrument that tests six logical
operations Conservation Proportional
Reasoning Controlling Variables Combinatorial
Reasoning Probabilistic Reasoning Correlation
Reasoning
Roadrangka et al., 1982
24
Which statement best characterizes students
taking introductory science courses?

• 1) Prefer studying relationships and unseen ideas
• I prefer to develop my own approach.
• 2) Prefer to apply ideas in a practical way
• What does this have to do with me?
• 3) Prefer cookbook approach (what, how, when)
  Does this method work every time?

25
Levels of cognitive development at open
enrollment, public university
Intro Science 2001-02 n316 Concrete 0-4
(25) Transitional 5-7 (30) Formal
8-12 (45)
Data from The University of Akron Department of
Geology
How might this influence resource development
targeting at risk students?
26
Typical IBL Lecture

• Introductory exercise or questions to stimulate
  discussion
• Lecture broken into one or two 10 15 minute
  segments
• Advanced exercises to assess understanding and
  progress
• in Groups of 3-4

Steer et al., AGU, December 2001
27
Changes to logical thinking
7.4 overall increase (p lt 0.00001)
n 377
28
How do GALT scores relate to performance?
GALT Exam 1 Exam 1 Score
Content Questions Analysis Questions
1-4 59 51 5-7 67
60 8-12 74 69
Exam 1 Scientific Method, Rock Cycle, Geologic
Time
NOTE These results for Steer only (n 94)
Implication GALT is predictive
29
Who are we affecting?
Concrete thinkers experienced the greatest change
in logical thinking skills. Formal thinkers
experienced no change.
Based on paired T-tests
30
GALT score vs. Course grade
What other variables are at work?
General positive trend between GALT score and
student achievement
Course Grade, A/B
57
19
(0.6)
(1.04)
GALT score 0 - 4
GALT score 8 - 12
19
38
(5.3)
(7.8)
Course Grade, D/F
F 01 (n83)
31
Do groups really matter?
32
How are we affecting them?
Pre-GALT Score
N360
Active Learning
Control
N121
33
Conclusions

• Can predict student success on day 1.
• Many concrete thinkers in your class.
• Active learning techniques help all students.
• Group collaboration is particularly important.
• Can build a research program into your curriculum
  development.

NOTE If you intend to collect data, complete an
IRB application!