SLS 730 Seminar

1
SLS 730 Seminar Second Language Program Evaluation
Professor John M. Norris Spring 2006
2
Language Program Evaluation Scoping out the
domain
How core is L2 program evaluation work in
applied linguistics and/or language teaching?
What are some of the definitions of evaluation,
from mainstream and L2 perspectives?
What is the purpose of evaluation? How
contentious is this question?
Who are some of the main players in mainstream
and L2 program evaluation?
What are some of the main professional resources
available for L2 program evaluators?
What does the future hold for L2 program
evaluation?
3
Language Program Evaluation How core is it?
one e.g.
with chapters on program evaluation
Overview books on TBLT
Nunan (1989) Crookes Gass (1993) Willis
(1996) Skehan (1998) Lee (2000) Bygate, Skehan,
Swain (2001) Ellis (2003) Leaver Willis
(2004) Nunan (2005)
Ellis (2003)
4
Evaluation, assessment, measurement confusions
1. No hard and fast definitions or distinctions,
but
2. Distinct professions and action emphases, so
3. Different discourse communities different
meanings, therefore
4. Be careful to define exactly what you mean
whenever you use one of these terms!
5
Program Evaluation A few definitions
Lynch (2003) the systematic inquiry into
instructional sequences for the purpose of making
decisions or providing opportunity for reflection
and action
Patton (1996) the systematic collection of
information about the activities,
characteristics, and outcomes of programs to make
judgments about the program, improve program
effectiveness, and/or inform decisions abobut
future programming
Scriven (1997) process of determining the merit,
worth, and value of things
6
Program Evaluation Norris generic definition
gathering trustworthy information about program
elements in order to understand, demonstrate, and
improve program value
7
Program Evaluation Purpose
Deliberate
Generate Knowledge
Communicate
Hold Accountable
Revise
Purpose of program evaluation
Improve
Transform
Empower
Illuminate
Judge
Develop
Advocate
Test Theory
Educate
8
Program Evaluation Purpose
Pragmatic tenets of contemporary program
evaluation

• Multiple purposes, multiple methods
• Purpose of each evaluation made explicit
• Methods articulated to specific purpose
• Findings/outcomes/processes utilized to meet
  purpose

9
A few of the mainstream names
Scriven formative/summative distinction
goal-free evaluation
Stake advocacy, responsive evaluation, case
study
Cronbach knowledge-generation evaluation
Guba Lincoln qualitative evaluation
Weiss knowledge-creep, enlightenment
evaluation
Fetterman empowerment, participatory
evaluation
Cook Campbell program theory evaluation,
experimenting society
Chelimsky political evaluation
House deliberative-democratic evaluation
and many, many, others
10
L2 Program Eval Generation 1
Brian Lynch http//web.pdx.edu/bklynch/
Pauline Rea-Dickens http//www.bris.ac.uk/educat
ion/people/academicStaff/edprd
J.D. Brown http//www.hawaii.edu/sls/brown/index.
html
Charles Alderson http//www.ling.lancs.ac.uk/staf
f/charles/charles.htm
Richard Kiely http//www.bris.ac.uk/education/peo
ple/academicStaff/edrnk
Alan Beretta http//www.msu.edu/beretta/
11
L2 Program Eval Generation 1
See also work by Geoff Brindley, Ron Mackay,
Cyril Weir, and others
??? GENERATION 1.5 ???
12
Main journals
Mainstream eval journals American journal of
evaluation American educational research
journal Assessment and evaluation in higher
education Educational evaluation and policy
analysis Evaluation Educational research and
evaluation New directions for evaluation Practical
assessment, research, and evaluation among many
others!!!
L2 eval journals 0???
But, read occasional L2 evaluation work
in Applied linguistics ELT journal English for
specific purposes Foreign language
annals Language assessment quarterly Language
teaching research Language testing Modern
language journal TESOL Quarterly
13
Main professional organizations
American Evaluation Association
http//www.eval.org
Hawaii Pacific Evaluation Association (local
affiliate of the AEA, under organization)
European Evaluation Society http//www.europeanev
aluation.org
Some focus in L2 associations AAAL, TESOL,
ILTA, IATEFL, etc.
14
Main conferences for presenting eval
Annual meeting of the AEA http//www.eval.org
Annual meeting of the HPEA (1st meeting in Fall
2006)
Annual LTRC http//www.iltaonline.com
Annual TESOL convention http//www.tesol.org
Annual AAAL http//www.aaal.org
Annual ACTFL http//www.actfl.org
15
What does the future hold?
LAMPE degree and focus in UH DSLS.
FL program evaluation project here at UH.
Organization and participation in HPEA.
UH NFLRC summer workshop, research, etc.
Special issue of LTR. Perspectives focus of MLJ.
ADFL focus sessions, 2006 (Madison), 2007 (at
UH)
Your contributions???