Joint Committee on Standards for Educational Evaluation

1
Joint Committee on Standards for Educational
Evaluation

• Evaluation Standards
• Joan Kruger
• Spring 2008

2
Joint Committee on Standards for Educational
Evaluation

• http/www.wmich.edu/ evalctr/jc
• SPONSORING ORGANIZATIONS
• American Association of School Administrators
• American Counseling Association
• American Evaluation Association
• American Educational Research Association
• American Indian Higher Education Consortium
• American Psychological Association
• Canadian Evaluation Society
• Canadian Society for the Study of Education
• Consortium for Research on Educational
  Accountability and Teacher Evaluation

• Council of Chief State School Officer
• Council of the Great City Schools
• National Association of Elementary School
  Principals
• National Association of School Psychologists
• National Association of Secondary School
  Principals
• National Council on Measurement in Education
• National Education Association
• National Legislative Program Evaluation Society
• National Rural Education Association

3
What is it?

• Created in 1975, the Joint Committee (JC) is a
  coalition of major professional associations
  concerned with the quality of education
• It is incorporated in the US as a private
  non-profit organization
• It is accredited by the American National
  Standards Institute (ANSI) and all standards
  become certified by ANSI

4
What is its mission?

• To promote concern for evaluations of high
  quality based on sound evaluation practices and
  procedures
• To meet existing and emerging needs in the field
  of evaluation

5
Where is it located and what is the role of CES?

• The Joint Committee is housed at the Evaluation
  Center, Western Michigan University
• Canadian Evaluation Society has been a Sponsoring
  Organization since 1994 and participates as a
  voting member, participates in special projects,
  and promotes all Standards (Program, Personnel
  and Student) to CES members.

6
Direct Products of the JC

• Publications 3 sets of evaluation standards
• Personnel Evaluation Standards. 2nd ed. 2008.
  Corwin Press
• Program Evaluation Standards. 2nd ed. 1994.
  3rd in testing. Sage
• Student Evaluation Standards. 2003. Corwin
  Press.

7
Associated Accomplishments

• Adoption and adaptation by schools, states, and
  organizations nationally and internationally.
• Numerous associated materials (e.g., Checklists
  for metaevaluation).
• http//www.wmich.edu/evalctr/checklists/
• Widespread translation and/or dissemination
  internationally, for use in Europe (especially
  Germany and Switzerland, Latin America, Africa
  and others)

8
Pervasive Nature of Program Evaluations

• Primary tool for guiding program evaluation,
  crossing all academic disciplines and courses
• Evaluations regularly clients and stakeholders in
  a wide variety of decision situations.

9
Impact of Program Evaluations

• Evaluations done well can be of significant
  service.
• Clients are victims of and harmed by poor
  evaluations in both high- and low-stake
  evaluations

10
Barriers to Sound Program Evaluations

• Inadequate preparation
• Inadequate administrative and technical support
• Professional disagreements
• Lack of shared language
• Inadequate policy

11
Why do we need Program Evaluation Standards?

• What is available?
• Assessment measurement design
• Performance management systems
• Standards for student and personnel evaluation

• What are the gaps?
• No standards for the programs evaluations across
  disciplines.
• No desktop reference to ensure quality

12
What is a Standard?

• A principle commonly agreed upon by experts in
  the conduct and use of evaluation for the
  measurement of the value or quality of an
  evaluation.
• A standard is seen as a principle governing good
  practice vs. a rule
• Technical guide to conduct evaluations
• Criteria of good practice for those who receive
  and are affected by an evaluation

13
What is a Standard?

• A guideline may provide the evaluator with a
  recommendation to help ensure basic quality
  evaluation. Other evaluators, in the same
  situation in the same or similar institutions
  following the guidelines, may also ensure basic
  quality.
• The CES felt standards used as guidelines would
  help evaluators make more consistent quality
  judgements over time.

14
What is a Standard is not.

• Rules are much more restrictive. The evaluator
  must or should conform to rules. This may lead
  to a narrower rating of a program or project by
  not allowing the evaluator to adapt the standard
  to the norms of the evaluation setting.
• Rules are usually policed by a professional body,
  and offenders may be reprimanded.

15
How are the Standards Organized?

• All three sets of Standards are organized into
  four main categories
• Utility
• Propriety
• Feasibility
• Accuracy

16
Utility

• To ensure that an evaluation will serve the
  information needs of intended users.

17
Propriety

• To ensure that an evaluation will be conducted
  legally, ethically, and with due regard for the
  welfare of those involved in the evaluation as
  well as those affected by its results.
• Additional resource Canadian Evaluation Society
  Ethical Guidelines
• http//www.evaluationcanada.ca/site.cgi?s5ss4_
  langEN

18
Feasibility

• To ensure that an evaluation will be realistic,
  prudent, diplomatic, and frugal.

19
Accuracy

• To ensure that an evaluation will reveal and
  convey technically adequate information bout the
  features that determine worth and merit of the
  program being evaluated.

20
How are the Standards Presented in the Book?

• A uniform pattern is followed
• Standard Statement
• Overview (Explanation, Rationale, Caveats)
• Guidelines
• Common Errors
• Two illustrative case studies
• Supporting documentation (references)

21
Number of Standards(Program proposed 3rd ed)

Category Program Personnel Student
Utility 8 5 7
Propriety 7 5 7
Feasibility 4 3 3
Accuracy 8 8 12
TOTAL 27 21 29
22
National Public Hearings

• National Public Hearings The Joint Committee
  will hold open forums at meetings of each of the
  sponsoring organizations to gain additional
  feedback from members of these organization (See
  5.3.5 of the Operating Procedures-).
• 5.3.5 National Public Hearings
• National Public Hearings shall be held to provide
  an open forum for discussion and critique of the
  standards. Hearing schedules shall be publicized
  and all interested parties will be encouraged to
  participate. In addition, each Sponsoring
  Organization will be asked to select members of
  their organization and encourage them to respond
  to the draft through the vehicle of the hearings.
  (NOTE If the standards under consideration are
  to be submitted to the American National
  Standards Institute for approval as American
  National Standards, the Joint Committee shall
  also arrange for announcement of the draft in
  ANSI's STANDARDS ACTION for comment, in
  accordance with Section 1.2.6 of the ANSI
  Procedures for the Development and Coordination
  of American National Standards.)

23
Process for Hearing

• Overview of the Standards
• Comments from CES members and others (criticisms,
  observations, recommendations)
• Comments from Field test participants

24
Further Communication

• Visit the JC web site at
• http//www.wmich.edu/evalctr/jc/ and enter ES
  review page.
• Contact Arlen Gullickson arlen.gullickson_at_wmich.ed
  u

25
Next Steps

• Complete field tests
• Complete National Hearings
• Revise
• Final Approval by the Joint Committee
• Submission to American National Standards
  Institute for certification
• Publish

26
Evaluation Web Sites

• Canadian Evaluation Society (CES)
• http//www.evaluationcanada.ca/
• Canadian Evaluation Society Educational Fund
  (CESEF) for students and those new to eval.
• http//www.evaluation-education.org/index.html
• Saskatchewan Chapter, CES
• http//sk.evaluationcanada.ca/
• American Evaluation Association (AEA)
• http//www.eval.org/