Title: An Evaluation Tool for Development Education Consultation Day
1An Evaluation Tool for Development
EducationConsultation Day
- September 25th 2009, University of Limerick
2Why Participate?
- Irish Aid funded project to develop DE/ESD
Evaluation Tool for possible use as a standard
tool to compare programmes across Ireland - Help to design it to the specific needs of the
Irish DE/ESD Sectors - This tool will increase transparency,
communication, and possible comparisons - Allow for much-needed benchmarks and baselines to
be established
3Objectives of the day
- become familiar with general tool parameters and
strengths and limitations of quantitative tools - Identify and prioritise outcomes of successful
DE/ESD programmes - Identify optimal operational factors of a tool,
e.g. optimal time to complete the tool - Consider piloting this tool within your
organisation
4Project Background
- No one standard assessment to measure impact of
work carried out in formal or non-formal DE/ESD
sectors. -
- RCE-Irelands research project aims to develop a
quantitative evaluation tool for DE/ESD
programmes - Focus on values and attitudes, and critical
thinking - two key skills identified in DE/ESD
literature for action-orientation, (IRISH AID,
1999, Tormey, 2003, Tilbury and Wortman, 2004) - Acknowledgement of this tool as a significant
undertaking preliminary stages. By February
2010 the first pilot will be completed. Report
back to Irish Aid for possible further funding
and development of this invaluable tool
5Project Timeframe
- Sept. - ESD Evaluation Consultation Day
- Oct. - Finalisation of tool format
- Nov. - Evaluation Tool Pilot
- Jan. - Data analysis
- Feb. - Finalise results and tool
6Factors influencing decision-making on
development and sustainability Issues
- Situational factors (e.g if supports and
incentives exist to make sustainable choices like
recycle banks) - knowledge and ability factors (e.g. knowing about
energy saving and how to do it - intrapersonal factors (e.g. attitudes, values,
acceptance of responsibility) - (Hines, Hungerford and Tomera,1986)
- The intrapersonal factor MUST be dealt with for
LASTING behavioural change
7What to test for?
Knowledge
Skills
Attitudes
8Introduction to Critical Thinking
- Critical thinking is reasonable, reflective
thinking that is focused on deciding what to
believe or do. - (Ennis, 1993)
- Some critical thinking skills are applicable
across contexts and others are context-specific
- DE/ESD is cross-curricular
- Some critical thinking skills are more relevant
to DE/ESD than others, e.g. - being open-minded
- being able to judge the credibility of sources
- ask clarifying questions and draw conclusions
- identify conclusions, reasons, and assumptions
- (Ennis, 1993)
- Critical thinking tests general-content based,
subject specific, multiple choice tests exist
(Illinois Critical Thinking Project), essay-style
tests (simulated situations) (The Ennis-Weir
Critical Thinking Essay Test) - Prioritising key critical thinking skills for
DE/ESD - break-out session II
9Evaluation Tool Design Basis
- July 2008 - April 2009 RCE-Ireland Project
Situated Learning and DE/ESD Activism -
questionnaire tool developed based on 3 variables
identified by Hines, Homerford and Tomera (1986),
(situational, knowledge ability, intrapersonal) - Defining Issues Test (Bebeau and Thoma, 2003)
- moral reasoning psychometric test
- Story-format or scenarios
- participants rank statements in order of
importance, in relation to decision they made in
each scenario - based on the work of Lawrence Kohlberg.
(Kohlberg, 1973). The DIT measures moral
reasoning on a scale corresponding to these
stages - Adapting scenarios model for new DE/ESD
evaluation tool to test for values and attitudes
as well as priority critical thinking skills - Develop DE/ESD scale for action-orientation
10Evaluation Tool Parameters
- Quantitative
- Comparative
- less time to use and analyse
- Addresses pre-determined questions
- Statistically analytic
- Dependent on evaluator knowledge
- Operational
- Reliable and valid
- Useable data for the end user
- Implementable in a reasonable timeframe
- Web-based tool or pencil and paper
- Self-corrected Vs. externally evaluated
- Measuring increase in levels of critical thinking
and values and attitudes (i.e., cognitive and
affective dimensions), before and after a DE/ESD
programme
- Acknowledging Tool Limits
- Action-orientation is measured indirectly
- Prerequisites for making long-term behavioural
change are, having corresponding values and
attitudes, and critical thinking skills - Final tool used in conjunction with qualitative
evaluation tools. Tool kit to accompany the final
tool
11Over to you
- Break-out Session I Appreciative Enquiry
- Outcome to identify successful factors of a
quantitative evaluation tool - share general experience of quantitative
evaluation tools small groups, to categorise and
summarise these and briefly feedback to the whole
group - Break-out Session II - Success Indicators of
DE/ESD - Outcome to identify key indicators to measure
how successful a DE/ESD programme was - If you had a quantitative tool, what would you
want it to measure? - What are your hoped-for outcomes for
participants?
12RCE-IRELAND Nancy L. Serrano, RCE Research
Assistant, Department of Education and
Professional Studies, University of
Limerick Phone 061-212927 E-mail
nancy.l.serrano_at_ul.ie