How to evaluate a Comenius project

1
How to evaluate a Comenius project?

• Agnieszka Fijalkowska, FRSE
• Contact seminar 16-20.10.2007, Kraków, Poland

2
Plan of my presentation

• Defining evaluation
• Very short history of evaluation
• Description of a process of evaluation
• Methods of evaluation
• Making reports of evaluation research
• Standards of evaluation

3
Defining evaluation

• Evaluation is a social research, so it uses
  methodology of social sciences.
• There are many definitions of evaluation, but we
  can find some common features of them
• Evaluation is a process.
• The process of collecting and analyzing data.
• Evaluation is provided to make decisions.
• And some say that the outcomes of evaluation
  arent more important than a process of
  evaluation.

4
History of evaluation

• History of evaluation began in the 60s.
  Evaluation was a tool of verifying effectiveness
  of social services, programs.

5

• Functions of evaluation
• To decide which program is good or not
• Improvement of programmes during their
  realization
• Support of decision making
• Influence on policy of social changes
• Tool of learning

• Who is benefiting from evaluation
• Persons in charge of making decisions
• Persons who are providing programme
• Experts
• Politicians
• Everyone, who is involved in programme, who is a
  stakeholder

6
Now we say about democratic evaluation, which
means that

• Involved persons should can influence the process
  of evaluation
• Every agreements connected with process of
  evaluation should be public (especially criteria
  of evaluation)
• Outcomes of evaluation should be disseminated

7
Evaluation - a tool of learning

• What pupils can learn/experience
• Get involved in a process (learning, project)
• Make decisions
• Cooperate
• Talking about their mistakes as an opportunity to
  learn
• Thinking about problems to solve, not looking for
  guilty person
• Make changes for better future

• What teachers can learn/experience
• Let pupils make decisions
• Cooperate
• Talking about their mistakes as an opportunity to
  learn
• Thinking about problems to solve, not looking for
  guilty person
• Using open/transparent criteria/procedures
• Make changes for better future

8
Evaluation - a tool of learning

• Planning a project we have to plan how pupils
  will be involved in planning, realizing and
  assessing our project.
• The same refers to the process of evaluation.
  Involvement of all stakeholders is necessary to
  realize really reasonable process and have a
  chance of using the outcomes.

9
Kinds of evaluation

• We have to plan process of evaluation at the
  same time when we plan our project.
• There are different kinds of evaluation
• at the beginning of a project we can check if
  we formulated goals of a project properly or if
  school community really need our project
• formative we can observe progress of realizing
  project, it is going on at the same time as the
  project

10
Kinds of evaluation

• summative we can observe different results of a
  project, it is realizing at the end of a project
• after the end of a project e.g. 6 months later
  we can check if results of our project are
  permanent

11
Process of evaluation

• Goal of an evaluation (what for we do it?)
• Object of evaluation (what we research)
• Key questions (about what we are asking, what we
  want to know)
• Criteria of evaluation (what is the most
  important for us, measure of our success)
• Methods of research (how we will look for the
  answers for key questions)
• Sample (whom we will ask, what we will observe or
  which documents we will analyze)
• Organization of a process of evaluation (who and
  when will do all needed research)
• Plan of data analyzing (what kind of data we will
  have and how we will analyze them)
• Report of evaluation (for whom and how we will
  prepare report)

12
Methods of data collecting

• Survey
• Interview

• Observation
• Analysis of documents
• And many other ways of data collecting as talking
  walls, suitcase and litter etc.

13
Methods of data collecting

• Usage all of these methods should have a
  documentary evidence e.g.
• interview should be recorded,
• observation should be recorded or write down in a
  special sheet, etc.
• You can analyse only documents you have your
  memory is very unstable.

14
LLP aplication form

• How will you evaluate whether the aims of
  partnership have been met
• And the expected impact has been achieved?

15
Aim of evaluation (what for we want to
evaluate)Object of evaluation (what is going to
be evaluated)
16
Report of evaluation research organizing data
in the report

• 1. Key question
• Questionnaire
• Interview
• 2. Key question
• Observation
• Document analyzes
• 3 Key question
• Questionnaire
• Document analyzes

criteria
criteria
criteria
17
Report of evaluation research agenda

• Description of a project
• Description of evaluation (what was done and what
  for if in accordance with the plan)
• Results of research (mainly - key questions)
• Conclusions of evaluation
• Attachments (project of evaluation, research
  instruments, tables, graphs)

18
Report of evaluation research for whom is our
report

• Pupils
• Teachers
• Local community
• Head of a school
• Authorities
• Inspectorate
• National Agency
• Who else?

Maybe we will need different versions of report
for some of receivers. Presentation of evaluation
results also can be useful.
19
Standards of evaluation

• Usefulness evaluation is planned in accordance
  with needs of stakeholders
• Fusibility evaluation is well planed, realistic
  and costs worthy
• Compliance evaluation is realised in accordance
  with law, ethic and rules of social sciences
• Properness evaluation gives useful information
• Joint Committee on Standards for Educational
  Evaluation

20
Usefull websites

• mTool (Comenius network 3) www.m-tool.org
• Quas (Comenius network 3) www.qas.programkontoret.
  se
• Grundtvig 1 www.sealll.eu

21
If you asses rather yourself than others then
you are really wise.John MacBeathevaluator,
professor of Cambridge University
22
Thank you for your attention

• agnieszka.fijalkowska_at_frse.org.pl