TCA VENet project evaluation

1
TCA VENet project evaluation

• Gerd Beidernikl
• Center for Education and Economy, Research
  Consulting
• Graz, Austria
• 1st project meeting
• 11. September 2005, Cyprus

2
the goals of this session

• To introduce myself and the ZBW
• To develop a common understanding WHY we are
  evaluating our work and HOW we are going to do it
• To present the (preliminary) evaluation concept
  and to get feedback on it
• To make sure the words and concepts used are
  familiar to everybody
• To develop a common understanding of how to
  develop criteria that could be used as criteria
  for success
• Overview on the upcoming steps

3
Introduction
4
my background

• Profession Sociology, further training in the
  field of evaluation and quality management
• Researcher in the Center for Education and
  Economy, Graz, Austria
• 5 full time employees
• Public and private clients
• My main working areas labour market research,
  research on vocational education and training,
  evaluation, statistics and surveys
• Ca. 20 evaluation projects in the last 4 years

5
project references on EU level

• Evaluation of the ongoing Leonardo project
  inclusive training for disability care workers
• Self-evaluation of the Equal project girls crack
  IT
• Evaluation of the Styrian Territorial Employment
  Pacts
• Evaluation of the article VI project GO BEST
• Project management of an ongoing Leonardo project
  online consulting tool for IT professions
• Member of a proposal development group in the 6th
  framework programme, priority 7
• Reasearch activities for the CEDEFOP reports on
  the Austrian training system, comperative country
  studies, administrating the ERO database
  (European Research Overview)

6
workflow of this presentation

• Evaluation
• What does it mean?
• What can it do for us?
• What do we need?

Concept Log Frame Elements of tender Gender
Mainstreaming
Feedback
Feedback
Defining the Log Frame Indicators Terms of
Reference
Next steps Timeline At this meeting After the
meeting
7
Evaluation
There are as many different definitions of
evaluation as there are evaluators. 8
what is evaluation

• Evaluation is the systematic investigation of
  the merit or worth of an object or process for
  the purpose of reducing uncertainty in decision
  making.
• 3 types of evaluation regarding the point in
  time
• Ex-ante or predictive evaluation ? concept
• Acompanying or formative evaluation ? processes
• Ex-post or summary evaluation ? results
• Different context, different focus
• Generating and securing knowledge
• Controlling
• Legitimizing actions
• Dialogue with stakeholders, etc..

9
why evaluation? whats the benefit?

• An evaluation offers a systematic way of
  approaching a topic ? benefits for project
  planning
• Gives neutral feedback for all project partners
• Informs about performance, efficiancy,
  effectiveness and quality of the project
• Makes the project results visible ? marketing,
  proof
• Possibility to learn in the runtime of a project
• Possibility to counteract in case of anticipated
  problems
• Secure Knowledge
• Identify best practice and transferable models

10
what this evaluation should be

• Support for the transnational cooperation
• Answering the question Do we deliver what we are
  supposed to deliver?
• Neutral reflexion of what is going on
• Suggestions for changes and improvement
• Ensuring positive project development by giving
  periodic input
• Fostering the coperation and communication

11
what an / this evaluation should not be

• A judgement of good and bad, of right or wrong
• Evaluators are no project outsiders, they are
  project partners
• Evaluators are no pitbull terriers, lurking for
  any mistake
• Evaluation ? project controlling
• No analysis of cost efficiancy
• No evaluation of the benefits for the target
  groups

12
What are your expectations?
13
Evaluation concept
14
logical framework

• One commonly used apporach logical framework
  LogFrame
• The LogFrame is the starting point of the
  evaluation
• Displays the main logic of the evaluation subject
• A skeleton of the project that has to be
  evaluated
• Represents the causal relations between different
  levels of objectives, actions and results

15
LogFrame
Concept
Indikators
sustainability
Overall objectives
Impact
Specific objectives
Outcome
Performance
Expected Products
Output
Structure
Ressources / Input
Processes / Mechanisms
Effects / results
Goals
Needs
Effectiveness
Efficiency
Relevance
Secondary effects - unexpected
Context effects
16
preconditions

• There are commonly shared, defined and agreed
  goals
• Goals have to be measureable
• There are defined ressources
• There are defined processes
• The skeleton of the log frame has to be filled
  based on information in the kick off stage of the
  project.
• Should be kept in mind in every working session
  of this meeting.
• Will be worked out by the evaluation team after
  the meeting in cooperation with the project
  management

17
call for tenders

• The call for tender asked for
• Evaluation of the agreed procedures, tasks and
  responsibilities ? formal evaluation
• Evaluation of the results and the processes to
  reach these results ? content evaluation
• Participation at every meeting
• Monitoring of the Gender Mainstreaming Process
• Periodically reporting

18
main elements of the tender
Evaluation TCA VENet
Formal Evaluation
Content Evaluation
Concept Evaluation
Process Evaluation
Structural Evaluation
Result Evaluation
Gender Mainstreaming
VENet Model European Mainsteaming Spport
System Internet Platform Final Activity
Gender Mainstreaming cross-sectional topic
19
formal evaluation - concept

• Concept analysis
• Assessment of the codified project structure and
  plan
• Assessment of management tools
• Assessment of communication tools
• Data
• project documents (e.g. proposal)
• meetings
• protocols (e.g. steering group)
• interviews

20
formal evaluation - structure

• Structural analysis
• Focus transnational cooperation als system of
  interacting organizations resp. individuals
• Assessment of flow of communication
• Assessment of the cooperation patterns
• Data
• Hard facts contribution on formal level
  (reports, website,..)
• Soft facts
• Social network survey (x2, email)
• Satisfaction survey (x2, email)
• Event questionnaire (x5, at every project
  meeting)
• Interviews
• Documents, Internet Platform (?),.

21
Social network analysis (SNA)

• SNA research method that focuses on the
  relation (communication, cooperation, exchange of
  information,) between any given set of actors.
• A SNA survey asks the partners to rate their own
  network behaviour according to different
  criterias (frequeny of interaction, channels of
  interaction, topics of interaction,).
• The answers on these questions are producing a
  picture of the overall network. All individual
  answers are displayed on the network level.
• Benefit
• You get SNA plots of the network structure
• Key fiugures on well and not well established
  elements

22
network plots
23
Gender Mainstreaming

• GM are the efforts we make to ensure that gender
  aspects and gender equality is considered in all
  parts and stages of the project (working plan,
  objectives, actions,).
• A cross sectional topic
• GM aspects are considered in every part of the
  evaluation
• and the evaluation itself
• 3 main GM approaches
• Equal opportunities approach ? neutral treatment
• Diversity approach ? awareness of the different
  needs
• Affirmative action approach ? focus on women

24
Gender Mainstreaming

• Data protocols, interviews, survey questions,
  products,

25
Content evaluation

• Two levels
• Processes
• Results
• Main questions
• Have the intended results been reached (quantity
  and quality)?
• Have the processes to reach these results been
  carried out as intended?
• Did the partners contribute as scheduled?
• Are there any positive or negative deviations? If
  so, why?
• Which deductions for the following project work
  can be made?

26
Content evaluation

• For each of the five main products
• VENet model, internet platform,..
• Data
• Documents
• Interviews
• The products themselves
• Check lists resp. progress reports for the
  project (in cooperation with the project
  management)
• Questions in the email survey

27
deliverables of the evaluation

• Short reports before each meeting
• Distributed electronicly (internet platform)
• Participation at every transnational meeting (1
  w.d.)
• Presentation of evaluation results and workshop
  on the findings (1-2,5h)
• 1 mid-term report
• 1 final report

28
Feedback on the evaluation model What is your
impression?Does the evaluation provide the
information we need?GM in our project?How do we
take the evaluation results up?
29
Defining the LogFrame
30
Defintion

• By now the LogFrame is empty
• Will be based on the transnational project plan
  with defined goals, milestones and
  responsibilities
• Next Step Filling the LogFrame with information
• Results from this meeting (e.g. working plan,)
• Documents
• In cooperation with transnational coordination
• The rough LogFrame will increase in its density
  as the project goes on

31
Definition

• The defined aims of the project are crucial for
  the evaluation!
• Important If you set up any aims, goals and
  objectives
• be SMART
• Sspecific general aims must be broken down
• Mmeasurable no untouchable aims
• Aacceptance agreement among all partners
• Rrealistic no trivial aims, no unrealistic
  visions
• Ttimely fixed and with fixed responsibilities

32
Definition

• The LogFrame is the catalogue of the evaluation.
• Beneath this we will set up the so called Terms
  of Reference
• Kind of evaluation handbook, the practice version
  of the tender
• Puts down the procedures of the evaluation
• Commonly shared evaluation principles
• Will beset up by the ZBW

33
Next Steps
34
At this meeting

• Consider the LogFrame allready when working out
  objectives (SMART), products and processes
• Evaluation grows as the project grows.
• Some short interviews on the 2nd resp. 3rd day of
  the meeting transnational coordinators
• A concluding event questionaire in the last
  session of the meeting
• Define one main contact person regarding the
  evaluation in each partner organization

35
After the meeting

• ZBW will work out the LogFrame in cooperation
  with the transnational coordination
• Set up the Terms of Reference
• Set up dates for the next evaluation steps
• Inform the partners about the evaluation

36
Timeline
37
Your responsibilities!

• To commit and take active part in the evaluation
  process
• To fill in questionnaires carefully
• To submit documents and information when asked
  for
• To return you contribution to the evaluation in
  time respect deadlines
• To evaluate the evaluation! Feedback is welcome!!

38
Contact information

• Thank you!
• Lets get it on!

Mag. Gerd Beidernikl Zentrum für Bildung und
Wirtschaft Joanneumring 5/4, 8010 Graz, Austria
(valid 1st October 2005) Mail
gerd.beidernikl_at_zbw.at Phone 43 / 316 / 72 17
44 -13 Fax 43 / 316 / 72 17 44 -21 Mobile 32
/ 650 / 46 21 061