Evaluation and Audit of European Programmes

1
Evaluation and Audit of European Programmes
Philippe Le Guen (leguen_at_racine.fr)

• Ankara October 19, 2004

2
PART ONE

• Evaluation
• A man of genius makes no mistakes. His errors
  are all premeditated and are the portals of
  discovery

3
Why do we need to evaluate?

• Evaluation involves investigating how effectively
  the European programmes are working and how well
  they are progressing towards meeting targets.
• Each Member State routinely carries out
  evaluation of all its programmes and policies.
  The European Commission requires that all member
  states evaluate how well the programmes or funds
  are working in their country.
• Member States analyse the information on the
  Project Applications Forms and the Project
  Closure Forms, which are usually collected
  quarterly. They also commission independent
  research organisations to carry out research
  projects on their behalf and evaluate the
  programmes.

4
Example of methodology of programme evaluation

• It is generally based around a case study
  approach, supported by interviews with programme
  beneficiaries and key programme stakeholders.
• The evaluation also comprises multiple visits to
  projects or beneficiaries (featuring interviews
  with project managers, partners and target
  groups) and on-going contact with stakeholders,
  and policy representatives.
• According to the nature of the programme and its
  specific objectives, the evaluation can be
  performed according to stages corresponding to
  the different project phases
• Project Commencement
• Project Fieldwork implementation
• Final Fieldwork/Project Completion.
• This allows project development to be tracked
  from initial approval to the final outcomes.

5
Evaluating programme and process

• The application phase, and especially
• The overall awareness of the programme i.e
  promotional activities and tools available for
  knowledge of the call for bids and tender
  specifications.
• Involvement of policy stakeholders at the
  programme development and application phases to
  increase mainstreaming potential.
• Transparency and impartiality in the bid
  assessment process and criteria used for
  selection.
• The development phase
• Preparatory works, refining plans internal
  consultations and discussions
• Implementation of the work plan
• Administration and financial works
• The implementation phase
• Delays in implementation, progress against work
  plans
• Change or respect of planned activities
• Evidence of substantive achievement or output
• Impact expectations, strategies for mainstreaming
  the results

6
Evaluating a European project

• The performance of the project should be
  evaluated as soon as possible after the project
  has finished. This will make sure that it is
  still fresh in the minds of those who were
  involved.
• The starting point for the evaluation should be
  the original application form, and ask the
  following questions
• Did the project meet its overall aims and
  objectives?
• Did the project meet the aims and objectives of
  the organisation?
• Did you recruit the right number of
  beneficiaries?
• Did you recruit the target beneficiaries
  according to the profile in the original
  application?
• Were the outcomes of the project as you expected?
  
• Did the beneficiaries meet their targets?
• Did the project keep within its budget?
• Did the project provide value for money?
• Did all of the partners and subcontractors
  deliver to specification?
• What lessons have you learned and which can help
  you to shape future provision?
• Did you learn any administrative lessons and how
  can you run projects better in future?
• The results of the evaluation should be
  circulated to appropriate personnel and senior
  management team. The main benefit of evaluation
  will be to make sure that the organisation can
  improve future delivery and pass on lessons
  learned for the next projects. So, it is
  essential that evaluation results are not filed
  and forgotten.

7
Evaluating project and programme management

• The evaluation of a European programme will also
  include
• Monitoring requirements and systems available
• Strategies for self-evaluation of projects and
  evaluation reports
• On-going support and guidance received from
  programme managing authorities or technical
  support structures
• Financial procedures, payments and control/audit
  policies

8
Self-Evaluating the effectiveness of a European
project for disadvantaged groups
Did it work?
Did we do what we planned?
Disadvantaged people situation
Is the situation of the disadvantaged improved?
PROJ. PURPOSE
Purpose achieved?
Process/product perspective
Have we improved our offering?
Has it enabled better products and services?
Internal process perspective
Have our activities been changed or added to?
Has it enabled better processes?
PROJ. RESULTS
Capabilities perspective
Have we changed or added to our tangible and
intangible capital?
Effectiveness evaluation provides information and
evidence concerning "have better capabilities
really helped to improve the processes and have
the improved processes really helped deliver
better products/services and thus more satisfied
clients?"
9
PART TWO

• Audit Control procedures

10
A reinforced priority on control

• When addressing financial rules and procedures it
  is important to remember that there are two sorts
  of rules
• EU-wide rules that are the same for every Member
  State
• National differences also in financial matters
  (e.g. national co-financing)
• Controls take place at all levels European,
  National and Regional/local

11
The Audit Trail

• The audit trail refers, in euro-jargon to a chain
  of procedures for managing the European
  programmes and funds.
• The audit trail is required by the EU regulations
  for a number of simple reasons
• The European Commission is responsible for the EU
  budget before the European Parliament,
• The Commission must ensure that the budget has
  been spent in a correct manner,
• But the Commission is not in the Member States
  and therefore must rely on them,
• Thus the Commission asks Member States to follow
  a strict procedure.
• The objective of the audit trail is the
  reconciliation of the summary accounts certified
  to the Commission with the individual expenditure
  records and supporting documents held at the
  various administrative levels and by final
  beneficiaries including, where the latter are not
  the final recipients of funding, the bodies or
  firms carrying out operations and verification of
  the allocation and the transfers of the available
  Community and national Funds.
• In other words, the audit trail is carried out to
  identify what has been done with every single
  euro of European money. This is what is called
  the traceability of the assistance. The whole
  chain must be clear ands strictly adhered to. All
  those who manage European Funds are asked to
  guarantee and maintain this chain of procedures.

12

• The audit trail can be listed here in order of
  chronology of implementation. Several
  institutions work towards the different phases,
  but some are performed exclusively by one
  institution.
• Programming
• Selection of actions
• Contracting with final recipients
• Monitoring the activities
• Payment
• Verification the validity of expenditure claims /
  certification
• Control

13

• Audit trail must be clear and transparent (
  reconciliation of accountancy possible)
• enrolment of participants is clear (who is
  benefiting of European money and who do not)
• activities and expenses well identified, eligible
  expenses understood and isolated (to control
  them)
• Selection procedure clear (how the project was
  selected)
• contract signed including the forecasted amount
  and main physical indicators (how many
  participants, gender, status)
• Clear path of invoices (difference between
  forecasted or provisional expenses and REAL
  expenses)
• monitoring system installed (financial and
  physical) capacity to check the stage of
  development of the actions and programme at all
  levels

14
Who may inspect your record?

• The Auditors could be one or several of the
  following
• European Commission, Programme Division,
  Verification and Audit Team
• Government Office Inspection Teams
• External Auditors
• Depending on the type of your organisation you
  could also be visited by
• National Audit Office
• The European Court of Auditors
• European Anti Fraud Office

15
What is the purpose of the audit visit?

• To verify that
• Expenditure has been incurred in a proper
  manner
• Financial management has been sound
• The project actually ran and was delivered in
  line with the approved programme application and
  project closure report
• Interim / project closure report entries are
  supported by evidence of activity and expenditure
• The project represented value for money

16

• The Verification and Audit Team from the European
  Commission will carry out audit inspection visits
  to a representative sample of organisations based
  on Commission Regulation (EC) No 438/2001 Chapter
  1V 'Sample checks on operations' and checks will
  cover at least 5 of total eligible expenditure
  in each year.
• Delivery of all projects is verified by comparing
  actual reports with profiles and checks of claims
  submitted (Article 4). The Verification and Audit
  Team co-ordinate the inspection process and may
  delegate Government Office Auditors.
  Verifications are made on the spot on a sample
  basis.
• A system based approach is followed by sample
  checks of costs, matched funding and
  beneficiary/invoice records, down to source
  documentation.
• The review of any previous years for the project
  will be performed to check if similar projects
  were run by the organisation in the past.

17
What do you need to do before the audit?

• Identify key staff and ensure they are available
  to answer questions
• who compiled the claim/final report
• where the source documentation is held
• who managed the project
• who was in charge of the finance
• Where records are computerised ensure an operator
  will be on hand
• Any item may be subject to review therefore the
  organisation needs to
• Ensure all invoices are available and clearly
  labelled
• Be prepared to explain and demonstrate with
  source documentation how staffing costs are
  attributed to the project
• If organisation / firm costs methodology has been
  used provide information used in the construction
  of the claim
• Give explanation and provide source documentation
  of apportionment and allocation of indirect and
  overhead costs

18
What documents will be inspected?

• The following documentation should be available
  for each project dossier
• Original programme application form
• Letter of approval / contract
• Advance and interim claim forms
• External evaluator's report (if any)
• Project Closure Report forms
• General Statements of Expenditure
• Sub-contract Arrangements
• Project Records
• Project timetable and programme of activities
• Application / eligibility assessment forms
• List of beneficiaries on the project (if any)
  and individual attendance records
• Project register and attendance records
• Project start and finish dates
• In and end year monitoring
• Records of achievement
• Follow-up information

19

• Financial Records
• All items on the interim claim and project
  closure report must be supported by source
  documentation
• working papers which show how the claim was
  compiled
• staff costs - detailed salary records
• beneficiary costs
• other costs
• match funding (if any)
• Invoices and payment receipts
• Accounts and bank statements
• Organisation / firm based unit cost calculation
• Apportionment methodology

20
What level of detail is required?

• The level of detail should be sufficient to
  demonstrate the
• Actuality of the expenditure
• Propriety of the expenditure
• Eligibility of the expenditure
• Links to approved activities
• The eligibility of expenditure will be assessed
  against the
• Date when the expenditure was incurred which must
  be within the dates of the approved application
• Purpose of the expenditure, which must be in
  accordance with the approved application
• Type and level of expenditure
• Auditors will also require
• Overview of applicant organisation and
  partnership arrangements
• Comprehensive project specification
• Market needs analysis and market research
• The decision making process for determining the
  need for the project
• Applicant's monitoring systems
• Project evaluation and follow-up

21

• What action may be taken as a result of an
  inspection visit?
• The organisation may be asked to recalculate and
  resubmit project closure report
• Project closure report may be reduced if items
  claimed cannot be substantiated
• Internal system changes may be requested in line
  with audit recommendations
• No action

22

• What are the main areas of concern from previous
  audits?
• Significant changes not reported to the
  administrating authority i.e. project not meeting
  its original objectives as outlined on approved
  application form
• Added value not transparent
• Actual detail not recorded on claim, i.e.
  includes notional or estimated items
• Insufficient detail to explain or substantiate
  entries, e.g. vague headings, unjustifiably high
  or low costs
• Unjustified central overhead costs
• Inconsistencies in the methodologies used in
  calculations
• Use of weighting
• Lack of management checking on claims, e.g.
  eligibility, arithmetic
• Full costs of project not included
• Revenue not declared or included in calculations
• More EU money received than actually paid out
• Nil or inadequate publicity
• Inappropriate use of consultants

23
(No Transcript)