EasyEco Training : the Plan Loire Case study

1
Easy-Eco Training the Plan Loire Case study

• Szentendre 25/01 03/02 2006

2
Day 1

• Analysing the call for evaluation

3
Geographical elements
4
Physical characteristics

• Loire river is more than 1 000 km long
• As a whole, it represents an hydrographic
  network of 135 000 km, with very different
  hydrologic patterns
• Two old mountain areas massifs at the extremities
  (in Brittany and massif central)
• Very few ground water under these old mountains,
  some important stocks, but with high levels of use

5
Administrative features

• France is characterised by the coexistence of
  many administrative levels.
• At the national level, the important stakeholder,
  in charge of the plan Loire, is the Ministry of
  ecology and sustainable development
• At the local level, three categories of
  institutions are of importance
• Regional councils 10 out of 22 are integrated in
  the Loire basin
• General councils 31 departements are partially
  or totally concerned (100 in France)
• Municipalities 7 283 in the Loire basin (36 000
  in France)
• 11,5 millions inhabitants (French population is
  about 60 millions)

6
French administrative structure
7
Economic and environmental Characteristics

• A territory dominated by rural areas (average
  density is 75 p/km²)
• More than 2 700 municipalities with less than 400
  inhabitants
• 50 of national cereal productions, mainly in
  the two regions Centre and Poitou Charentes
• Industrial activity is linked to this
  agricultural feature, with a large agro-food
  production in the west and centre of the basin
• Tourism is also of importance

8
History

• Loire is considered as one of the last savage
  rivers in Europe
• A series of huge floods during the 19th century
• A programme of construction for 19 spillways,
  reinforcement of dykes and creation of dams has
  been partially achieved before 1900
• After a long period of inaction, a new programme
  for the construction of 4 dams has been set in
  1986, considered as necessary after the death of
  8 people in 1980 due to the sudden rising of
  waters

9
History (part II)

• This programme supported by the Etablissement
  Public Loire was considered as the unique
  solution to recurrent flooding
• But this vision was disputed by environmental
  associations leading to major conflicts at the
  local level
• As a response to this deadlock, the government
  decided the creation of the Plan Loire Grandeur
  Nature in 1994

10
The Plan Loire

• Initial aims
• To guaranty the safety of populations against
  flood risks
• To satisfy water needs
• To restore the ecologic diversity of the aquatic
  environment
• A ten years period but a renewal in 1999 to take
  into account institutional evolutions and to add
  a new theme heritage
• A sustainable development approach (safety,
  environment, economic development)

11
Detailed description of objectives

• Aim 1 Safety
• Identification of areas liable to flooding
• Control of space planning of these areas
• Reinforcement of alert systems and water rising
  announcement
• Planning measures for specific areas
• Aim 2 Water resources
• Water supply of Allier and Cher valleys
• Raising of the waterline
• Aim 3 Ecologic diversity of the environment
• Improving the movement of migrating fish
• Restoration of natural environments
• Reconquest of the Loire estuary
• Protecting natural spaces and landscapes

12
Means for implementation

• 3 major partners
• Government
• Etablissement public Loire and local authorities
• Water basin agency
• Financial means
• At least 600 millions euros were dedicated to the
  programme for studies, works, communication,
• The largest part was given to safety works
• Furthermore, it was not possible to identify the
  exact number of people working for the plan
  Loire, because of the numerous structures involved

13
Questions

• Which aspects of SD do you recognize in the
  tender of the case study ?
• Which aspects are relevant for the evaluation ?
• Which aspects are beyond the scope of an
  evaluation and can not be taken into account in
  the bid?

14
Answer to question 1

• Which aspects of SD do you recognize in the
  tender of the case study ?
• The Plan Loire is a SD programme but it is not
  totally clear in the tender
• The three pillars of SD are taken into account by
  the four themes of the plan Loire
• Safety can be considered as being part of the
  social pillar
• Water resource management and restoration of
  natural environments are clearly representing the
  environmental pillar
• The heritage development is the economic
  development pillar

15
Answer to question 2

• Which aspects are relevant for the evaluation ?
• All aspects are relevant as
• 5 evaluation questions are dedicated to the
  safety theme, 5 questions to water resource
  management and restoration of natural
  environments and 3 questions to heritage
  development
• but
• This a sector-based approach
• The economic pillar is not very well developed
• And the governance aspects are not clearly
  considered

16
Answer to question 3

• Which aspects are beyond the scope of an
  evaluation and can not be taken into account in
  the bid?
• Long term impact assessment and for instance the
  capacity of the programme to reduce flood risk
  (rising of waters necessary to estimate the
  impact)
• More generally environmental impact assessment is
  beyond the scope
• Transversal, integrated approach outcomes and
  impacts

17
Day 2

• Developing quality checklists for case analysis

18
Content of the afternoon session

• How should we consider the Plan Loire evaluation
  according to standards ?
• What kind of quality criteria should be used for
  the evaluation ?
• How this list of criteria fits into the
  evaluation process ?

19
What need for evaluation on the market ?
SOCIETY
20
Specificity of activities in the public sector
Public Authorities
SOCIETY
21
Specificity of evaluation
Resources
Conformity to texts
Standards of good management
General Interest, point of vueof stakeholders
22
Evaluation estimates the impact
23
Evaluation and policy cycle
Needs Stakes Problems
?
Impacts
Strategic Objectives
Outcomes
Specific Objectives
Outputs
Operational Objectives
Resources
24
Main evaluation criteria
Public Action
Resources
Objectives
Outputs
Evaluation
25
Evaluation standards

• 5 main dimensions
• What is the object of the evaluation ?
• What is its purpose ?
• What kind of effects are searched for ?
• What is the reference level ?
• What is the distance between evaluation and the
  evaluandum ?

26
Answer

• Object of the evaluation
• A programme and its organization
• Purpose
• Accountability and improvement
• Effects
• Main objectives
• Reference level
• Outcomes and impacts
• Distance
• Participative

27
Quality criteria

• Three groups
• Selection of 10 to 12 criteria in the list
• Discussion about the choices of the different
  groups and final selection

28
Relevance of the criteria within the evaluation
process
29
Day 3

• Stakeholder analysis and involvement strategy

30
Objectives of the afternoon

• First, to finish the selection of quality
  criteria
• Second, few words about stakeholders involvement
• Third, to organise a role game
• Description of the roles
• Preparation of positions
• Discussion of interests
• Fourth, to discuss the role game

31
Stakeholders involvement

• Who ?
• Why ?
• How ? (information, consultation, participation)

32
Detailing the evaluation organization
Public Authorities
Evaluation committee
Operators
Evaluation team
Recipients
33
The role of the committee
Directing evaluator towards useful information
sources
Steering the evaluation
Producing judgement (recommendations)
Examining and validating reports
34
Benefits of an evaluation committee

• Better acceptation by people to be evaluated by
  creating relations of confidence
• Easier Access to information and better
  understanding of facts and events
• Interpretations et recommendations better taking
  into account all legitimate points of view
• Faster diffusion of conclusions and use of
  recommendations

35
Composition of the evaluation committee
Actors in charge of the implementation
DonorsDecision-makers Elected representatives
ManagersOperatorsAgencies
Evaluation Committee
Researchers Qualified People
Direct and indirect recipients(winners and
losers)
Experts
Concerned Public
Steering Group
36
List of stakeholders
37
Sociogram
Decision-makers
Operators
)

Recipients
38
Sociogram
39
Preparation of positions

• What should be the aims of the evaluation ?
• Who should be in the steering committee ?
• Which kind of consultation of stakeholders should
  be carried out during the evaluation ?

40
Description of the roles
41
Observation

• Timing and nature of conflict how the conflict
  is managed do we reach a consensus?
• Quality of arguments
• Attitude of stakeholders (dominant, passive, )

42
Day 4

• Developing an evaluation design

43
Content of the afternoon session

• You formulate questions regarding the different
  phases of the Plan Loire evaluation
• Stakeholder involvement
• Tendering process
• Data collection
• Analysis
• Judgement
• Recommendations and communication
• I tell you which ones have been discussed
• I try to answer

44
The six steps of an evaluation process
Evaluation Mandate
Decisions
45
Formulation of the evaluation mandate

• Defining the field of evaluation
• Defining the main questions
• Precising the evaluation organisation

46
Structuration of the terms of reference
1 Regulatory framework 2 - Action to be
evaluated
3 Who will use the evaluation and why ?
4 - Evaluative Questions
5 - Method (guidelines) 6 - Sources of information
7 - Organisation of the evaluation
8 Indicative time schedule
9 Indicative Budget
10 Skills and competencies
11 - Structure of the offer 12 Selection
criteria
47
Criteria to assess the quality of the evaluation
9. Clearness of reports
48
Day 5

• Making an offer

49
Todays programme

• Improving the training
• Logical framework
• Work packages for the offer
• Comparison with actual bid

50
Changes that were proposed

• How to take into account the different levels of
  experience within the group?
• More games !
• How do you want to influence the organisation of
  the case study ?
• Complexity of the case study what remains
  unclear ?

51
Plan Loire 2000-2006
Specific Impacts
Global Impacts
Actions
Outcomes
Works of the multidisciplinary team
Reinforcement of alert and rescue means
Better forecast of flood risks
Setting up of tools to inform elected
representatives and citizens
Protection from floods is increased
Making of SIEL
Reinforcement of dykes and restoration of the bed
Safety of people is warranted
Better prevention of risk
Support to vulnerability reduction actions,
localized protection actions and preventive
information carried out by local authorities
Setting up a network of local leaders
Better management of water and uses
Elaboration of SAGE
Coordinated management of low water levels
Water resources and quality are improved
Quality of water is increased
Measures favouring water savings
A Territory conciliating safety of population,
preservation and valorization of environment and
economic development
Alert network for accidental pollution
Measures against pollution (eutrophication, black
points)
Aquatic milieu is restored
Raising the water line
Natural aquatic environments and rural spaces of
the valleys are restored
Restoration and preservation of hydraulic
annexes, and sensible and remarkable natural
spaces
Fishes can migrate
Measures to reclaim mobility spaces of the river
Estuary is restored
Implementation of salmon breeding facility in
haut Allier
Implementation of controlling stations
Natural, landscape and cultural heritage of
valleys is valorised
Modelling study of estuary operation
Landscape is valorised
World heritage classifiication and restoration
works
New forms of tourism are developed
Environmental and socio-economic studies
Loire cycling scheme
Loire identity and culture is reinforced
Actions to raise awareness and develop Loire
Houses
Inventories and valorisation Loire heritage
52
Making the offer

• What are the steps ?
• What should be the work packages ?
• Aims
• Method
• Outputs
• Resources
• What should be the budget ?

53
The actual offer
54
Articulating global and focused approaches
55
The different steps

• Step 1. Assessment of ouputs
• Step 2. Analysis of outcomes and impacts of Plan
  Loire
• Step 3. Analysis of implementation processes of
  Plan Loire
• Step 4. Production of judgement

56
Tools categories

• 5 tools for collecting information
• Analysis of documents
• Quantitative Analysis
• Typology
• Participatives Observations
• Interviews
• 3 tools for analysis
• Case studies
• Focus groups
• Scoring
• 3 tools for judging
• Benchmarking
• Panel of expert / focus group
• SWOT

57
Step 1. Assessment of outputs

• Collect and analysis of documents
• Consolidation of output matrix
• Writing a synthetic assessment
• 40 working days

58
Step 2. Analysis of outcomes and impacts of Plan
Loire

• Analysis of documents
• Individual and collective interviews
• Case studies
• Scoring
• 55 working days

59
Step 3. Analysis of implementation processes of
Plan Loire

• Analysis of documents
• Interviews with partners
• Interregional meetings
• Specific focus groups
• 30 working days

60
Step 4. Production of judgement

• Scoring
• Benchmarking
• SWOT
• Focus groups
• 20 working days

61
Programming the evaluation
62
Day 6

• Handling of quantitative and qualitative data

63
Menu of the morning session

• How to answer evaluation questions ? (2 groups)
• Comparison of methods developped by the two
  groups
• What has been done actually ?

64
The final set of evaluation questions Theme
Safety

• To which extent did the Plan Loire allow for
  raising stakeholders awareness and culture
  regarding flood risk ?
• How and to which extent preventive actions have
  been implemented by the Plan Loire and to which
  extent vulnerability of potentially affected
  areas has been reduced ?
• Were the technical, financial and human resources
  used in the Plan Loire enough to implement the
  actions regarding the dykes and the bed of the
  Loire river ?
• What can be said about the involvement of local
  authorities in the management of flood risk in
  the framework of the Plan Loire ?

65
Themes 2 and 3

• To which extent did the actions carried out in
  the Plan Loire improve the availability of the
  water resources both in quantitative and
  qualitative terms ?
• To which extent did the Plan Loire durably
  improve the conditions of circulation and
  reproduction of migrating fish, and at which
  costs ?
• In the Loire estuary, to which extent did the
  Plan Loire allow to improve the situation of the
  aquatic environment, while authorising the
  development of the Nantes-Saint Nazaire ports
  activity ?
• How did the wetlands and aquatic natural spaces
  linked to the Loire bed benefit from measures of
  the Plan Loire ?

66
Theme 4

• How did the achieved actions related to the
  heritage theme contribute to the emergence or
  reinforcement of a basin solidarity ?
• To which extent did the Plan Loire contribute to
  valorize the landscape and river related heritage
  ?

67
Transversal questions

• To which extent are the knowledge produced in the
  framework of Plan Loire, and its results,
  satisfying according to objectives and use for
  operational actions?
• Did the decision related structures and processes
  that have been put in place allow for the
  elaboration of a strategy and objectives shared
  by the different partners?
• To which extent was the organisational structure
  adapted for an effective implementation of
  actions ?

68
Framework structure
69
Data collection process for Q2

• See table

70
Tools to observe and collect information

• Statistical tools
• Scoring
• Individual interview
• Focus group
• Survey
• Case study

71
Synthesis of collection analysis process
72
Day 7

• Analysing data and judging

73
Afternoon

• Methods of data analysis
• Presentation of results for the evaluation
  question 2
• Development of conclusions
• Comparison

74
2 methodological options
Inductive Process
Hypothetico-deductive process
Formulation of verifiable hypothesis
Test of hypothesis
Exploration
Description
Observationof events
Identification of recurrences
Causal Explanation
Observation
Discovery of correlations
Measure of the amplitude of effects
75
Inductive or hypothetico-deductive approach?
Option to choose
Hypothetico-déductive
Inductive
Features of the context
Combination
Nber of "decision-makers"
A dominating commissionner
Several influent stakeholders
1
Measure of outcomes
Understanding of processes
Objectives of the evaluation
2
Aim of the evaluation
3
Learning
Accountability
Many existing studies
Knowledge base
Few studies in the field
4
Attitude of stakeholders
5
Conflict
Consensus
Type of intervention
6
Routine like
Very innovative
76
Tools for inductive analysis
77
Tools for deductive analysis
Delphi Group
78
Vulnerability reduction (1.01)
79
Implementation of Risk Prevention Plans (1.02)
80
Real time measuring networks (1.04)
81
Water rise and floods forecasting (1.05)
82
Localised protections (1.10)
83
Judgement for question 2

• Two case studies
• Vulnerability reduction actions for firms (Region
  Centre and Brive Charensac)
• Vulnerability reduction in Blois (la Bouillie)
• Main judgements
• Prevention actions have been achieved in two of
  the most critical areas
• But these are pilot, experimental actions which
  have not been diffused
• Not all actions have been carried out
• Degree of vulnerability did not change

84
Day 8

• Recommendations and communication

85
Menu

• The main conclusions of the Plan Loire evaluation
• What are your recommendations ?
• How to communicate the results ?

86
Conclusion 1 Plan Loire I, a challenge in a
period of conflicts

• Opponents became partners
• Conflicts have been largely overcome
• Even if some of them remain hidden

87
Conclusion 2 Plan Loire II, complex mechanics

• The second phase has been supported by the
  contracts between the state and region
• This increased the complexity both in terms of
  organisation, decision and financial support

88
Conclusion 3 undeniable achievements

• Actions planned in the first plan have been all
  carried out, and a large part of actions of the
  2nd plan are undertaken
• Assessment has been difficult, given the
  complexity of the plan Loire
• The plan Loire was an innovative approach by the
  diversity of actions and its global character
  sustainable development

89
Conclusion 4 outcomes and impacts which are
still difficult to estimate

• Outcomes and impacts to be considered on the long
  run (several decades), so difficult to identify a
  few years after
• However, some of them are on the right track but
  need to be consolidated
• Irreversible effects have not been reached
• The sector-based approach limits the capacity of
  actors to anticipate, understand and measure
  synergies between actions

90
Conclusion 5 a lab for producing knowledge,
methods, know-how

• A better knowledge of flood risk but also of
  hydrosystems
• While conflict in the early stage was based on a
  partial and unshared information, now discussion
  is based on shared and reliable information in
  many fields
• This knowledge base also displayed the complexity
  of systems and the necessity to integrate the
  long term in decision-making processes

91
Conclusion 6 No global, explicit nor shared
strategy

• There is no real strategy but just official
  documents with general aims
• Decision processes which have been set could not
  allow for designing such a strategy
• Not having such a strategy was normal in 1994
  (conflicts, insufficient knowledge) it is not
  any more
• Where are the priorities in the Plan Loire ? What
  is the coherence of actions ?

92
Conclusion 7 difficulties to implement actions
because of decision-making and organisational
processes

• No clear responsibilities, incomplete
  decision-making structures, no readibility
• No single and partner-based structure for
  decision-making process leading to be dependent
  on external factors (government decisions,
  individual conflicts, competition between
  organisations)
• A very difficult coordination due to financial
  processes
• A plan which runs thanks to individuals
• No actor with a global vision, no organised
  monitoring system but slight changes since 2003
  by the reinforcement of operational structures

93
Conclusion 8 insufficient communication

• A first communication strategy which has not been
  implemented
• The sum of individual strategies does not make a
  coherent communication and is even leading to
  blur the image of the plan Loire
• While some are communicating on less results

94
Elaboration of recommendations and communication

• 3 groups ministry / EP Loire / NGO
• What would be your recommendations on the basis
  of the 8 conclusions ?
• What kind of communication would you have ?
• Role game !!!