SHUJI YUMITORI

1
Management and Evaluation of National RD in
NEDO
2004.6.17
SHUJI YUMITORI
Director RD Project Evaluation Dept. New Energy
Industrial Technology Organization
2
Introduction of NEDO (1)

• Establishment October, 1980
• as a semi-governmental
  organization
• Mission
• ? Development of oil-alternative energy
  technology
• ? Development of energy conservation technology
• ? Development of industrial technology
• Personnel 1,030 (2004)
• Budget US 2.3 billion (2004)

3
Introduction of NEDO (2)
Incorporated Administrative Agency in October
2003
Core Concepts of New Management Policies
Results-based Management Effective and
flexible resource management Output-oriented
goals
Usability Improvement Reduce red tape and
increase efficiency Simplify contract and
audit procedures
4
Organization of NEDO
General Management Dept. (General Affairs,
Accounting, Inspection Operational Management,
Assets Management, Information and Systems )
Budget (million yen)
Policy Planning and Coordination Dept
9,406
Research and Development Project Evaluation Dept.
400
Research and Development Promotion Dept.
27,945
Energy and Environment Policy Dept.
27,854
International Projects Dept.
11,448
Electronic and Information Technology Development
Dept.
18,547
Nanotechnology and Materials Technology
Development Dept.
16,653
Biotechnology and Medical Technology Development
Dept.
17,985
5
NEDOs Project Evaluation Guidelines

• General Guidelines for the Evaluation of RD
• (approved by the Prime Minister in 1997)
• To reflect economic and social needs in RD
• To distribute resources efficiently
• To gain accountability with the public
• Others
• NEDOs Guidelines fr Technology Evaluation (2004)
• Pursuant to the General Guidelines, NEDO has the
  following rules
• Use external evaluators
• Open all of the evaluation procedure to the
  public
• Encourage debate among evaluators and RD
  participants
• Promote self-reform using evaluations

6
Project Evaluation in NEDO
Types and Secretary for Project Evaluations
Pre-project Evaluations (at RD theme
selection)
Follow-up Monitoring Evaluations (5 years after
the project)
Post-project Evaluations (after project
completion)
Interim Evaluations (during the project.)
Project
7
Basic Concept of Project Evaluations in NEDO

• Open panel discussion
• Include project participants
• External evaluators
• Four(4) evaluation viewpoints are easy to
  reflect in a project
• Adopt a rating method and qualitative evaluation

• ? Explicit
• ? Transparent
• ? Neutral
• ? Simple
• ? Effective

8
Procedure and Basic Concept
Public (citizens)
Opening of evaluation results to the public
NEDO
Reflection of evaluation results in projects,
etc.
Chairman
(13 persons)
Evaluation report
RD Project Evaluation Committee
Deliberation and approval of the evaluation
report (draft)
(4-9 persons)
Preparation of evaluation report (draft)
Explanation of the project
9
Evaluation Flow Chart (Overall)
About 2 months
Selection of evaluators(secretariat)
Explanation of evaluation to evaluators (secretari
at)
Implementation of the evaluation by the
subcommittee(interim evaluations, post-project
evaluations)
Deliberation and report to the RD Project
Evaluation Committee
Start of evaluation
Meeting before the evaluation ? Explanation
of evaluation process to the promoting
department of a project ? Confirmation of
presentation materials (promoter- secretariat)
Determination of evaluation
About 5 to 6 months
10
Flow Chart of Subcommittee Interim Evaluations
11
Necessary Documents for an Evaluation (1)
12
Necessary Documents for an Evaluation (2)
13
RD Evaluation Methodology (Structure of
Viewpoints for Evaluations)
Criteria
Checkpoints for Criteria
View points
Checkpoints for Criteria
Criteria
If the project is conducted under a specific
program or system, does it conform to the
selection criteria of the program or system?
(1) Appropriateness as a NEDO project.

1. Positioning and Necessity of Projects
Dose the project require NEDOs involvement
because of difficulty in improvement through
private activities alone, or because of high
publicity?

(2) Appropriateness of project purpose
In terms of the socioeconomic background and
research and development trends, is the project
purpose appropriate?




Has applicability as an industrial technology
been clarified?
(1) Practicability
4.Prospect of Utilization and Industrialization
14
NEDOs Evaluation View (image)
Relevance to National Policy (Purpose and
Political Positioning)
NEDO Project
Management
Socioeconomic Effect (Prospect of Utilization
and Industrialization)
Science Technology (Results)
15
RD Evaluation Methodology (NEDOs Evaluation
Viewpoints and Criteria)
1.Purpose and Political Positioning (1)
Relevancy as a NEDO (Government) Project (2)
Appropriateness of the Purpose and Political
Positioning 2.Management (1) Validity of
Target (2) Validity of Plan (3) Validity of
Project Team Formation (4) Response to Change of
Situation 3.Results (1) Achievement of
Target (2) Importance of Developed
Technology (3) Acquisition of Intellectual
Rights (4) Publicity of Results 4.Prospect
of Utilization and Industrialization (1)
Practicability (2) Spill-over Effect of
Results (3) Scenario for Industrial Use
16
RD Evaluation Methodology (Evaluation
Viewpoints, Criteria Check points)
1.Purpose and Political Positioning (1)
Relevancy as a NEDO (Government) Project a)
Does the project conform to the criteria of
governments program? b) Does the project
require NEDO's involvement? (difficult to conduct
private activities alone?) c) Are the effects
of the project adequate in comparison with the
budget? (2) Appropriateness of the Purpose and
Political Positioning a) Is the project
purpose appropriate based on socioeconomic
background and RD trends? 2.Management (1)
Validity of Target a) Have the targets been
decided based on the technological and market
trends? b) Have concrete and clear goals been
established for their attainment? c) Have
appropriate indicators to measure attainment been
established? (2) Validity of Plan a) Are the
schedule and budget appropriate for the
attainment of goals? b) Are elemental
technologies required to reach the goal? c)
Are the relation and order of elemental
technologies appropriate? d) In the case of a
continued or long-term project, how will the
obtained results and knowledge be utilized?
(3) Validity of Project Team Formation a) Is
the project formation appropriate to attain the
goals? b) Have the researchers selected
appropriately? c) Has the project leader has
been selected appropriately and the environment
ensuring successful activities been
created? d) Is the formation adequate to urge
collaboration and/or competition among the
participants? e) Based on a scenario for
practical application, is there a system for
propagating the results to receivers
(target persons for utilization and practical
application) and requesting their
involvement? (4) Response to Change of Situation
a) Will progress be constantly monitored and
properly reflected in the plan? b) Can there
be a timely and appropriate response to changes
in the socioeconomic situation and political and
technological trends? c) Is the policy
for revising the plan consistent?
17
RD Evaluation Methodology (Evaluation vie
points, Criteria Check Points )
3.Results (1) Achievement of Target a) Can the
results achieve the quantitative target? b)
To what extent have the overall goals been
attained? (2) Importance of Developed Technology
a) Can market expansion or creation will be
expected with the results? b) Are the results
a world first or at the highest level in the
world? c) Can the results be expected to
create new technological fields? d) Will the
results be widely usable? e) Are the results
obtained commensurate to the budget? (3)
Acquisition of Intellectual Rights a) Can the
intellectual property be applied appropriately
according to the projects strategies? b) Can
international application of intellectual
property be made appropriately? (4) Publicity of
Results a) Is the publication of papers
qualitatively and quantitatively sufficient?
b) Have the results been appropriately propagated
to target persons for utilization and practical
application? c) Can the information obtained
from the project be widely provided to the
public? 4.Prospect of Utilization and
Industrialization (1) Practicability a) Has
applicability as an industrial technology been
clarified? (2) Spill-over Effect of Results
a) What technological and economic effects are
expected? b) Has implementation of the project
itself produced ripple effects such as promoted
research and development in relevant field? (3)
Scenario for Industrial Use a) What are the
prospects for cost reduction, introduction and
promotion, time for commercialization,
commercialization and associated economic
effects, etc.?
18
RD Evaluation Methodology (Qualitative
Evaluation Method)
Summary opinion of subcommittee
(General remarks) 1. General opinion 2. Advice
for direction of RD
ltPositive opinionsgt ? Comment by each
evaluator lt Problems and opinions for
improvementgt ? Comment by each evaluator ltOther
opinionsgt ? Comment by each evaluator

• (Viewpoints)
• Purpose and political
• positioning
• 2. Management
• 3. Results
• 4. Prospect of utilization and industrialization

19
RD Evaluation Methodology (Rating Method)
????????
Excellent ( A 3 points ) Good ( B
2 points ) Acceptable ( C 1 point
) Not Acceptable ( D 0 points )
20
RD Evaluation Methodology (How to Summarize
Scores in Comparison with Other Projects)
? Score Results of Post-project Evaluation in
Fiscal 2003
Excellent
XY4
Good
Acceptable
XY3
21
Reflection of Evaluation Results (How to Reflect
Evaluation Results to NEDOs Budget)
Reflection 1) Interim evaluation
?Reflecting to the budget and project plan
2) Post-project evaluation ?
Reflecting to similar kinds of new projects
RD Evaluation Committee
Report to
Research and Development Project Evaluation Dept.
SEE
Chairman
Project Promoting Dept.
Evaluated Budget
Budget Request
DO
Executive Committee
Policy Planning and Coordination Dept.
PLAN
Budget Plan
22
Reflection of Evaluation Results (Reflection to
Budget)
Note The total for Fy2003 does not equal the
sum of individual actions due to overlap.
23
Reflection of Evaluation Results (Comparison
among Projects High Evaluation)
Projects Highly Evaluated
(All scores for Management, Results of
research and development, and Prospect for
practical application, etc. are above the
average.)
24
Reflection of Evaluation Results (Comparison
among Projects Severe Evaluation)
Projects Severely Evaluated
(Judged from the scores for Management,
Results of research and development, and
Prospect for practical application, etc.)
25
Follow-up Monitoring Evaluation
1. Concept

• Results of the Trial
• Implementation in 2003

26
Concept of Follow-Up Monitoring Evaluation
Avoid excessive burden on personnel
1. Simple procedure
Management
Technological strategy
Accountability
2. Clear vision for utilization
of evaluation results
Current status of projects
Patents, literature, prizes products, etc.
History of projects
3. Usefulness for constructing a database
27
Procedure of Follow-up Monitoring Evaluation
Organize necessary information for follow-up
monitoring evaluation 1)Background information
of each project
2)identification of contact person 3)Setting up
target level (Ex.ample
Research ?Development ?Practical
application?Placing on the market)
Preparation
Grasp the current situation compare to the target
1) Every year for 5 years 2) Collect data
through Web site (progress, patent, paper work,
award-winning, products, etc.) 3) Change in
a contact person
1st Step Simplified monitoring
1)Final target? 2)Delay of 2 years against preset
target? 3)Any other necessity?
3rd Step Evaluation
Implementation under the evaluation
committee. 1)Decision of subject (several
evaluations in a year) 2)Improvement of the
accountability to the public, improvement of
the management system of NEDO and reflect to
future technology strategy.
28
Trial Implementation in 2003
1) Objective To establish a follow-up
monitoring system 2) Method 9 projects
(completed in Fy2000) 90 organizations
(companies, universities, etc.)
Questionnaire by mail Follow-up notice
after 2 week wait
29
Example of the Questionnaire for Simplified
Follow-up Monitoring
30
Results of Trial Implementation in 2003 (Average
questionnaire response rate 87)
?1 Number inside ( ) indicates the
participants aiming at practical application. ?2
Number inside ( ) indicates the number of
participants. ?3 Venture company has been
established by a researcher of a project.
31
Results Obtained from Simplified Monitoring
?
?
?
?
?
?
?
?
?
On the market
?


?
?
?
?

?
?
?
?

?
Delay
Practical application
Initial Target
Discontinuance
On schedule
?
?
Development
?
?
?
?
?

Research
Ahead of schedule
?
?
On the market
Discontinuance
Research
Development
Practical application
Current Status
32
Some Ideas concerning Follow-up Evaluations
To improve accountability to the public
Example Project F (the highest ratio of
practical application) Competitiveness of
the product / process ( progressivity,
innovativeness, etc.) Ripple effects in the
economy (new market, job creation, etc.)
Create a wide range of spin-off effects ( other
product / technology, environment (LCA), energy
security, etc.)
To improve NEDOs management system
Post-project Evaluation
Follow-up Survey
Ratio of the companies accomplishing practical
application
Prospect of utilization and industrialization

33
Summary

• NEDOs project evaluation system is composed of
• a) Pre-project Evaluations
• b) Interim Evaluations
• c) Post-Project Evaluations
• d) Follow-up Monitoring Evaluations
• 2. The combination of qualitative evaluation by
  external evaluators and rating method works well.
  We should, however, be careful not to focus too
  much on only the score for a project.
• 3. Follow-up monitoring and evaluation is now
  ready to proceed. Providing incentives to
  project participants for follow-up monitoring
  and evaluation needs to be considered.