Complex knowledge production issues of impact evaluation

1
Complex knowledge production - issues of impact
evaluation

• Prof. Dr. Stefan Kuhlmann

page 1
2
Overview

• Changing knowledge production and innovation
• From mechanistic" to reflexive" research and
  innovation policymaking
• Need for strategic intelligence for research
  and innovation policy
• Dimensions of policy impact
• Principles and requirements for distributed
  intelligence

3
Changing knowledge production and innovation

• Emergence of a postmodern research system (A.
  Rip 1994) postacademic science (J. Ziman
  2001) Mode 1 problems are set and solved in a
  context governed by the largely academic
  interests of a specific community. By contrast,
  Mode 2 knowledge is carried out in a context of
  application. Mode 1 is disciplinary while Mode 2
  is transdisciplinary. Mode 1 is characterised by
  homogeneity, Mode 2 by heterogeneity.
  Organisationally, Mode 1 is hierarchical and
  tends to preserve its form, while Mode 2 is more
  heterarchical and transient (M. Gibbons et al.
  1994, 3).
• Empirical evidence of more inter- and
  trans-disciplinary research and fusion of
  heterogeneous technological trajectories (Grupp
  1992 Kodama 1995).
• Changing performance dimensions (virtual lab).
• Internationalisation of industrial research.
• Increased share of contract research in
  universities and non-university research
  institutesfierce competition for contracts.
• Soft side of innovation" of growing importance
  (den Hertog et al. 1997 Coombs 1999 Smits
  2001).

4
Institutional borders blurred (virtual lab)-
(in advanced science and technology)

• Virtual laboratory network based polycentric
  knowledge production(e.g. neural nets plant
  biotechnology nanotech also social science
  innovation research)
• with participants from heterogeneous disciplines,
  institutions, culture behaviour
• Heterogeneous performance dimensions for
  researchers
• publication and teaching dominated by discipline
• targeted basic research characterised by
  interdisciplinary communication and interaction
• applied research led by industrial and economic
  rationales

5
From "Mechanistic" to Reflexive" Research and
Innovation Policymaking (I)

• Strategy targets from excellence of individual
  researchers resp. competitiveness of single
  companies to modernisation of institutes,
  corporations, sectors, regions, and
  (national, regional, sectoral) "innovation
  systems"
• Policy means from rd subsidies to adoption
  of new technologies, to liaison/brokerage
  services, to network or cluster
  stimulation, to continuing education, to
  regulatory policies (e.g. IPR)
• Potential impacts/benefits from rd results, to
  innovative products/processes, to
  knowledge base building, to increasing
  "absorptive capacities", to awareness of
  innovation needs

6
From "Mechanistic" to Reflexive" Research and
Innovation Policymaking (II)

• Policy actors from sectoral
  policymakers/beneficiaries, to innovation
  multi-sector/multilevel policy networks,
  to related networks (horizontal and
  vertical) to transnational clustering and
  networking (EU)
• Strategic policy development (public and
  private)
• from performance measurement
  (legitimization) to socio-economic
  assessment of mid-term performance to
  reflexive policy learning with contesting
  stakeholders
• ? need for multi-perspective, reflexive
  evaluation and learning i.e. strategic
  intelligence"

7
Research and Innovation Systems and Stakeholder
Arena

• Differing interests, perspectives and values
• Competition for funds
• No dominant player?
• Contested policies
• Need for alignment, otherwise exit

Slide 7
8
The search for impacts of public interventions
...

• Evaluation ? identification of impact of (public)
  action
• ? scientific, technological, economic,
  societal, political, ...
• ? past/future, direct/indirect,
  intended/non-intended, ...
• Condition Model of input/output relation, of
  cause/effect, of actors and beneficiaries
  ...
• Impact ? a rational construction of more or
  less complexity

9
Impact dimensions of public research and
innovation spending
10
Scope and limitations of impact measurement of
public RI
11
Variety of RI evaluation methods a metrix?
12
Strategic Intelligence (SI) for Innovation
Strategy and Policy ...

• ... builds on enhanced tools like

• Policy strategy evaluation
• Technology society foresight
• Technology assessment

13
Research and Innovation Policy Evaluation

• State of the art
• from peer review (ex ante ex post) to impact
  analysis broad variety of methodologies
• some intractable problems biased peers
  attributionof impacts ...
• Practical use in policymaking
• ranging from legitimization of funding, to
  support for fund allocation decisions ...
• to support for stakeholder-oriented mediation of
  decisionmaking
• Enhanced tool for strategic intelligence ...
• e.g. combination of impact evaluation with
  foresight exercises may improve policy planning
  ...

14
Technology and Society Foresight

• State of the art
• standard methods Delphi, workshops, specific
  surveys, seminars, etc. - still improving
• paradoxical nature of TF aiming at conflicting
  goals building consensus and preserving variety
  of visions
• Practical use in policymaking
• during 1990s quite trendy in Europe several
  major national efforts partly combined with
  fund allocation mechanisms
• big companies develop in-house exercises SMEs
  need public initiatives
• Enhanced tool for strategic intelligence ...
• linking national and regional exercises on
  European scale
• linking TF with policy evaluation and TA, e.g.
  for programme design

15
Technology Assessment

• State of the art
• systematic experience since 1970s
• broad variety of methodologies, but anticipation
  control dilemma
• Practical use in policymaking
• public service TA focus on risk assessment
• private domain TA picking the winners
  approach
• agenda-building TA seeking for stakeholder
  consensus for technology development
  (Constructive TA)
• Enhanced tool for strategic intelligence ...
• e.g., TA competence (assessing potential techn.
  impacts) could inform TF exercise

16
General principles of strategic intelligence

• Principle of participation strategic
  intelligence realises the multiplicity of actors
  and stakeholders values and interests involved
  in innovation policymaking (multiple perspective
  approach).
• Principle of "objectivisation" strategic
  intelligence "injects objectivised" information
  into the policy arena, i.e. the results of
  policy/strategy evaluations, foresight exercises
  or technology assessment, and also of analyses of
  changing innovation processes, of the dynamics of
  changing research systems and changing functions
  of public policies.
• Principle of mediation and alignment strategic
  intelligence facilitates debates and "discourses"
  between contesting actors in related policy
  arenas, thus mediating and "moderating",
  supported by "objectivised" information to be
  "digested" by the struggling parties.
• Principle of decision support strategic
  intelligence requires forums for negotiation and
  the preparation of policy decisions.

17
General requirements of distributed strategic
intelligence (DI)

• Networking requirement "infrastructures" for DI
  allow for multiple vertical and horizontal links
  amongst and across existing regional, national,
  sectoral, and transnational infrastructures and
  facilities of the related innovation systems and
  policy arenas.
• Active node requirement the infrastructure
  offers brokering "nodes" for managing the
  infrastructure. 3 types (a) enabling facilities,
  e.g. a "foresight bank". (b) "directory" allowing
  direct connections between relevant actors. (c)
  "register" for free access to all public
  strategic intelligence exercises undertaken.
• Transparent access requirement clear rules for
  access to DI, spanning from public domain
  information areas to restricted services charging
  a fee.
• Public support requirement to guarantee high
  degree of independence the DI infrastructure
  needs a regular and reliable support by public
  funding sources.
• Quality assurance requirement (a) bottom-up
  institutionalisation by providers of DI, e.g.
  professional associations (like AEA, EES,
  national ES). Scientific and expert journals
  university teaching (e.g. S/T policy programs"
  at US universities). (b) accreditation for DI
  providers, based on a vivid "scene" of experts.
  (c) reliable support with repeated and "fresh"
  strategic intelligence exercises and new
  combinations of actors, levels, and methods.

18
Contactstefan.kuhlmann_at_isi.fhg.de

• Info
• www.isi.fhg.de

18