Nessun titolo diapositiva

1
MANAGEMENT DELLA RICERCA Processi e Protocolli
di Valutazione
INDICE

• Valutazione, oggetto, criteri, metodologie
• Il Civr e la sua attività
• Valutazione della ricerca diffusa, anagrafe,
  indici di produttività, il CIV,
• un esempio di valutazione dei ricercatori (SPAC)
• Valutazione ai fini di selezione di progetti
  PRA, PRIN
• Valutazione ai fini di selezione di strutture
  (CRS)
• Monitoraggio diversificato per progetti e
  strutture
• Valutazione ex-post di progetti e strutture

2
MANAGEMENT DELLA RICERCA La valutazione EX-POST

• Valutazione ex-post dei progetti si basa su
• report annuale del Monitore
• report annuale con rendiconti del Project Manager
• acquisizione dei report da parte del Consiglio
  Scientifico
• Valutazione ex-post delle strutture si basa su
• report annuale del Direttore
• site visit annuale del Review Panel
• report annuale del Review panel

3
Programma per la Revisione dei Centri di Ricerca
e Sviluppo INFM

• Ad ogni centro annualmente viene richiesta la
  seguente documentazione
• report annuale del Consiglio Scientifico del
  Centro
• report annuale richiesto dallINFM
• --gt al Review Panel di esperti internazionali,
  indipendenti, nominato annualmente, e chiamato
  ogni anno a fornire una dettagliata valutazione
  dellattivita del Centro nel suo complesso

4
Programma per la Revisione dei Centri di Ricerca
e Sviluppo INFM

• Il Panel e costituito da 5 esperti, dei quali,
  dove necessario, uno potrebbe provenire da
  unindustria laddove il centro abbia una
  connotazione rivolta alla produzione di device e
  comunque forti legami con le industrie.

5
Programma per la Revisione dei Centri di Ricerca
e Sviluppo INFM

• Tutti i membri del Panel devono essere
  opportunamente informati sulla natura del
  programma CRS dellIstituto, e sulla sua
  importanza strategica, nonché sensibilizzati
  sulla rilevanza degli aspetti formativi e di
  trasferimento tecnologico, come anche
  dellorganizzazione manageriale dei Centri.
• valuta qualita scientifica, della formazione,
  del trasferimento tecnologico e del management

6
Tempistica 1/3

• Tempistica esempio
• fine ottobre 2002 richiesta del report da parte
  dellINFM
• entro novembre nomina del Review Panel per ogni
  Centro
• entro meta dicembre invio dei report da parte
  dei Centri
• meta dicembre invio della documentazione ai
  Panel
• primi di gennaio site visit del Panel presso il
  centro

7
Tempistica 2/3

• In generale la tempistica segue queste linee
• luglio richiesta del Report da parte dellINFM
  con deadline meta/fine settembre
• richiesta di una scaletta per la site visit di
  novembre (interventi, spazi di discussione etc)
• luglio nomina Review Panel
• settembre invio dei report ai Panel

8
Tempistica 3/3

• ottobre consulta elettronica dei Panel per
  stabilire se sia necessario richiedere ai Centri
  ulteriori informazioni o chiarimenti alla luce
  dei contenuti del Report
• ottobre eventuale richiesta di ulteriori
  chiarimenti al Centro da includere nelle
  presentazioni da preparare per la site visit
• novembre site visit
• In questo modo il processo di revisione si
  conclude prima dellinizio della selezione di
  altri grandi progetti dellIstituto.

9
Site visit

• Giorno 1 arrivo in serata dei membri del panel
  in luogo e primo breve briefing del panel con
  nomina di un Chair allinterno del panel stesso e
  approvazione di eventuali modifiche sulla
  scaletta ufficiale della visita come proposta dal
  Centro.
• Giorno 2 presentazioni da parte del direttore e
  dei suoi collaboratori, discussioni tra il Centro
  e il panel e private del Panel
• Giorno 3 incontro con gli studenti del Centro,
  stesura e firma del report da parte del Panel
• Il report viene consegnato immediatamente o il
  giorno successivo al Direttore del Centro.

10
Reporting Req. I. Cover Page

• 1.1 Reporting year
• 1.2 Name of the Centre
• 1.3 Name of the Centre Director
• 1.4 Centre URL
• 1.5 Contact information of the Centre Director if
  changed during the reporting period
• 1.6 (Excecutive) Summary (max 2 pages)
• (Provide a brief description of the
  accomplishements and the performance of the
  Centre during the reporting period. Describe any
  significant changes from the originals plans as
  far as goals and vision of the Centre. List any
  new senior or other researchers.)

11
Reporting Req. II. Research1/2

• 2.1 Activites
• (Provide a brief description of the centre
  activities, and list names and status of
  participating staff Faculty, Student,
  Technician, Postdoc, etc-. Provide information on
  the goals and the outcomes and/or impacts in the
  reporting period. Discuss the plans for the next
  reporting period.)
• 2.1.1 Describe any relevant changes in the
  research objectives during the reporting period.
• 2.1.2 Discuss any problems encountered in
  achieving the Centres goals during the reporting
  period or any problems anticipated in the next
  reporting period.

12
Reporting Req. II. Research1/2

• 2.2 Seed Activities
• (Provide a brief description of the seed
  activities, and list names and status of
  participating staff Faculty, Student,
  Technician, Postdoc, etc-. Provide information on
  the goals and the outcomes and/or impacts in the
  reporting period. Discuss the plans for the next
  reporting period.)

13
Reporting Req. II. Research2/2

• 2.2.1 Discuss any problems encountered in
  achieving the goals of the planned seed
  activities during the reporting period or
  anticipated in the next reporting period.
• 2.3 Experimental Facilities
• (Provide a brief description of progress made
  towards the development of new experimental
  facilities, their use within the Centre community
  and within the national and international
  community. Discuss any problems encountered in
  the reporting period, and provide information on
  the plans for the next reporting period.)
• Total max 10 pages

14
Reporting Req. III. Education, Human Resources
Development and Outreach

• 3.1 Describe the Centres overall educational
  activities in the reporting period, and how they
  fit within the Centres goals.
• 3.1.1 Describe any relevant changes in the
  educational objectives during the reporting
  period.
• 3.1.2 Discuss any problems encountered in
  achieving the Centres educational goals in the
  reporting period or any problems anticipated in
  the next reporting period.

15
Reporting Req. III. Education, Human Resources
Development and Outreach

• 3.2 Describe how the Centre integrated research
  and education during the reporting period, and
  discuss how the students at the Centre
  participated in professional development
  activities.
• 3.3 Describe any media or other material produced
  by the Centre to disseminate information relating
  to its activities.
• 3.4 List undergraduate students and PhD students
  who graduated during the reporting period, and
  specify the placement, as well as the years taken
  to complete the degree. List postdoctoral
  associates woh left the centre during the
  reporting period, with placements.
• Total max 3 pages

16
Reporting Req. IV. Collaboration with Other
Sectors

• 4.1 Collaboration with Other Sectors
• (Describe and discuss the activities that were
  conducted within collaboration with other sectors
  or institutions, and, where appropriate, with
  industries, during the reporting period. Describe
  the role of the staff involved, and the impact of
  the collaborations. Discuss the plans for the
  next reporting period.)
• 4.1.1 Describe any relevant changes in the
  collaborations during the reporting period.
• 4.1.2 Discuss any problems encountered in
  achieving the Centres goals within
  collaborations carried out during the reporting
  period or any problems anticipated in the next
  reporting period.
• Total max 2 pages

17
Reporting Req. V. Management

• 5.1 Management
• (Describe and discuss the management and
  communications systems used to develop a fully
  integrated Centre, as well as any problems
  encountered in achieving this integration.)
• 5.1.1 Describe any relevant changes in the
  Centres organizational strategy and /or its
  underlying rationale, during the reporting
  period.
• 5.1.2 Discuss any problems (technical, personnel,
  communication, etc) encountered in realizing the
  Centres organization and management objectives
  in the reporting period, and describe any
  problems anticipated in the next reporting
  period. Include the strategies for addressing any
  problems.

18
Reporting Req. VI. Centre Outputs and Issues

• 6.1 Role of the Centre at the national and
  international level
• (Briefly describe the role of the Centre within
  the INFM research network, within the European
  Research Area, and in the international context
  -if applicable, also within the regional
  development framework- during the reporting
  period. Discuss the international and scientific
  or engineering benefits to the research and
  education program during the reporting period.
  Include a description of any internship program
  for INFM and non-INFM scientists during the
  reporting period.)
• 6.2 List all Centre publications in the reporting
  period using a standard citation format,
  distinguishing between peer reviewed articles,
  conference proceedings, others.

19
Reporting Req. VI. Centre Outputs and Issues

• 6.3 List any awards and other honors with names
  of awardees in the reporting period.
• 6.4 List the general outputs of knowledge
  transfer activities during the reporting period,
  including patents and spinoff companies.
• 6.5 Observations
• (Describe other outputs, impacts or
  influences related to the Centres progress and
  achievements during the reporting period that may
  not have been discussed elsewhere in this
  report.)
• Total max 10 pages

20
Reporting Req. VII. Budget

• 7.1 Summary Table of total (INFM and co-funding)
  Support for the reporting period.
• 7.2 Summary Table of INFM support

Reporting Req.VIII.Attachments

• Biographical Information of new Centres senior
  investigators

21
Purpose of Research Assessment in the UK

• To inform funding
• To provide accountability
• To Stimulate improvement

22
Universities are funded separately to do teaching
and research

• 5 billion Euros per year for teaching
• funded equally, based on student numbers
• 1.5 billion euros per year for research
• funded very unequally, based on research quality
• this represent about one third of the total which
  universities receive for research

23
There is a clear rationale for this very uneven
funding for research

• To maintain top quality research
• To enable the UK to retain its position in the
  world
• The price is explicitly understood
• hierarchy of esteem
• the possibility of development is limited
• But as a strategy it has been spectacularly
  successful

24
Objective indicators of research quality

• No of papers per and no of citations per
• UK ranks first in world
• UK share of most cited 1 of papers up 63 in 5
  years since 1995 - from 11 to 18
• Extent to which UK citation rate outperforms rest
  of world up nearly 50 since 1996

25
UK research impact relative to rest of world
26
Results borne out by other indicators
Papers per M
Citations per M
Papers per researcher
27
Selectivity consequences

• Very differentiated funding
• Much more funding for some institutions than
  others
• Difference of esteem
• Etc

28
RAE 2001 - Funding consequences

• 75 of funds to 24 HEIs (25 in 2001)

29
Relationship between Research Grant and SSRs
30
Funding for Research Method

• Distribute money between 69 subjects (subject
  quanta)
• Relative cost and volume
• Allocate subject quanta between universities
• Quality
• Volume
• Quality is determined through the Research
  Assessment Exercise

31
The Research Assessment Exercise (RAE)

• Peer Review
• 69 Subjects
• All subjects assessed on a common 7 point scale
• Results used directly in funding

32
RAE Process

• 69 Panels Appointed - nominated by their peers
• Panels consider and publish their draft criteria
  - and they consult on them with the community
• Invite submissions from universities active in
  their subject
• The most important criterion the quality of the
  4 best publications for each member of the
  Department

33
RAE Process

• Universities select staff to submit in each
  subject
• Submission from each university contains similar
  info
• quantitative (research grants, PhD students, etc)
• qualitative (the best 4 publications for each
  staff member)
• textual (any further info about the department,
  research strategy, background etc)
• Panel considers the submission in context of
  previously announced criteria
• Grade awarded according to a common scale with
  common definitions
• Panels exercise expert judgement

34
Results of the RAE

• RAE rating
• 5
• 5
• 4
• 3a
• 3b
• 2
• 1

• Value for Funding
• 4.05
• 3.375
• 2.25
• 1.5
• 1
• 0
• 0

35
Example of Application of RAE in Chemistry(NB
10M to allocate, only 2 universities do
chemistry)

• University A
• Grade 3a in RAE
• 100 Academic Staff
• Index of 150(1001.5)
• Therefore University A receives 150/250 of
  10M6M
• University B receives 100/250 of 10M4M

• University B
• Grade 3b in RAE
• 100 Academic Staff
• Index of 100(1001.0)

36
Effects of RAE (positive)

• Terrific improvement in research in UK
• Research is a closely and well managed activity
• The RAE has enabled us to allocate limited funds
  highly selectively, on the basis of reasonably
  objective criteria
• The results are broadly credible and largely
  accepted as reasonable

37
Effects of RAE (negative)

• Stress
• Distortions
• Possible effects upon nature of research
  conducted
• interdisciplinary research/ emerging
  sub-disciplines
• long-term research
• Indicator chasing
• Staffing decisions
• transfer market
• young staff
• equalities
• Cost (estimated at 37m in 1996- including
  opportunity costs)
• Damage to non-research activities
• Comparability of judgements

38
Future of the RAE

• We continue to need a basis for differentiation
• Victim of success
• bunching
• increasingly fine judgements
• Whenever we have looked before the alternatives
  looked less attractive
• But we need now to look fundamentally at whether
  the process remains fit for purpose

39
RAE - Some possible alternatives

• Amending current exercise to meet specific
  criticisms
• A more quantitatively based exercise eg
• Formula based upon research grant income (perhaps
  complemented by earmarked funding for
  researcher-driven research)
• Some use of bibliometric analysis and citation
  rates
• Two-track exercise light touch for existing
  centres of excellence
• Reputational assessment opinion poll of UK
  foreign competitors

40
RAE Principles

• Transparency
• Assess what you see
• Peer review
• Ownership by the sector
• Credibility
• Similar standards