Survey methodology and procedures: General advice from Frascati Manual

1
Survey methodology and procedures General advice
from Frascati Manual
Regional Workshop on Science, Technology and
Innovation (STI) Indicators for Gulf
countries Doha, Qatar15 to 17 October 2012
2
Sources of this presentation

• Chapter 7 of the Frascati Manual - RD Survey
  Methodology
• Section 8 of the Annex to the Frascati Manual
  (Measuring RD in developing countries) -
  Strengthening RD Statistical Systems
• Chapter 7 of UIS Technical Paper no. 5 (Measuring
  RD in developing countries) - Strengthening RD
  Statistical Systems

3
Institutionalization of STI statistics

• Political support
• Infrastructure and sustained staff
  training/capacity building
• Involvement of NSOs Official statistics status
  for RD surveys.
• Adequate legal framework

4
User-producer networks

• Recommendations
• User-producer networks and other forms of
  stakeholder consultation should be instituted.
• Establishing national ST statistics groups.
• Involve multiple actors.
• Coordinating/networking among institutions/databas
  es.
• Partnering with business associations.
• Conducting face-to-face visits by statisticians
  and project leaders.
• Exploit pre-existing personnel ties.
• Get NSO involved to deal with privacy of
  information.
• Training of interviewers/primary data producers.

5
General issues

• Statistics on RD require regular, systematic and
  harmonised special surveys
• Other sources provide information, but
• concepts of RD used often different from FM
  concepts
• concepts may change over time
• very difficult to obtain all data for the same
  period
• difficult to avoid double counting when tracking
  flows from financial statements and other sources
  
• Estimates are a necessary supplement to surveys
• especially in higher education sector

6
Scope of RD surveys

• RD surveys should identify and measure all
  financial and personnel resources devoted to all
  RD activities in all RD units
• RD surveys are mainly addressed to
  RD-performing units
• Chapter 7 of the FM only addresses
  performer-based surveys
• Statistical methodologies and other procedures
  have to be established to capture all RD,
  especially for units in the business enterprise
  sector with little RD

7
Identifying target population and survey
respondents General issues

• Exhaustive survey not possible in most countries
• Constraints include
• number of respondents may have to be restricted
  to keep costs down
• RD survey may have to be taken in conjunction
  with another survey
• surveys of some groups may require the
  participation of other agencies with different
  data needs and hence different questions for
  respondents
• One size does not fit all every country has
  different constraints advice is therefore of
  general nature

8
Establishing registers

• RD in developing countries tends to be very much
  the purview of public bodies
• Recommendations
• Establishing a database of public sector RD
  projects
• include human and financial resources align with
  national policies.
• design could reflect the RD statistical
  reporting/definitions.
• source for evaluation of such projects.
• Establishing Science and Technology Management
  Information System (STMIS)
• provide overview of research system.
• framework for establishing complete registers as
  sample frames for RD surveys.

9
Science and Technology Management Information
System (STMIS) and other secondary sources

• STMIS (e.g. database of scientists, research
  grants, CV databases, etc) frequent source for
  the production of RD statistics.
• Recommendations
• need close integration between the statistical
  system and the STMIS.
• need adjustments to produce comparable
  statistics, taking into account issues of
  definitions and coverage.
• need a balanced approach using both STMIS and
  surveys.
• need different approach to Private sector
  organizations as they are frequently not covered
  by these systems.

10
Establishing registers

• Other sources
• Associations (trade, academic).
• Learned societies.
• Registers or databases of scientists and
  engineers.
• Database of research grants.
• Databases of scientific publications.
• Patents and other IP documents.
• Business registers.

11
Survey procedure for each sector

• Each sector has different management styles,
  approaches and institutional culture.
• Consider the existing norms in relation to data
  exchange.
• First RD Survey through interviews rather than
  relying on telephonic, e-mail or postal survey.
• Higher cost and labour intensive.
• Who is the target of the survey?
• Need to consider the sector and the size and
  complexity of the organizations.

12
Government sector Identifying target population
and survey respondents

• Units to include in surveys are
• RD institutes Public research institutes
  (PRIs) Department-based research institutions
  (DBRIs)
• RD activities of general administrations of
  central or state government.
• Public institutions dealing with STS
  statistical, meteorological, geological and other
  public services, museums, hospitals.
• RD activities at the municipality level.
• Recommendation the best way to survey is to send
  questionnaires to all units known or assumed to
  perform RD.

13
Government sector cont..

• Department-based research institutions (DBRIs)
• Director-General or Permanent Secretary
• Issues non-availability of information in
  compiled form
• Public research institutes (PRIs)
• Chief Executive Officer or executive responsible
  for research management

14
Higher education sector Identifying target
population and survey respondents

• Recommendation The surveys and estimation
  procedures should cover all universities and
  corresponding institutions, especially those
  awarding degrees at the doctorate level. Other
  institutions in the sector known or assumed to
  perform RD should also be included.
• Identification generally easy.
• preferable to use smaller units, such as
  departments or institutes of the university, as
  statistical units.

15
Higher Education sector cont..

• Higher Education institutes (HEIs) are the main
  seat of RD activity
• Different degree of autonomy
• Staff employed as civil servant list of
  employees is available
• Academics directly employed by HEI staff
  details are protected
• Maturity of HEIs and historic relationships with
  Government
• Researcher CV database
• Publications databases (Web of Science or
  Scopus)
• If there is no central registry approach
  through Vice Chancellor or Dean of Faculties,
  Dean of Research or, Head of Departments.

16
Business enterprise sector Identifying target
population and survey respondents

• The enterprise is recommended as the main
  statistical unit in the business enterprise
  sector
• Some enterprises perform RD on a regular basis
  from year to year, and may have one or several
  RD units
• Other enterprises perform RD only occasionally
• It is recommended that all enterprises performing
  RD, either continuously or occasionally, should
  be included in RD surveys.

17
Business enterprise sector Survey population
first possible approach

• A census-based survey of large enterprises and a
  sample of smaller ones in order to identify RD
  performers and request the information from them
• RD performed in the past in the enterprise is
  not considered
• this is the approach followed in innovation
  surveys
• very small enterprises and enterprises in certain
  less RD-intensive industries often excluded for
  cost reasons
• when the sample size is very small, estimates may
  be less reliable, owing to raising factors
• Method not strictly followed in any country

18
Business enterprise sector Survey population
second possible approach

• Try to survey all enterprises known or assumed to
  perform RD, based on a register of
  RD-performing enterprises
• lists of enterprises receiving government grants
  and contracts for RD
• lists of enterprises reporting RD activities in
  previous RD surveys, in innovation surveys or
  other enterprise surveys
• directories of RD laboratories
• members of industrial research associations
• employers of very highly qualified personnel
• lists of enterprises claiming tax deductions for
  RD.

19
Business enterprise sector Survey population
joint approach

• Recommendation
• To include in RD surveys of the business
  enterprise sector all firms known or supposed to
  perform RD.
• To identify RD performers not known or supposed
  to perform RD by a census/sample of all other
  firms
• In the industries on the next slide.
• In principle, enterprises in all size classes
  should be included, but if a cut-off point is
  necessary, it should be at ten employees.

20
Business enterprise sector Industries to be
included
Industry ISIC Rev. 3/NACE Rev. 1
Mining 14
Manufacturing 15-37
Utilities, construction 40,41,45
Wholesale 50
Transport, storage and communication 60-64
Financial intermediation 65-67
Computer and related activities 72
RD services 73
Architectural, engineering and other technical activities 742
Plus any other industry relevant for the country
21
Business sector cont

• RD performed in business sector remains low in
  many developing and emerging economies.
• How to detect RD activity in Business?
• trade associations, or chambers of commerce.
• businesses listed on the main stock exchange.
• large firms/MNC - discussion with the Chief
  Financial Officer or Chief Technology Officer.
• missing a large firm might result in significant
  error.
• exclude holding companies, construction, retail,
  and utilities as sub-sectors likely to perform
  little or no RD.
• list of business beneficiaries of research or
  innovation grants by NRC.
• cooperation with the departments responsible for
  RD tax incentives.

22
Business enterprise sector Structural issues

• Publicly-owned businesses play a major role in
  RD in some developing countries
• Recommendations
• should consider issuing data for publicly-owned
  businesses separately from the fully private
  enterprise sector.
• private enterprises could also be disaggregated
  by ownership, in particular the various degrees
  of foreign ownership.

23
Business enterprise sector Structural
issues cont..

• Business enterprise RD is presumed to be
  generally weak in developing countries when
  compared to industrial countries.
• Recommendations
• take into account when conducting sample surveys,
  perhaps by over-sampling, especially amongst
  larger companies.
• big companies should not be missed out as it
  might imply significant error.
• invest time in interviewing key firms to
  understand their RD function and obtain a clear
  picture of their activity.

24
Private non-profit sector Identifying target
population and survey

• Private non-profit (PNP) sector make a
  significant contribution to RD in developing
  countries, but the sector tends to be very
  volatile
• Same challenges as in business difficulty in
  identify PNPs engaged in RD
• Not clear about, status ownership.
• Engaged in wide range of activities.
• Perform in-house RD as well as contract RD.

25
Private non-profit sector cont..

• The sources for identifying possible survey
  respondents are mainly the same as for the
  government sector.
• Register information may be less comprehensive
  and could be completed by information from
  researchers or research administrations.
• This sector may be more relevant for surveys on
  RD funding.

26
Who is the right respondent?

• RD Manager
• Better understanding of RD and FM norms
• But may not be able to supply exact figures

• Accountant or personnel manager
• May not refer exactly to RD as defined in FM
• But able to supply exact figures

• Cooperation of all three may be needed
• Useful to identify in advance the person
  responsible for providing information and for
  co-ordinating information from smaller sub-units

27
Working with respondents

• Questionnaire simple and short, logical and with
  clear definitions and instructions
• Optional simpler survey for smaller units
• Test questionnaires on a sample of respondents

28
Survey procedure and estimation

• Recommendations
• Attention needs to be paid to questionnaire
  design.
• Frequency of survey.
• Prioritize area of work accompanied by
  step-by-step approach.
• Use of survey questionnaires of other countries
  for inspiration need adaptations to local
  situation.
• Get expertise from the NSO, in conducting survey,
  in sampling..
• Different questionnaires might be designed for
  different sectors based on stakeholder
  consultations. One size does not fit all.
• Procedures need to be developed for estimating
  missing data.

29
Encouraging co-operation

• Secure co-operation of respondent
• Make them appreciate the potential uses of the
  data
• Respect confidential data
• Minimise the response burden
• Share the results (option customised
  information)
• Provide technical assistance and contact details

30
Estimations

• RD measurement could be done in three stages
• Identification of all specialised RD units and
  measurement of their total activity.
• Estimates of the non-RD portions of their
  activity and subtraction of these estimates from
  the total.
• Estimates of the inputs used for RD in other
  units and addition of these estimates to the
  total.

31
Operational criteria

• Tools for translating theoretical FM concepts
  into practical questionnaire
• Explanatory notes
• Hypothetical examples
• Guidance to individual respondents
• Documentation on treatment of different cases

32
Estimation procedures

• Imputation methods for item non-response
• Use previous answer
• Hot decking (use info from same survey)
• Cold decking (use info from previous survey)
• Imputation methods for unit non-response
• Use past RD data (adjusted for sales or
  employment growth)
• Impute as a function of the relation to personnel
  or sales (test with non-response analysis)

33
Thank you!

• http//www.uis.unesco.org
• r.pathirage_at_unesco.org