Method Issues in the CBSE survey An industrial study in Norway, Germany and Italy Seminar on CBSE co

1
Method Issues in the CBSE surveyAn industrial
study in Norway, Germany and ItalySeminar on
CBSE (component-based software engineering)Simula
Research Lab., Oslo, 4 Feb. 2005http//www.idi.n
tnu.no/grupper/su/cbse-survey

• Odd Petter N. Slyngstad
• Department of Computer and Information Science
  (IDI)
• Norwegian University of Science and Technology
  (NTNU)
• oslyngst_at_idi.ntnu.no

2
Overview

• Survey details
• The ICT-sector in Norway
• Sample sampling process
• Company contact process
• Prestudy convenience sample
• Resulting representative sample selection
• SESE tool
• Reflection
• Conclusions

3
Survey details

• First survey with a representative sample
  performed since Frakes et.al. Sixteen Questions
  About Software Reuse (1995)
• High cost in terms of effort
• Questionnaire took ½ year to finalize
• A pretest was performed in early 2004 to check
  the questionnaire (16 projects in 13 companies)
• Finally, the questionnaire was translated into
  the native language of each country before
  starting the survey
• Project is the unit of study

4
The ICT-sector in Norway

• Nace codes (OECD)

• ICT- manufacturing industry (hardware)
• 30.xx - 33.xx series
• Manufacture of computers (and other equipment),
  transmitters, receivers, wire, cable etc.
• ex Ericsson, Nera, etc.
• ICT- retail
• 51.xxx, 52.xxx series
• Wholesale of software, hardware etc.
• ex Computerland, etc.
• ICT- telecommunications
• 64.xx series telecommunications
• ex Telenor, Netcom, etc.
• ICT- software
• 72.xx series
• Consultancy, data processing, database activities
  etc.
• ex EDB, Software Innovation, Opera Software,
  etc.

Source Norwegian Census Bureau
5
The ICT-sector in Norway (2)
Source Norwegian Census Bureau
6
The ICT-sector in Norway (3)

• ICT sector (overall and software dev.)
• Second largest industry in Norway
• median 2
• mean 11
• Three groups with approximately 1/3 of the total
  employees in each
• Companies with 0-19 employees
• Companies with 20-99 employees
• Companies with over 100 employees

Source Norwegian Census Bureau
7
Sample

• Main problem a representative sample of
  companies does not mean a representative sample
  of CBSE projects
• Some business areas may be more inclined than
  others to use CBSE and COTS/OSS
• Important issues include security, quality
  certification
• Initially, we only had access to company lists -
  no knowledge of individual projects in these
  companies

8
Sample (2)

• But a representative sample is better than a
  convenience sample used in previous studies
• Morisio, Torchiano, TSE 2002, April 2002
  success/failure factors analysis of 30 EU
  projects
• Torchiano, Morisio (rejects) Common Wisdom on
  COTS 7 structured interviews

9
Sampling process

• Process in Norway
• Obtained lists of company names from the
  Norwegian Census Bureau/IKT-Norge/Infosector.net
  company
• Three groups with approximately 1/3 of the total
  employees in each
• Companies with 0-19 employees from 72.xx series
• Companies with 20-99 employees from 72.xx series
• 100 largest companies (by turnover) in the ICT
  sector 3 largest in 5 sectors (finance etc.)
• Lots of imprecise information, wrong
  classifications (no software dev. even in nace
  72.xx)
• Process in the other countries
• Italy
• Obtained 400 companies from yellow pages, checked
  if they had sw development
• Germany
• Data points from the German census bureau, took
  subset of IESE contacts. Have also included some
  major companies.

10
Company contact process

• Made guidelines for the process as follows
• Call/contact company
• get contact person information
• Call this contact person (gatekeeper)
• explain context
• send email with questionnaire if any CBSE dev.
  ok/relevant (30 of initial sample)
• Benefits for respondents include report, lottery
  and free seminar
• All respondent information entered into the SESE
  tool

11
Company contact process (2)

• Rephone
• is it really relevant, willing to participate?
  (40 of relevant)
• if yes, send login info for web questionnaire and
  explain respondent options
• Electronic questionnaire (web - preferred)
• Phone interview
• Word questionnaire (by email or printed version
  by fax)
• then wait for response, rephone etc. to remind
  many times

12
Prestudy convenience sample
initial selection
relevant for
willing to participate
50
i.e. 37,5 of total
75
13
Resulting representative sample selection
initial (three groups) selection
relevant for
willing to participate
30
i.e. 12 of total
40

• Of the companies which were willing to
  participate, 10 had several
• projects

14
SESE webtool at Simula

• Pros
• All researchers have access to all data
  simultaneously
• Ease of use in filling in the questionnaire
• Issues
• Requires a small learning curve for researchers
• Minor technical problems experienced, reporting
  back to Simula Research Lab.

15
Reflection

• High effort cost, but yet low response rate (12
  in Norway)
• In Germany and Italy many questionnaires were
  filled out in phone interviews
• this approach was discussed and rejected in March
  2004, due to the presumed effort required
• Motivation environment differences
• Italy lost summer student in October 2004
• IESE (Germany) had 3 months delay
• in PhD student upstart
• Norway hired extra research assistant for 4
  months
• (March June 2004)

16
Conclusions

• Lessons learned
• Perform survey by phone interview if less than
  100 respondents
• Keep active contact with contact persons
  (gatekeepers)/respondents agree on deadline
  for next contact/completing questionnaire,
  preferably by phone
• Allow all researchers involved to share data
  by electronic means (online DB or similar)

17
Questions Comments?

• oslyngst_at_idi.ntnu.no