A MONITORING AND EVALUATION FRAMEWORK TO BENCHMARK THE PERFORMANCE OF WOMEN IN SET Presentation at Women in ICT Workshop 31 January 2006

1
A MONITORING AND EVALUATION FRAMEWORK TO
BENCHMARK THE PERFORMANCE OF WOMEN IN
SETPresentation at Women in ICT Workshop31
January 2006
2
About the ME framework

• Brief to develop a monitoring and evaluation
  framework for women in SET that should support
  planning and resourcing of the National System of
  Innovation
• Designed to provide a comprehensive national
  profile of women in SET in South Africa, that
  will tell us
• how many women are potentially available to
  participate in the NSI
• how women are distributed horizontally and
  vertically within the NSI
• how women are supported to participate in the
  NSI,
• what recognition women get as scientists
• and what womens contributions are to scientific
  output.

3
South African Monitoring and Evaluation
Framework Constructs
1. SET Potential 6. Scientific Recognition
2. SET Labour Force 7. Scientific Agenda Setting
3. RD Workforce 8. Scientific Output
4. Fairness and Success in Funding 9. Scientific Collaboration and Networking
5. Rank and Status
4
South African Monitoring and Evaluation
Framework Descriptions

• SET Potential
• Leakages in the pipeline
• Distribution across study fields
• Size and potential of SET and RD pool
• SET Labour Force
• SET human resource capacity
• Horizontal distribution
• Absorption of graduates
• RD Workforce
• RD human resource capacity
• Horizontal distribution
• Absorption of graduates

5
South African Monitoring and Evaluation
Framework Descriptions

• Fairness and Success in Funding
• Access to Funding
• Rank and Status
• Vertical distribution
• Scientific Recognition
• Recognition by peers
• Scientific Agenda Setting
• Representation on scientific boards and councils
• Scientific Output
• Authorships and citation ratings
• Scientific Collaboration and Networking
• Co-authorships, collaborative projects and
  conference attendance

6
Where does ICT fit in?

• Broad field of study that can be compared to
  participation in Natural Sciences and
  Engineering, Health Sciences and Social Sciences
  and Humanities
• SET occupational field that can be compared
  across sectors and occupational levels
• Sectors include Higher Education,
  Government/Science Council, Business/Industry and
  Not-for-profit
• Participation indicators include gender, race,
  age, nationality and qualification level

7
What study, research and occupational fields are
included in ICT?
RD Survey HEMIS Institute for Science Information (ISI)
Information systems Code systems Computer science, artificial intelligence
Hardware Communication technology Computer science, cybernetics
Software Cybernetics Computer science, hardware architecture
Current information technology Innovative communication Computer science, information systems
Communication Applications in Computer Sc. Data Processing Computer science, interdisciplinary applications
Security system Computer Ops. and Operations Control Computer science, software engineering
Computer Hardware Systems Computer science, theory methods
Computer Hardware
Information and Data Base Systems
Numerical Computations
Programming Languages
Programming Systems
Software Methodology
Theory of Computation
Computer Engineering and Technology
8
Monitoring-for-policy questions (1)

• SET Potential
• How do the gender and race profiles of students
  compare at each level of study?
• Are there differences between men and women
  students in drop-out level? If so, are these
  differences related to qualification level?
• Are women students starting and completing
  postgraduate studies at a later age than men?
• Are women students overly clustered in broad
  fields of study and under-represented in others?
• Are there certain fields of study that attract
  more foreign students than others?

9
Monitoring-for-policy questions (2)

• SET Labour Force
• What proportion of the total labour force is made
  up of SET workers?
• What proportion of SET workers are female?
• Are SET graduates moving into SET occupations?
• Are women SET workers overly represented in
  certain occupations and under-represented in
  others?
• 3. RD Workforce
• What proportion of the total labour force is made
  up of RD workers?
• What proportion of RD workers are female?
• Are female researchers overly represented in
  certain sectors and under-represented in others?
• Are certain sectors attracting more foreign RD
  workers than others?

10
Monitoring-for-policy questions (3)

• Fairness and success in funding
• Are there gender and race differences in applying
  for funding?
• Are there gender and race differences in the
  awarding of funds?
• Are there gender and race differences in the
  monetary value of funds awarded?
• Do foreign researchers have differential access
  to certain funding sources?
• 5. Rank and employment
• Are there gender and race differences between the
  lower ranks and higher ranks in Higher Education?
• How is gender and race distributed across
  different scientific fields and between the
  lower ranks and higher ranks?
• Are there gender and race differences in the
  appointment of permanent researchers across
  sectors?
• Are there gender and race differences in the
  promotion patterns of researchers across sectors?

11
Monitoring-for-policy questions (4)

• Scientific Agenda Setting
• What is the representation of women on scientific
  boards and councils?
• What proportion of executive and senior managers
  across sectors are women?
• 7. Scientific Recognition
• What proportion of reviewers for national and
  international funding agencies are South African
  women?
• What proportion of reviewers for scientific
  journals are South African women?
• What is the representation of women scientists in
  national academies?
• Are there differences in citation ratings for
  South African researchers by gender and by field?

12
Monitoring-for-policy questions (5)

• Scientific Output
• What is the contribution of women scientists to
  scientific output in the system?
• Are there differential patterns of scientific
  production by field and gender?
• 9. Scientific Collaboration and Networking
• Are there gender and race differences in the
  undertaking of collaborative research projects?
• What is the proportion of female co-authored
  articles?
• What proportion of papers presented at
  international conferences is by female
  researchers?
• What proportion of academics taking overseas
  sabbaticals is female?

13
The Monitoring and Evaluation Framework

• Constructs
• Indicator categories and sub-categories
• Data tables
• Indicators

14
Example of an indicator category with its
indicator subcategories

• 14. Share of female students enrolled for a
  doctoral degree or equivalent
• 14.1. Students enrolled for a doctoral
  degree or equivalent, by gender and by race
• 14.1.1. Students enrolled for a
  doctoral degree or equivalent in Social Sciences
  and Humanities, by gender and by
  race
• 14.1.2. Students enrolled for a
  doctoral degree or equivalent in ICT, by gender
  and by race
• 14.1.3. Students enrolled for a
  doctoral degree or equivalent in Natural Sciences
  and Engineering, by gender and by
  race
• 14.2. Students enrolled for a doctoral
  degree or equivalent, by gender and by
  nationality
• 14.2.1. Students enrolled for a
  doctoral degree or equivalent in Social Sciences
  and Humanities, by gender and by
  nationality
• 14.2.2. Students enrolled for a
  doctoral degree or equivalent in ICT, by gender
  and by nationality
• 14.2.3. Students enrolled for a
  doctoral degree or equivalent in Natural Sciences
  and Engineering, by gender and by
  nationality
• 14.3. Students enrolled for a doctoral
  degree or equivalent, by gender and by science
  field
• 14.4. Mean age of women and men enrolled
  for a doctoral degree or equivalent
• 14.4.1. Mean age of students
  enrolled for a doctoral degree or equivalent, by
  gender and by race
• 14.4.1.1. Mean age of
  students enrolled for a doctoral degree or
  equivalent in Social
  Sciences and Humanities, by gender and by race
• 14.4.1.2. Mean age of
  students enrolled for a doctoral degree or
  equivalent in ICT, by gender and by race
• 14.4.1.3. Mean age of
  students enrolled for a doctoral degree or
  equivalent in Natural
  Sciences and Engineering, by gender and by race
  
• 14.4.2. Mean age of students
  enrolled for a doctoral degree or equivalent, by
  gender and by science field
• Green Indicator category Red
  Indicator subcategory 1
• Blue Indicator subcategory 2 Brown
  Indicator subcategory 3

15
Example of a data table
Gender x Race Gender x Race Social Sciences Humanities ICT Natural Sciences Engineering Total
Women African
Women Coloured
Women Indian
Women White
Women Total wt
Men African
Men Coloured
Men Indian
Men White
Men Total mt
Total women men Total women men gt
16
How to derive the indicators (1)

• Indicator category14. Share of female students
  enrolled for a doctoral degree or equivalent
• Women as of students enrolled for a doctoral
  degree or equivalent

Gender x Race Gender x Race Social Sciences Humanities ICT Natural Sciences Engineering Total
Women African
Women Coloured
Women Indian
Women White
Women Total wt
Men African
Men Coloured
Men Indian
Men White
Men Total mt
Total women men Total women men gt
17
How to derive the indicators (2)
Set 1
Indicator subcategory 114.1. Students enrolled
for a doctoral degree or equivalent, by gender
and by race
Gender x Race Gender x Race Social Sciences Humanities ICT Natural Sciences Engineering Total
Women African
Women Coloured
Women Indian
Women White
Women Total wt
Men African
Men Coloured
Men Indian
Men White
Men Total mt
Total women men Total women men gt
Set 1 African women as of students enrolled
African men as of students enrolled
Coloured women as of students enrolled
Coloured men as of students enrolled Indian
women as of students enrolled Indian men as
of students enrolled White women as of
students enrolled White men as of students
enrolled for a doctoral degree or equivalent
18
How to derive the indicators (3)
Set 2
Indicator subcategory 114.1. Students enrolled
for a doctoral degree or equivalent, by gender
and by race
Gender x Race Gender x Race Social Sciences Humanities ICT Natural Sciences Engineering Total
Women African
Women Coloured
Women Indian
Women White
Women Total wt
Men African
Men Coloured
Men Indian
Men White
Men Total mt
Total women men Total women men gt
Set 2 African women as of African students
enrolled Coloured women as of Coloured
students enrolled Indian women as of Indian
students enrolled White women as of White
students enrolled for a doctoral degree or
equivalent
19
How to derive the indicators (4)
Set 3
Indicator subcategory 114.1. Students enrolled
for a doctoral degree or equivalent, by gender
and by race
Gender x Race Gender x Race Social Sciences Humanities ICT Natural Sciences Engineering Total
Women African
Women Coloured
Women Indian
Women White
Women Total wt
Men African
Men Coloured
Men Indian
Men White
Men Total mt
Total women men Total women men gt
Set 3 African women as of women enrolled
Coloured women as of women enrolled Indian
women as of women enrolled White women as
of women enrolled African men as of men
enrolled Coloured men as of men enrolled
Indian men as of men enrolled White men as
of men enrolled for a doctoral degree or
equivalent
20
How to derive the indicators (5)
Set 1
Indicator subcategory 214.1.3. Students enrolled
for a doctoral degree or equivalent in ICT, by
gender and by race
Gender x Race Gender x Race Social Sciences Humanities ICT Natural Sciences Engineering Total
Women African
Women Coloured
Women Indian
Women White
Women Total wt
Men African
Men Coloured
Men Indian
Men White
Men Total mt
Total women men Total women men gt
Set 1 African women as of students enrolled
African men as of students enrolled
Coloured women as of students enrolled
Coloured men as of students enrolled Indian
women as of students enrolled Indian men as
of students enrolled White women as of
students enrolled White men as of students
enrolled for a doctoral degree or equivalent
in ICT
21
Application of the Framework

• Three factors to consider
• Purpose of monitoring and evaluation
• Data availability
• Audience
• They influence
• Selection of indicators
• Frequency of data collection
• Form of reporting

22
Application of the FrameworkData availability

• A. Routinely collected data that are readily
  accessible
• The data are either available in the public
  domain or can easily be obtained from the data
  collection agency in the desired format.
• B. Routinely collected data that are not readily
  accessible
• Special requests and negotiations are required to
  solve issues of data ownership and/or to arrange
  for data permutations as the available data are
  not in the desired format.
• C. Data not routinely collected
• Procedures for collecting this data can be
  introduced requiring different degrees of
  effort/investment of time and money.

23
Application of the FrameworkApplication
Scenarios (1)

• Scenario A System monitoring scenario
• Annual reporting on the system
• Routinely collected data
• Selected indicator categories
• Scenario B Sector monitoring scenario
• 3 sectors HE Gov/SETI Business/industry
• Inform sector level policies and interventions
• Three-year cycle, with one sector report per year
• Invest time and resources to collect data
• Adapt indicators for sectors

24
Application of the FrameworkApplication
Scenarios (2)

• Scenario C International benchmarking scenario
• International comparisons
• Every three years
• Selected indicator categories
• Scenario D System review scenario
• Comprehensive review of the system
• Every six years
• Inform all stakeholders of all aspects of the NSI
• Include all constructs and main indicator
  categories

25
Summary
YEAR 1 YEAR 2 YEAR 3 YEAR 4 YEAR 5 YEAR 6
Scenario A System monitoring System monitoring System monitoring System monitoring System monitoring System monitoring
Scenario B HE Sector monitoring Govt Sector monitoring Industry Sector monitoring HE Sector monitoring Govt Sector monitoring Industry Sector monitoring
Scenario C International benchmarking International benchmarking
Scenario D System review
26
Conclusion

• The ME framework is a dynamic measuring
  instrument that expands or contracts in terms of
  constructs, indicator categories and indicators,
  depending on the purpose it is to serve.
• Although the four application scenarios are
  complimentary activities, the implementation of
  these scenarios would have to be based on careful
  consideration of time and resources (financial
  and human).

27
THE END