Information for Accountability

1
Information for Accountability
Quality assurance, data, and planning capacity
Keys to assessing performance of tertiary
institutions and systems

• Workshop on Tertiary Education
• Tbilisi, Georgia
• December 14, 2009
• Richard Hopper
• Senior Education Specialist
• Europe and Central Asia Region
• The World Bank

2
Launch of Quality Assurance

• QA steps in Georgia remarkable
• Reduced from 217 institutions to 39 in 2005
• A consumer protection tool to start
• Protect students and families who invest
• Protect employers who seek skilled labor
• Rationalize the value of tertiary education

3
Bold, yet not uncommon

• Important and brave step
• Similar to other countries with rapid growth in
  private sector
• Chile, South Africa, etc.
• Also responding to Bologna Process
• Moving from institutional to program
  accreditation (anticipated 2011)

4
What next?

• Avoid the mistakes of others
• Move forward from a traditional system
• Regulatory, punitive
• Transition to satisfy Bologna
• Accountability, outcomes
• Yet continue the evolution to be strategic
• Integrate quality into organizational cultures
• Generate data for wide-ranging purposes

5
Cart before the horse

• Quality assurance systems are emerging/reforming
• Institutions are adapting to rapid change
• Financing systems are linking to performance
  measures
• Institutions undertaking strategic planning
  exercises
• International benchmarking is increasing in
  importance
• ?Often occur in the absence of any systematic
  data-collection and analysis systems within each
  institution
• Causes frustration as staff struggle to satisfy
  multiple requests
• Causes frustration as consumers get inconsistent
  data
• Causes sub-optimal analysis, conclusions,
  directions

6
Demand for information

• Strategies are key to develop tertiary education
  institutions and systems in a competitive
  environment
• Comprehensive data and analysis underpin
  institutional and system planning processes
• Focus on rapid development of human capital
• requires rapid assessment of institution and
  system dynamics
• Emphasis on quality assurance, value-for-money,
  and evidence-based decision making
• demands accurate, transparent, timely,
  publicly-available data
• ?Data exist, so why not extract value from the
  data?
• Best accomplished with a systematic, bottom-up
  approach

7
Forewarned is forearmed

• Most institutions develop data and analysis
  systems late
• They respond to overwhelming requests for
  information
• They respond to the introduction of HEMIS
• Developing such systems under pressure is not fun
• Institutional systems often start with a narrow
  mandates
• Admissions, registrar, financial officer, human
  resources, academic departments, etc.
• Recording data for a narrow purpose without
  extracting value for wider use is short-sighted
• ?Why not anticipate the growing need for
  institution-wide information and encourage your
  universities now?

8
When institution-level data are not readily
available

• Unable to clearly define problems
• Unable to indentify quality factors
• Unable to calculate costs, inefficiencies
• Unable to properly strategize, plan
• Unable to react quickly
• Unable to implement MIS, respond to accreditation
• Management decisions made without evidence
• ?Ad hoc information from multiple sources
• costly, inefficient, and prevents longitudinal
  analysis
• often prevents verification, corroboration
• diffuse data holders have undue power and
  potential weakness for corrupt practices

9
When institution-level data are not comparable

• Unable to clearly define common problems
• Unable to indentify system trends and deviations
• Unable to compare costs, inefficiencies
• Unable to properly strategize, plan at system
  level
• Unable to react with sensitivity to variation
• Unable to benchmark institutional performance
• Unable to describe system characteristics to
  outsiders
• Unable to design fair, performance-based
  incentives
• ?Data variability easily misused and manipulated
• can cause confusion, misunderstanding, poor
  judgments
• generally raises more questions than answers

10
Institutional research and planning office

• Curious name
• Institutional research means research about the
  institution itself
• Formally organized institutional function
• Inward looking
• Self reflective
• ?Surprisingly widespread practice
• USA (1950s), Western Europe (1970s), South
  Africa, and now Latin America

11
What is it?

• Collects data on institutional operations
• Analyzes and interprets the data
• Transforms data into useful information
• Supports
• institutional planning, policy making, decision
  making, quality assurance, and assessment
• Makes information readily available

12
What does it do?

• External internal reporting
• official numbers, accountability reporting
• Planning special projects
• Studies on institutional effectiveness,
  benchmarking, enrolment projections
• Data management technical support
• student database, personnel database, financial
  database, data warehousing
• Surveys
• Graduate tracer studies

13
Unique data

• Finance
• Physical resources
• Human resources
• Student body
• Student activities and performance
• Academic research activity
• Data bound by local culture, needs

14
Yet comparable data

• Collects data that are comparable
• across institutions
• across countries
• US common data set of shared items
• Agreed among institutions
• Common definitions
• Common reporting

15
Linkage with financing

• Higher Education Funding Council of England
  (HEFCE) established in 1992
• To fund higher education institutions in the UK
  to ensure financing is linked to institutional
  performance
• UK Higher Education Statistics Agency (HESA)
  established in 1993
• Central source of higher education statistics
• Standardized data collection
• Information for HEFCE, QAA, and others

16
Linkage with quality assurance

• Irish universities underwent quality review by
  European Universities Association (EUA) 2005
• Irelands performance could not be adequately
  benchmarked against European competitors
• first step in benchmarking exercise was to
  develop complete and reliable data about the
  sector
• Irish universities were urged to strengthen
  capacity for institutional analysis

17
Institutional research now part of all Irish
universities
18
Other countries

• Australia
• Chile
• India
• South Africa

19
Widespread professional associations

• Association for Institutional Research (AIR)
• www.airweb.org
• European Association for Institutional Research
  (EAIR)
• www.eair.nl
• Australasian Association for Institutional
  Research (AAIR)
• www.aair.org.au
• Southeast Asian Association for Institutional
  Research
• www.seaair.au.edu
• Southern African Association for Institutional
  Research (SAAIR)

20
Formalized training

• Post-masters certificates in institutional
  research in the US
• Some have on-line option
• Penn State University
• Indiana University
• Florida State University
• Missouri State University