Evaluating SME Programs in Mexico Using Panel Firm Data

1
Evaluating SME Programs in Mexico Using Panel
Firm Data

• Hong Tan and Gladys Lopez-Acevedo
• The World Bank
• SME Evaluation Workshop
• Mexico City, September 23-24, 2004

2
Goals of Presentation

• Describe two impact evaluation exercises
• Re-evaluation of CIMO / PAC program of STPS
• Using 2001 ENESTYC to compare program impacts
• Highlight evaluation approaches and the issues
  they try and address
• Report some tentative findings
• Suggest ways of improving program evaluations

3
I. Re-evaluation of CIMO/PAC

• Overview of CIMO/PAC program
• 1995 1997 Impact Evaluation Studies
• Re-evaluating the impacts of CIMO/PAC
• Positive intermediate outcomes, negative
  productivity impacts
• Selectivity bias from weak firms participating in
  CIMO/PAC, non-comparable control group
• Using difference-in-differences approach
• Results and suggestions

4
Overview of CIMO/PAC

• Subsidizes training provision and other support
  services to MSMEs by public or private providers
• Promoters do diagnostic to identify production
  and skills constraints training technical
  assistance offered on a cost-sharing basis
• In 2001, CIMO/PAC provided support to 94 thousand
  firms (3 of Mexican firms), benefiting 333,500
  workers

5
Previous Evaluations

• STPS 1995 and 1997
• Quasi-experimental design
• treatment group from CIMO/PAC program and a
  matched control group that did not participate,
  but otherwise similar in firm size, sector, and
  geographic location
• Surveys applied by CIMO/PAC to treatment group,
  and by INEGI to the control group

6
Data Collection Strategy1995 1997 CIMO Studies
7
Summary of Previous Studies

• In comparison to control group, both studies
  found positive impacts on intermediate outcomes
• treatment group more likely to provide training,
  higher training spending per worker, introduced
  organizational changes, and implemented quality
  control systems
• But negative impacts on productivity levels in
  the treatment group as compared to the control
  group
• Both studies found lower productivity levels in
  the treatment group than in the control group
• Production functions estimated on post-program
  data do not take this into account

8
Re-evaluation of CIMO/PAC

• Challenge resolving apparent contradiction
  between positive intermediate outcomes but
  negative final program impacts on firm
  performance
• Re-examination of CIMO panel data
• cleaning and making data comparable over time
• addressing selectivity bias - CIMO attracting
  weaker firms into program than other SMEs
• using a difference-in-differences approach that
  fully uses the panel data

9
A First Look at the Data

• Program improved intermediate outcomes of
  participating firms relative to the control group

Note All estimated effects are statistically
significant.
10
Group Means in Labor Productivity
In 1994 prices
11
Productivity Growth CIMO and Treatment Groups

Value-added per worker
Value-added per worker
20,000
CIMO firms
Control Group
8000
5000
3000
Second study 1993-1995
First study 1991-1993
12
Selectivity Bias

• CIMO and Non-CIMO firms not directly comparable
• Control groups have higher productivity levels
  than CIMO firms with similar observable
  attributes
• Due to selection of weak firms into CIMO, or poor
  choice of control group, or both.
• A productivity regression will generally yield a
  negative coefficient (impact) on a CIMO indicator
  variable
• The solution estimate a fixed effects or
  first difference model, remove level
  differences between the two groups, study changes
  over time in outcomes

13
Addressing Selectivity Bias

• Estimating CIMO effects ? in a levels and
  first differenced production function
• Levels Model
• Log(VAt) ?Log(Kt) ?Log(Lt) ?CIMO
• VAvalue-added, Kcapital, Llabor, tyear
• First Differenced Model
• ?Log(VAt) ? ? Log(Kt) ? ? Log(Lt) ?CIMO
• ? Xt Xt-1

14
Levels versus DifferencesImpacts on productivity
productivity growth
Denotes significance at the 5 level.
15
Summary of Results and Conclusions

• CIMO/PAC has positive effects on intermediate
  outcomes training, organization change, QC
• Negative impacts on productivity attributable to
  selectivity bias and choice of control group
• Positive impact on productivity growth in the
  1991-1993 period, but not in 1993-1995.
• LESSONS The critical importance of
• Selecting an appropriate control group
• Addressing selectivity bias in program
  participation

16
Suggestions

• CIMO/PAC quasi-experimental design a good model
  to use to evaluate impacts of specific SME
  programs
• Time-line of 2 years to collect pre-
  post-program data on treatment and control
  groups, 6 months to 1 year for analysis plan
  and budget accordingly
• Design surveys to collect information specific to
  programs and common outcome or performance
  indicator variables for comparability
• Reports to include details on data collection and
  analytic methods for transparency

17
II. Evaluations Using ENESTYC

• Objectives
• Investigate potential of ENESTYC for impact
  evaluations and comparisons of different SME
  programs
• Testing different impact evaluation approaches
• Overview of 2001 ENESTYC and SME module, and
  links of 1995 and 1999 ENESTYC
• Some tentative results on program impacts and
  implications for evaluation studies
• Suggestions for improving usefulness of future
  ENESTYC surveys

18
2001 ENESTYC Survey

• Fielded by INEGI with over 8,000 firms
• Firm-level information on ownership, employment,
  location, workforce attributes, wages,
  production, technology, workplace practices, and
  training

19
2001 ENESTYC Survey

• Fielded for STPS by INEGI with over 8,000 firms
• Firm-level information on ownership, employment,
  location, workforce attributes, wages,
  production, technology, workplace practices, and
  training
• SME module retrospective questions on
• 10 major SME programs
• CONOCER, CIMO, COMPITE, CRECE, FIDECAP, FAMPYME,
  MEX-EX, PATCI, PMT, PCI, PAIDEC (on average 400
  firms in the largest programs)
• familiarity, participation, date started, form of
  participation

20
Links with 1995 and 1999 ENESTYC

• Linking to create panel data allows
• Identification of pre- and post-program periods
• Selection of control group from large pool of
  non-participants
• Estimation of impacts on performance over time
• Control for unobserved heterogeneity and
  selectivity bias

21
Data and SME Programs Studied

• Limitations of ENESTYC Random sampling produces
  small samples of program beneficiaries when
  linked to earlier ENESTYC surveys
• Focus on the three largest programs CIMO,
  COMPITE and CRECE with the largest sample sizes
  of program beneficiaries
• CIPI administrative data base were used to
  augment self-reported participation information
  from the 2001 ENESTYC to increase sample sizes

22
Program Participants
Source Estimates from ENESTYC databases and CIPI
administrative records
23
Methodology

• Begin with two simple approaches
• mean values of key outcome measures of the
  treatment and control groups
• production functions to measure impacts on
  productivity, controlling for firm-specific
  effects
• Naïve approaches subject to several limitations,
  but they provide useful initial insights into the
  program impacts

24
Test for Differences in Means Treatment versus
Control Groups
Bold significant at the 10 level
25
Production Function Estimates in Levels and First
Differences
Bold significant at the 5 level
26
Methodology

• Propensity Score Matching addresses inappropriate
  choice of control group
• Differencing addresses potential selection bias
  associated with program participation

27
Propensity Score Matching

• Duration of the pre- and post-participation
  period varies across cohorts from 3 to 6 years
  (outcomes may only appear with a time lag)

28
Propensity Score Matching

• Match each of the cohort treatment groups with a
  control group using one summary indicator
• Logit model to predict program participation
• The indicator is the predicted probability or
  propensity score the that a firm would
  participate
• Variables included in model
• economic sector, state, firm size, age of the
  firm, share of permanent workers, share of
  unskilled labor, and fixed assets per worker

29
Propensity Score Matching

• The matching was based on pre-program
  participation characteristics for each cohort
• Matching algorithm was the method of nearest
  neighbor with equal weights
• Impacts estimated using the difference-in-differen
  ces (DID) approach

30
Estimated Program Impacts (DID)
Bold significant at the 5 level
31
Summary and Implications

• Programs appear to have positive impacts on
  intermediate outcomes, impacts on final outcomes
  are still elusive
• May be due to small sample size or may suggest
  need to improve program design and delivery, and
  if warranted, even consolidation or termination
  of some non-performing programs
• Future research
• may need more sophisticated methods than a
  time-invariant DID estimator
• estimate the effects of differential treatment
  doses

32
Suggestions

• ENESTYC potentially useful vehicle for impact
  evaluations of specific programs and
  cross-program comparisons
• Need to add purposive sample to augment sample
  sizes of program beneficiaries, from CIPI data
  base
• Augmenting ENESTYC sample requires additional
  budget, contributions from different agencies?
• Greater coordination and knowledge-sharing across
  SME programs, of evaluation methods and lessons
  learnt
• Cross-program comparisons not a replacement for
  program specific evaluations and continuous
  monitoring