How to Compare Countries Lecture 4

1
How to Compare CountriesLecture 4

• Michaelmas Term 2004
• Dr. David Rueda

2
Today

• Making Small-N Analyses More Systematic.
• Making Small-N Analyses More Systematic
  Practical Recommendations.
• Making Small-N Analyses More Systematic
  Problems?
• Combining Methods Small-N Analyses Some
  Large-N Analysis.
• Combining Methods Problems.
• How to Combine Methods?
• What We Have Done in This Class.
• Final Recommendations.
• Questions and Comments.

3
Preliminary Point

• An important part of the lecture today is based
  on
• King, Gary, Robert O. Keohane, and Sidney Verba.
  1994. Designing Social Inquiry Scientific
  Inference in Qualitative Research. Princeton
  Princeton University Press.
• Why?
• Some good recommendations. Although there are
  some issues we will talk about.
• The main points in KKV have become accepted.
• KKVs main recommendations
• Making Qualitative Small-N Analysis More
  Systematic.
• Combining methods.

4
Making Small-N Analyses More Systematic

• KKVs Social Science
• The goal is inference
• It can be descriptive inference (using
  observations from the world to learn about
  unobserved facts) or causal inference (using
  observations from the world to learn about the
  causes of what we see).
• The procedures are public
• We should be able to assess the reliability of
  social science analysis.
• The conclusions are uncertain
• Without a reasonable estimate of uncertainty, a
  description of the real world or an inference
  about a causal effect in the real world is
  uninterpretable (KKV 9).
• The content is the method
• Science is about methods and rules, not about
  subject matter (anything can be analyzed
  scientifically).

5
Making Small-N Analyses More Systematic (2)

• What do the previous goals mean regarding our
  small-N analyses
• The goal is inference
• Design your small-N analysis with your
  theoretical goals in mind.
• The procedures are public
• Design your small-N analysis thinking about how
  others will be able assess its reliability. Make
  your analysis clear and available.
• The conclusions are uncertain
• Make it possible to estimate the uncertainty of
  your results.

6
Making Small-N Analyses More Systematic
Practical Recommendations

• There are 4 major components of research design
  research question, theory, data, and use of data.
• Improving the research question
• Ask important questions.
• Make a specific contribution to an identifiable
  scholarly literature.
• Improving the theory
• Make falsifiable theoretical claims.
• Make theoretical claim with observable
  implications.
• Make parsimonious theoretical claims (explain as
  much as possible with as little as possible).
• Can the theory be altered after observing the
  data?
• First, our knowledge of the data (cases, etc)
  will inform our theories.
• It is problematic to alter theory after
  observing the data (specially in a restrictive
  way) without gathering more data to test the new
  theory.

7
Making Small-N Analyses More Systematic
Practical Recommendations (2)

• Improving the data
• Record and report the process by which the data
  are generated.
• Collect data on as many observable implications
  of the theory as possible.
• Maximize the validity of measurement.
• Maximize the reliability of the data collection
  method (would you get the same observations if
  repeated?).
• If possible, the data and the data analyses
  should be replicable.
• Improving the use of the data
• Use data to generate inferences that are
  unbiased. If repeated, would your method produce
  the right answer on average? Are there problems
  like selection bias or omitted variable bias?
• Maximize efficiency use all available data, use
  all relevant information in the data.

8
Making Small-N Analyses More Systematic Problems?

• Is it productive to apply quantitative guidelines
  to our qualitative work?
• What do you think (in your own field)?
• Are all recommendations useful to you?
• My own reservations
• It is not clear to me that the quantification of
  qualitative data would not imply the loss of
  those qualities that King, Keohane and Verba
  themselves are quick to recognize (depth, causal
  sophistication, historical richness, etc).

9
Combining Methods Small-N Analyses Some
Large-N Analysis.

• Why?
• Three reasons
• Combining approaches and putting together
  qualitative and quantitative strategies can
  maximize the benefits and minimize the costs.
• There are things that are best understood through
  quantitative analysis. There are things that are
  best understood through qualitative analysis.
• By combining methods, we may be able to get to a
  point that neither quantitative nor qualitative
  approaches can reach on their own.

10
Combining Methods Small-N Analyses Some
Large-N Analysis (2).

• KKV
• The best research often combines features of
  each qualitative and quantitative. In the same
  research project, some data may be collected that
  is amenable to statistical analysis, while other
  significant information is not. Patterns and
  trends in social, political, or economic behavior
  are more readily subjected to quantitative
  analysis than is the flow of ideas among people
  or the difference made by exceptional individual
  leadership (KKV 5).
• If we are to understand the rapidly changing
  social world, we will need to include information
  that cannot be easily quantified as well as that
  which can (KKV 5).
• All social science requires comparison, which
  entails judgment of which phenomena are more or
  less alike in degree (i.e., quantitative
  differences) or in kind (i.e., qualitative
  differences) (KKV 5).

11
Combining Methods Problems.

• There is the danger of not maximizing the
  benefits or minimizing the costs.
• Worst case scenario
• The quantitative part is excessively simplifying,
  excessively unsophisticated, not informative and
  not systematic. The analysis of the relationship
  between the variables is not convincing and it is
  doubtful it can be generalized.
• The qualitative part does not accomplish any of
  the advantages of a detailed analysis. The
  historical narrative concentrates on the
  succession of events and the relative coincidence
  of explanatory factors and outcomes. There is not
  a clear exploration of processes, a convincing
  explanation of causation, or an investigation of
  different possible combinations of factors.
• Examples?

12
How to Combine Methods?

• No actual recommendation from KKV
• KKVs approach is more directed to the
  quantification of qualitative research than to a
  fruitful integration of both strategies.
• Simple addition?
• The quantitative analysis provides clear
  theoretical goals and robust empirical
  generalizations (the generally acknowledged
  benefits of large-N studies Lijphart 1971 King,
  Keohane and Verba 1994).
• The qualitative analysis provides high degree of
  causal complexity and theoretical sophistication,
  theory corroboration, depth of knowledge, and
  extensive dialogue between data and theory (see
  Bradshaw and Wallace 1991, Eckstein 1975, Amenta
  1991).
• Repetition?

13
How to Combine Methods (2)?

• Providing something that neither method can on
  its own
• To combine quantitative and qualitative methods
  fruitfully, Tarrow (1995) recommends the
  combination of methods not only to increase the
  number of observations (KKVs main argument) but,
  more importantly, to trace processes to identify
  the reasons for particular event dynamics and to
  frame qualitative analyses within quantitative
  structures to avoid incorrect generalizations.
• Example Lisa Martin. 1992. Coercive Cooperation.
  Analysis of the degree of cooperation on economic
  sanctions.
• Quantitative analysis of 99 cases. Certain
  causal inferences are ambiguous so
• Six detailed cases of sanction episodes.

14
How to Combine Methods (3)?

• Triangulation?
• Combining methodologies can achieve
  triangulation.
• Metaphor Using multiple reference points to
  locate an objects position (Jick 1979 602).
• Use of quantitative and qualitative methodologies
  and different kinds of data (individual,
  collective, national, or international) to reach
  conclusions that the analysis of one methodology
  (or one kind of evidence) could not reach.
• When there is convergence in the conclusions
  produced by these different methodologies and
  kinds of evidence, triangulation provides a much
  greater degree of confidence.
• When there is not convergence, moreover,
  triangulation suggests ways in which the theory
  can be improved.

15
What we have Done in This Class

• Different Methods Experimental, Statistical and
  Small N Analyses.
• Why Should We Compare?
• Theory-Driven Small N Analysis.
• Choosing Cases in Theory-Driven Small N Analysis.
• Mills Methods for Small N Analysis.
• Most Similar Systems Design.
• Most Different Systems Design.
• Problems of Most Different and Most Similar
  Systems Design.
• Advantages of Most Different and Most Similar
  Systems Design.
• The Boolean Method, AKA Qualitative Comparative
  Analysis (QCA).
• The Boolean Method in Practice.
• Problems of QCA Analysis.
• A Brief History of Comparative Methods.

16
Final Recommendations

• Consider KKVs recommendations they can be
  useful.
• Theory-based methodological choices.
• Theory drives methodology.
• Theory should be falsifiable.
• Theory should be parsimonious. Maximize
  leverage.
• Theory should be internally consistent.
• Let the question drive the methods.
• Methodologies should be chosen in terms of the
  question that the researcher is trying to answer.
  
• The theoretical advantages and disadvantages of
  the methodologies become obvious when we compare
  them in terms of particular theoretical needs.

17
Final Recommendations

• Choosing and analyzing cases
• Let theory guide the choices.
• Think of your cases conceptually.
• Apply systematic rules to your qualitative
  analysis.
• Maximize variance (be conscious about dependent
  variable variance).
• Maximize the connection between theory and
  evidence (both cases and variables).
• Try to minimize the errors.
• Try control exogenous factors.
• Provide an assessment of uncertainty.
• Combining methods?
• In principle yes, but
• What is the value added? Is it worth the
  investment?

18
Questions and Comments?