How to review and criticize journal articles

1
How to review and criticize journal articles?

• The following slides shall give you some ideas
  for the structure of your critique. However the
  slidesd are not complete and only make sense in
  the context of the lecture and the additional
  slides from the same lecture!

2
Validity

• Which if any concept is used to validate a
  measurement?
• Content validity
• Criterion validity
• Construct validity

3
Design

• Strength of design
• Randomized Experiment
• Main problem external validity
• Panel additional problems
• Non-Equivalence of groups
• Are methods like instrumental variables or
  propensity matching used?
• Are all plausible control variables introduced?
• If no, third variables may modify the correlation
• cross sectional additional problem
• Direction of influence (endogeneity)

4
Data

• How were the data generated?
• What kind of sample was used?
• What is the mode of data collection?
• What are the exact items?
• Are they the same for every unit? (especially
  important in cross-cultural comparisons)
• What about missing data?
• Is the amount documented (response rates, item
  missing)?
• What was done about missing data?
• Weighting
• Imputation

5
Data Analysis

• Are the procedures adequate for the data
• Are the assumptions of the model met?
• Measurement scales
• What is done about missing data?
• Inference
• Multiple tests?
• Correction of confidence level?
• Are design effects taken into account
• For regression models
• Is model-fit discussesd?
• Are regression diagnostics performed?

6
Problems of Macro-Comparative Studies (Schnell
u.a. 2005, 253f.)

• Self-selected Sample
• Small n, but many variables
• Dependence of observations within the same unit
• Dependence of units by proximity/exchange
  relations
• Data collection differs between units
• Data collection changes across time
• Units change across time
• Often reliability and validity of indicators are
  not tested
• Often it is not shown that a measurement took
  place

7
Threats to Validity

• You can read a lot about threats to validity in
  chapters 2 and 3 of Shadish, Cook and Campbell
  (2002). Experimental and quasi-experimental
  designs for generalized causal inference. Boston
  Houghton Mifflin (additional slides at the end of
  the lecture).