Critical Appraisal

1
Critical Appraisal

• Or Making Reading More Worthwhile
• www.bradfordvts.co.uk

2
The Problem

• Vast and expanding literature.
• Limited time to read.
• Different reasons to read mean different
  strategies.
• Keeping up to date.
• Answering specific clinical questions.
• Pursuing a research interest.

3
Stages

• Clarify your reasons for reading.
• Specify your information need.
• Identify relevant literature.
• Critically appraise what you read.

4
Clarify Your Reasons for Reading.

• Keeping up to date.
• Skimming the main journals and summary bulletins.
• Answering specific clinical questions.
• Finding good quality literature on subject.
• Pursuing a research interest.
• Extensive literature searching.

5
Specify Your Information Need.

• What kind of reports do I want?
• How much detail do I need?
• How comprehensive do I need to be?
• How far back should I search?
• The answers to these questions should flow from
  the reasons for reading.

6
Identify Relevant Literature.

• There are many ways of finding literature.
• Remember to ask a librarian they are the
  experts.
• Selectivity is the key to successful critical
  appraisal.

7
Critically Appraise What You Read.

• Separating the wheat from the chaff.
• Time is limited you should aim to quickly stop
  reading the dross.
• Others contain useful information mixed with
  rubbish.
• Simple checklists enable the useful information
  to be identified.

8
Questions to Ask

• Is it of interest?
• Why was it done?
• How was it done?
• What has been found?
• What are the implications?
• What else is of interest?

9
Questions to Ask

• Is it of interest?
• Title, abstract, source.
• Why was it done?
• Introduction.
• Should end with a clear statement of the purpose
  of the study.
• The absence of such a statement can imply that
  the authors had no clear idea of what they were
  trying to find out.
• Or they didnt find anything but wanted to
  publish!

10
Questions to Ask

• How was it done?
• Methods.
• Brief but should include enough detail to enable
  one to judge quality.
• Must include who was studied and how they were
  recruited.
• Basic demographics must be there.
• An important guide to the quality of the paper.

11
Questions to Ask

• What has it found?
• Results.
• The data should be there not just statistics.
• Are the aims in the introduction addressed in the
  results?
• Look for illogical sequences, bland statements of
  results.
• ? Flaws and inconsistencies.
• All research has some flaws this is not nit
  picking, the impact of the flaws need to
  assessed.

12
Questions to Ask

• What are the implications?
• Abstract / discussion.
• The whole use of research is how far the results
  can be generalised.
• All authors will tend to think their work is more
  important than the rest of us!
• What is new here?
• What does it mean for health care?
• Is it relevant to my patients?

13
Questions to Ask

• What else is of interest?
• Introduction / discussion.
• Useful references?
• Important or novel ideas?
• Even if the results are discounted it doesnt
  mean there is nothing of value.

14
NOT JUST A FAULT FINDING EXERCISE !
15
What Is the Method?

• The first task alternative check lists for
  different methods.
• How was the study conducted to confirm the
  method?
• Authors sometimes use the wrong words to describe
  their work!

16
Surveys

• Describe how things are now.
• Samples of populations or special groups.
• The samples must be randomly selected.

17
Surveys

• Do not have separate control or comparison
  groups.
• Comparisons may be made between subgroups but
  this is not control.
• Use of the term survey should identify the method
  beware the use in what is really a cohort study.

18
Surveys

• Cross-sectional is seldom used to describe other
  methods.
• There are many ways of selecting a sample e.g
  stratified, cluster and systematic.

19
Cohort Studies

• Used to find out what happens to patients.
• A group is identified and then watched to see
  what events befall them.
• May have comparison or control groups who must
  be identified from the start.
• Not an essential feature tho.

20
Cohort Studies

• Must have the element of time flowing forward,
  from the point at which they are identified.
• This is sometimes called a retrospective cohort
  study.
• The term cohort should be diagnostic.

21
Clinical Trials

• Testing.
• Always concerned with effectiveness.
• The focus should always be on the outcome.
• The outcomes may not be beneficial in other
  words side effect trials.
• Sometimes cohort trials are used to assess
  effectiveness. This is very poor research and can
  usually be dismissed.

22
Clinical Trials

• When more than two things are compared it makes
  the study more complex and harder to get right.
• The key words to look for are random
  allocation, double blind, single blind,
  placebo-controlled.
• The term outcome is sometimes used in cohort
  studies as well.

23
Case-control Studies

• Ask what makes groups of patients different.
• Select a set of patients with a characteristic
  eg a disease.
• The characteristics of this set are then compared
  with a control group who do not have the
  characteristic being studied but all all other
  respects must be as identical as possible.

24
Case-control Studies

• Case studies look backward not forward as
  cohort studies do.
• Other terms used are case-referrent,
  case-comparator, case-comparison.
• May also be called retrospective as is used for
  some cohort studies.

25
The Results

• The major mental challenge.
• What do I think this really means?
• CAUTION.
• Large unexpected results are rare.
• Flawed studies and misleading findings are
  common.

26
Statistics

• A subject in itself.
• Some general thoughts are worth emphasising.
• Size matters.

27
Probability

• Is just that.
• It is not proof.
• Think of horse racing and the lottery.
• Think of the odds.

28
Pitfalls

• All statistical tests make assumptions about the
  raw data.
• If there is no raw data presented you cannot know
  if the tests are meaningful.
• Outliers.

29
Pitfalls

• Skew.
• Non-independence.
• Serendipity masquerading as hypothesis or data
  trawling!

30
Pitfalls

• Black box analyses. Modern computers make
  statistical testing easy the authors may not
  know what they are doing!
• Bias play devils advocate.
• Confounding.
• A very common problem in medicine.
• Colour televisions do not cause increases in
  hypertension.

31
Checklists

• Checklists for particular types of literature are
  a quick and easy way of learning critical
  appraisal.
• They all have 3 stages
• Basic questions.
• Essential appraisal.
• Detailed appraisal.

32
BUT, BUT, BUT

• Checklists do not tell you about the quality or
  usefulness.
• This is still a subjective question.
• All the lists do is enable a more structured and
  thoughtful response to the subjective question.

33
References

• The Pocket guide to critical appraisal.
• By Iain Crombie.
• Pub BMJ Books, 1996.
• How to lie with statistics.
• By Darrell Huff.
• Pub Pelican, 1989.