Critical Appraisal Skills Basic I

1
Critical Appraisal SkillsBasic I

• NHS Education Scotland
• Produced in collaboration with the
• Association of Scottish
• Medicines Information
• Pharmacists Group

2
What is critical appraisal?

• This is the term given to describe the skills
  used when reading a paper to enable one to assess
  the validity (i.e. how close to truth) and
  usefulness (i.e. can the results be applied to
  your practice) of the results.
• Forms an integral part of evidence based medicine
  (EBM).

3
What is EBM?

• EBM is the judicious use of current best
  evidence, combined with clinical experience, to
  make decisions about patient care.

4
5 key steps that underpin EBM

• Define the specific question to be answered.
• Find the best evidence to answer the question.
• Critically evaluate the evidence to assess it
  validity and usefulness.
• Apply the results of the critical evaluation to
  practice.
• Evaluate the performance of the intervention.

5
Why do we need evidence?

• Resources should only be allocated to
  interventions that are effective.
• The only way of judging effectiveness is EVIDENCE!

6
What are good sources of evidence?

• Less reliable sources
• Glossy literature from pharmaceutical companies
• Press releases from pharmaceutical companies
• Magazines such as Pulse
• Advertisements in medical journals
• Conference abstracts about clinical trials

• Trusted sources
• Scottish Medicines Consortium
• SIGN
• NICE
• Knowledge Network
• Peer reviewed clinical journals
• Summaries of information published by NHS bodies
  (e.g. UKMi QAs)

7
Hierarchy of evidence

• Evidence comes in different forms and can be
  ranked in terms of importance.
• Quite often there may not be any high-levels of
  evidence to support a clinical intervention. In
  these cases it may be necessary to use evidence
  from the lower end of the hierarchy scale.

8
Systematic reviews Meta-analyses
Hierarchy of evidence
RCTs
Cohort studies
Case control studies
Cross sectional surveys
Case series then Case reports
Expert opinion/ clinical experience from
respected sources
9
Format of clinical trials

• Clinical trials are usually written in a standard
  format. This normally consists of
• Title
• Authors
• Abstract contains a brief summary of the trial.
• Introduction
• Methods
• Results
• Discussion
• Conclusion

10
Key questions to ask when assessing how valid and
useful a clinical trial is

• Patient or population does the trial assess a
  commonly seen clinical condition and were the
  patients studied similar to your patient?
• Intervention what was the medicine being
  tested? Was it used at the normal dose?
• Comparison What was the experimental medicine
  compared to? Trials that compare a new medicine
  to a placebo are not useful when trying to decide
  where the medicine fits into current practice.
• Outcome Was the end-point relevant to the
  patient (i.e. was it patient-oriented like a
  reduction in risk for a having a heart attack)?

11
Outcomes

• Patient orientated
• Outcomes that directly improve the outcome for
  patients.
• e.g. Reduction in hip fracture improved
  cardiovascular mortality prevention of a stroke

• Disease orientated
• Outcomes that are the result of a change to a
  disease characteristic
• e.g. Change in bone mineral density lowering of
  blood pressure a reduction in cholesterol

12
Surrogate outcomes

• Outcomes that are not patient-orientated are
  surrogate outcomes.
• Often physiological or biochemical markers
• Cannot always assume that they are always a good
  indication of disease progression or improved
  survival.

13
Three key factors affecting the results of a trial

• The intervention
• Bias researchers take steps to minimise by use
  of a control group, randomisation and blinding
  but some biases can still exist.
• Chance statistical tests are used to assess
  this.

14
Types of Statistics

• Descriptive
• Summarises or describes the sample
• Please note that for purposes of critical
  appraisal this type will not be discussed in this
  training package.
• Inferential
• Concerned with generalising from the sample to
  make inferences and estimates about a wider
  population.

15
Inferences and Estimates

• Inferences
• Can conclusions be drawn from the sample be
  generalised to the population?
• e.g. If a better response is seen with a medicine
  in a sample will it hold true in the general
  population
• Help answer whether results may have occurred by
  chance in the trial.
• Estimates
• Given an observed size of effect in the sample,
  what is the likely value (or range of values) you
  will see in the population?
• Help assess usefulness of a trial.

16

• Inferential Statistics

17
Hypothesis Testing

• There are 1 of 2 assumptions made for
  interventions in clinical trials
• Null hypothesis (i.e. no difference between the
  control group and the experimental group).
• Alternative hypothesis (i.e. there is a
  difference between the control group and the
  experimental group).
• Generally it is the null hypothesis that is
  assumed however.

18
Probability (P)

• The probability that a difference will be seen
  between 2 interventions in a clinical trial.
• Measured on a scale of 0 (impossible for event to
  happen) to 1 (the event will certainly happen)
• i.e. P 1 would always happen
• P 0.05 would happen 1 time in 20
• P 0.02 would happen 1 time in 50
• P 0.01 would happen 1 time in 100
• P 0.001 would happen 1 time in 1000

19
P values

• If p-value is less than 1 in 20 (plt0.05) then
  the result is regarded as being statistically
  significant and the possibility of the
  difference observed arising by chance is low
• gt If this is the case then one
  can reject
• the null Hypothesis

20
Power

• The probability that a test will detect a real
  difference in treatment outcomes in a sample if
  it is present in the population
• Usually expressed as a percentage and often set
  at 80-90

21
Power and Sample Size

• Sample size determinants
• Size of the difference being investigated
• Level of significance

22
Errors that can arise when drawing conclusions
from data

• Type 1 error (alpha)
• The data suggests a difference between the groups
  when there is really no difference False
  positive
• Often called significance level
• A level of significance of plt0.05 represents a 5
  probability of making a type 1 error

23
Errors that can arise when drawing conclusions
from data

• Type 2 error (beta)
• The results fail to pick up a real difference
  that exists between the groups, and a conclusion
  is made that no difference exists False
  negative
• 100-(power) is the probability of making a
  type 2 error

24
Population Estimates
25
Standard Error of the Mean (SEM)

• A tool for inferring the characteristics /
  parameters of a whole population from the
  measurements in one sample
• One of the most widely misused terms in
  statistics
• In 95 of cases the True Population Mean will
  lie within /-2 SEM of the sample mean
• SEM SD
• ------
• ?n
• Should NEVER be used instead of SD to indicate
  dispersion of measurements

26
Confidence Intervals (CI)

• Represents the range of values within which the
  true population mean lies.
• Indicate the precision (or imprecision) with
  which a study sample estimates the true
  population value for the whole population.
• Important role whenever we wish to apply
  results of a clinical study to the general
  population
• Narrower the range the more reliable the results

27
Confidence Intervals

• Calculated by adding and subtracting multiples of
  the standard error of the
• mean to and from the sample mean
• 95 confidence interval normally used (i.e. can
  be 95 confident that the
• population value lies within this interval) or
  alternatively stated that there is a
• 1 in 20 chance (5) that the true value lies
  outside the range quoted.
• The narrower the CI, the more confident you can
  be the sample represents the
• population

28
How to Interpret?

• Comparing means
• No difference if CI overlap (i.e. even though 2
  mean values may
• differ, extensive overlap of their respective CIs
  may suggest that the
• difference is not statistically significant)
• When comparing differences between means
• No difference if CI includes 0
• For proportions (e.g. RR)
• No differences if CI includes 1
• For further information on this topic
• -Statistics in divided doses Number 8 (July
  2005).Produced by the North West Medicines
  Information Service. Available at
  http//www.ukmi.nhs.uk/filestore/misc/StatsinDivDo
  se8.pdf

29
Estimation Statistics

• Help assess usefulness of the trial by
  determining clinical importance and magnitude of
  the benefit by using data to estimate a range of
  probable values for the population.

30
Example study
Group Total Number of patients in each group Pain free within 2 hours
Intervention Group (Received Drug X) 2073 845
Control Group (Received Placebo) 1128 96
31
Event Rates

• Definition
• The proportion of patients in whom an event is
  observed
• Control Event Rate (CER)
• Vs
• Experimental Event Rate (EER)

32
Control Event Rate

• Control Event Rate (CER)
• Event Rate in control group
• Total patients in control group
• Example
• CER 96/1128 0.085 (9)

33
Experimental Event Rate

• Experimental Event Rate (EER)
• Event Rate in experimental group
• Total patients in experimental group
• Example
• EER 845/2073 0.41 (41)

34
Absolute Risk Reduction

• Absolute Risk Reduction (ARR) is way of
  expressing differences between groups.
• It is the difference in the event rate between
  the control event rate (CER) and the experimental
  event rate (EER).
• ARR CER-EER
• Example
• ARR 9-41 -32

35
Relative Risk Reduction

• Is an alternative means of expressing the
  difference between groups as a percentage
• The Relative Risk Reduction (RRR) is the percent
  reduction in events in the experimental event
  rate (EER) and the control event rate (CER).
• RRR (CER-EER) X 100
• CER
• Example
• RRR (9-41/9) X 100 356

36
Why calculate?

• Sometimes the trial may just state the
  treatment reduced the risk but does not state
  whether this is relative risk reduction or
  absolute risk reduction. Obviously the relative
  risk reduction looks more impressive since a
  larger number. Be aware of this and use the
  figures given to calculate both.
• Neither RRR or ARR are intuitive ways to look at
  data. Numbers needed to Treat (NNT) is the more
  relevant way to look at the figures.

37
Numbers Needed to Treat (NNT)

• Definition
• The number of people who needed to be treated
  to produce one particular clinical outcome
• (e.g. How many patients need to receive Drug X
  instead of placebo to allow one patient to be
  pain free at 2 hours?)
• NNT ____1____ or ____1____
• CER-EER ARR
• Example
• NNT 1/32 3

38
Numbers Needed to Harm (NNH)

• This value can be similarly calculated when
  looking at adverse effects in a clinical trial.
• It is the number of patients you would need to
  treat with the experimental medicine (rather
  than the placebo or control) for one additional
  patient to suffer an adverse effect.
• NNH 1/(EER-CER)

39
Example NNH

• Medicine Y is given to patients for treatment of
  hypertension. 12 of the 4000 patients given
  medicine Y experience a rash compared with 2 out
  of 3000 given placebo.
• CER 2/3000 0.00066
• EER 12/4000 0.003
• NNH 1/ (0.003-0.00066) 428
• Therefore 428 patients must be treated with
  medicine Y
• rather than placebo for an additional 1 patient
  to have an
• adverse effect.

40
Relative Risk

• The relative risk (RR) is the size of the effect
  in the experimental group
• relative to the size of the effect in the control
  group. The relative risk is
• often quoted in a clinical trial paper.
• RR CER/EER
• Example
• RR 9/41 0.21
• A relative risk of 1.0 means that there is no
  difference between the
• experimental and control groups. This result
  shows a RR lt 1.0
• indicating that the patients on the medication
  are more likely to be
• pain free at 2 hours than those receiving
  placebo.

41
Odds Ratio (OR)

• The ratio of patients in the treatment group
  succumbing to a particular end point compared to
  the control group
• Compares the probability of the event occurring
  with the probability that it will not occur.
• If gt1 event more likely to happen
• If lt1 event less likely to happen

42
Odds Ratio (II)

• The odds ratio must be calculated first for
  control and treatment group
• In isolation it is difficult to imagine what this
  figure means but an OR of 1
• means that the two groups were equally likely to
  be pain free within 2 hours.
• An OR higher than 1 means that the treatment
  group was more likely to
• experience the event (pain free within 2 hours)
  than the control group.
• An OR of less than 1 would mean that the
  treatment group was less likely to
• be pain free at 2 hours. In this case the OR was
  8 so the treatment group was
• 8x more likely to be pain free at 2 hours.

43
Summary

• Dont always believe everything you read!
• Choose your source of evidence wisely and
  systematically to answer your question.
• Use estimation statistics to help evaluate
  usefulness and clinical importance of a trial.
• Utilise population estimates to determine how
  reflective of the true population the trial
  results are likely to be.
• Statistical significance does not always equate
  to clinical significance.
• There is a lot of information available but you
  have to choose the best evidence available.
  Remember that all evidence is not equal!

44
References

• Brignell J. How do Relative Risk and Odds Ratio
  compare? (April 2006). Available at
  http//www.numberwatch.co.uk/rror.htm
• Burls A. What is Critical Appraisal in Evidenced
  Based Medicine 2nd ed. Oxford University of
  Oxford. Available at www.whatisseries.co.uk
• DeCaro, S. A. A student's guide to the conceptual
  side of inferential statistics (2003). Available
  from http//psychology.sdecnet.com/stathelp.htm.
• Easton V, McColl JH. Confidence Intervals in
  Statistics Glossary V1.1. Available from
  http//www.stats.gla.ac.uk/steps/glossary/confiden
  ce_intervals.htmlconfinterval
• Greenhalgh T. How to read a paper The basics of
  evidence based medicine 2nd edition. London BNJ
  Books 2001.
• Swinscow TDV, Campbell MJ. Statistics at Square
  One 10th edition. London British Medical
  Association 2002.
• Statistics in Divided Doses, Assessing the
  reliability of a sample (Number 3). North West
  Medicines Information Service (September 2001).
  Available at http//www.ukmi.nhs.uk/filestore/misc
  /StatsinDivDose3.pdf
• Statistics in Divided Doses, Variability,
  probability and power (Number 4). North West
  Medicines Information Service (May 2002).
  Available at http//www.ukmi.nhs.uk/filestore/misc
  /StatsinDivDose4.pdf
• Statistics in Divided Doses, First steps in
  analysis - comparing the means of large samples
  (Number 5). North West Medicines Information
  Service (November 2002). Available at
• http//www.ukmi.nhs.uk/filestore/misc/StatsinDivDo
  se5.pdf
• Statistics in Divided Doses, Confidence intervals
  (Number 8). North West Medicines Information
  Service (July 2005). Available at
  http//www.ukmi.nhs.uk/filestore/misc/StatsinDivDo
  se8.pdf
• Wills S et al. Critical Appraisal of Clinical
  Trials E-learning Module via NHS Education South
  Central. Available for free registration at
  http//www.learning.nesc.nhs.uk/