European Communication on Research Awareness Needs WP6

1
European Communication on Research Awareness
NeedsWP6 EBM guide for healthcare
journalists to the critical interpretation of
clinical trials
2
ECRAN EBM guide for healthcare journalists

• What is quality of scientific information?
• Lets sort out the wheat from the chaff
• The huge amount of research studies is a big
  challenge for health care journalists, who have
  to sort out the wheat of scientific excellence
  from the chaff of low quality or interested
  driven science. The purpose of this guide is to
  ensure excellence in medical journalism by means
  of
• applying the core principles of journalistic
  coverage and investigation in the field of
  science and medicine
• training journalists to properly cover news
  events, trends, and issues in all aspects of
  health care, sorting out the tangle of data about
  clinical trials and transferring them to medical
  practice in a comprehensive and multifactorial
  perspective

3
WP3 task 3.2 Others (1)
ECRAN EBM guide for healthcare journalists

• enabling properly trained journalists to better
  convey the complex dimensions of health care
  challenges from an overall point of view
• ensuring an independent coverage about health and
  research, analyzing health related data with
  fairness, accuracy and balance
• in order to
• increase the capacity of individuals and
  policymakers to adopt health care attitudes and
  policies that can improve the health of all the
  citizens
• improve the ability of citizens to make wise
  decisions about behaviors that promote health,
  treatment options, and their choice of health
  care providers.

4
WP3 task 3.2 Others (2)
ECRAN EBM guide for healthcare journalists

• What is science?
• A system of certain knowledge in a period of
  time built up, arranged and ensured by research,
  teaching and accumulated data.
• (translation from the German definition in
  Brockhaus 20032193)
• Which is the goal of science?
• The ultimate goal of science must always be to
  verify or reject existing (content-specific or
  methodical) knowledge and to replace it with new
  knowledge. Science is a cumulative process. The
  touchstone for assessing new knowledge is the
  comparison with existing knowledge. Systematic
  reviews, that collect and evaluate knowledge of
  several scientific studies, have been developed
  and implemented for 20 years as methodical tool
  for assessing research.

5
WP3 task 3.2 Others (3)
ECRAN EBM guide for healthcare journalists

• How should good science look like?
• Assessing good science according to its
  neutrality and objectivity is unrealistic,
  because of its several infrastructural, financial
  and performance-related implications. In the last
  decade the call of transparency has been playing
  a central role in this process, in order to
  ensure the integrity of research. Assessment of
  the framework conditions for research should be
  therefore integrative part of each journalistic
  investigation, in order to set an indicator for a
  preliminary assessment of the reliability of
  research outcomes.
• How should good journalistic investigation look
  like?
• Basic scientific understanding of scientific
  studies and their weak points enables journalists
  to facilitate orientation and to avoid lurking
  pitfalls in this field (e.g. look at the section
  Tips for understanding studies Common
  pitfalls of this guide).

6
ECRAN EBM guide for healthcare journalists

• Toolkit for journalists
• There are many challenges to achieving accuracy,
  balance and completeness in health care
  journalism. For a reporter, there are deadlines,
  editors, and corporate financial pressures.
• Though quality in health care journalism is far
  more than this, because it plays a major role in
  educating consumers! Consumers need to be well
  informed to make sound choices that affect their
  health and well being.
• The following rating criteria are an
  internationally validated methodology, that helps
  conjugate general and medical journalistic
  criteria to review stories claiming about
  specific treatments, tests, products or
  procedures (i.e. drugs or devices, vitamins or
  nutritional supplement, diagnostic and screening
  tests, dietary recommendations, surgical
  procedures).

7
ECRAN EBM guide for healthcare journalists

• Rating criteria for healthcare journalists

1 Quantification of benefits (treatment/test/produ
ct/ procedure)
2 Quantification of harms of intervention
3 Proper evaluation of quality of evidence
4 Further independent source and declaration of
conflicts of interest
5 Story solely or largely based on news release?
6 Establishing true novelty of approach
7 Comparing new approach with existing
alternatives
8 Establishing availability of treatment/test/prod
uct/ procedure
9 Adequate estimate of costs of intervention
10 Is there potential for disease mongering?
8
ECRAN Rating criteria for healthcare
journalists
1

• Quantification of benefits (treatment/test/product
  /procedure)
• Stories are expected to describe the
  effectiveness of a benefit. Few stories provide
  helpful numbers to back the real benefit of a new
  treatment, test, product or procedure. Empirical
  investigations have shown that news reports, drug
  ads and some studies coverage tend to report
  benefits as the relative reduction instead of the
  absolute risk reduction in the frequency of
  failures.
• Another challenging question may be How many
  patients have to be treated in order to prevent
  the problem for one patient?. This is the Number
  Needed to Treat (NNT) it is defined as the
  inverse of the absolute risk reduction and is the
  average number of patients who need to be treated
  to prevent one additional bad outcome.

9
ECRAN Rating criteria for healthcare journalists
1

• What does significantly increased mean? How was
  this measured? Apart from this, statistical
  significance may not be equal to clinical
  relevance. The real question is What difference
  does it make in peoples lives? Reports
  indicating surrogates and intermediate endpoints,
  e.g. changes in blood test scores, may not
  influence individual health scores (increased
  life expectancy or lower mortality rate) or
  quality of live.
• Moreover, anecdotal storytelling makes engaging
  reading but is not scientific data. Glowing
  personal stories or single unchallenged
  exaggerated success stories may not be a
  representative example of how a benefit works and
  require a critical appraisal for a balanced
  framing of it.

10
Criterion will not be fulfilled, in case of
ECRAN Rating criteria for healthcare journalists
1

• Data of possible benefit is not available at all.
• Using only relative, not absolute, risk reduction
  or benefit data.
• Relying too heavily on what may be
  unrepresentative patient anecdotes about
  benefits.
• Citing statistics appropriately, but allowing
  even a single unchallenged exaggerated quote to
  throw the story out of balance.
• Question to clear at the end of the day
• Is the benefit really as much more or less
  significant as it is presented?

11
Quantification of harms of intervention
ECRAN Rating criteria for healthcare
journalists
2

• Be aware of appearances! When you hear someone
  saying that a new approach appears to be safe,
  look for evidence supporting it.
• Stories may emphasize or exaggerate potential
  benefits of interventions, while minimizing or
  completely ignoring their potential harms. For
  some patients slight side effects may even have
  dramatic impact on their lives. An overall
  healthcare report should always consider possible
  even the slightest adverse effects of
  interventions, ideally reporting their frequency
  and severity.
• Any healthcare decision implies tradeoffs, all
  treatments may have side effects, risks and
  complications.

12
ECRAN Rating criteria for healthcare journalists
2

• Even a screening test may turn out to be harmful
  to the individual getting it in fact, it may
  produce a false positive result, which causes
  anxiety, and will probably lead to more
• testing (i.e. biopsy) and possibly treatment
  (i.e. surgical intervention) that comes with its
  own harms. Moreover, enthusiasm for early tests
  in people without symptoms is often not supported
  by evidence.
• PSA testing, for example, often turns out to
  cause more harms than benefits, including false
  positive results and overdiagnosis, both of which
  could lead to unnecessary treatments associated
  with significant side effects.

13
ECRAN Rating criteria for healthcare
journalists
2

• Criterion will not be fulfilled, in case of
• Failing to mention potential harms.
• Failing to quantify potential harms.
• Failing to describe the severity of potential
  harms.
• Failing to account for minor side effects that
  could have a significant impact on a patients
  life.
• Relying too heavily on a patient anecdote about
  safety.
• Relying too heavily on a researchers comment
  that an approach appears to be safe without
  supporting data.
• Question to clear at the end of the day
• Which are the risks and the side effects of this
  intervention or product?

14
ECRAN Rating criteria for healthcare journalists
3

• Proper evaluation of quality of evidence
• Healthcare coverage should critically evaluate
  the evidence and not merely take published or
  presented research as gospel. Quality of evidence
  has to be classified according to evidence
  hierarchy

15
ECRAN Rating criteria for healthcare journalists
3

• Science works cyclically with ebbs and flows
  knowledge stream keeps flowing and keeps changing
  in the course of time. So, ask yourself
• At which point in the stream did I grasp the
  evidence?
• Not all evidence is bullet-proof, even if it is
  published in high impact journals, because not
  all studies are equal. Eminence is not evidence
  and study stories often turn out to be ongoing
  stories.
• A solid journalistic investigation is expected to
  scrutinize limitations of evidence, number of
  probands, duration of the study, relevance of
  outcomes to assess whether what appears as the
  latest story is already existing in larger and
  better-designed stories on the same topic or
  whether the statements generalize small group
  data (e.g. a single medical center with
  experienced surgeons).

16
ECRAN Rating criteria for healthcare
journalists
3

• Criterion will not be fulfilled, in case of
• Failing to dig out the limits of the study design
  (i.e. lack of control group, lack of blinding,
  missing drop out rate).
• Not cautioning on the interpretation of
  uncontrolled data.
• Conflating causation and association (i.e.
  limitations of observational studies).
• Generalizing small group data (i.e. at one
  medical center with experienced surgeons).
• Confounding surrogates with real health outcomes
  (illness or death vs. markers or scores).
• Not mentioning that scientific meeting outcomes
  lack adequate peer review.
• Not pointing out that findings from animal or lab
  experiments are not directly applicable to human
  health.
• Presenting anecdotic data as evidence for
  intervention.
• Question to clear at the end of the day
• Does this study design justify these allegations?

17
ECRAN Rating criteria for healthcare
journalists
4

• Further independent source and declaration of
  conflicts of interest
• Claims about new treatments, tests, products, and
  procedures should be based on evidence, not on
  beliefs or business hopes. In health care
  journalism, it is therefore essential that second
  opinions be sought from independent experts who
  have no vested interest in the approach being
  discussed. For this scope two things must occur
• there must be an independent expert source
  quoted (someone not directly connected with the
  research),
• there must be some attempt to let the public know
  about potential conflicts of interest.

18
ECRAN Rating criteria for healthcare
journalists
4

• Hyperbolic healthcare claims made by drug
  companies, medical device manufacturers and
  academic research centers are to be confronted
  with this question
• Do they have something to gain?

May be that a trial is paid for by the drug
manufacturer researchers are getting fees from a
drug company claims are made by a spokesman for
a device manufacturer doctors are being early
adapters of a new device there are underlying
inventors interests.
19
ECRAN Rating criteria for healthcare journalists
4

• A study funded by industry doesnt necessarily
  mean that findings lack integrity, but evidence
  from recent studies shows that potential
  conflicts might lurk around every corner in
  health care. So, the next time you go to your
  doctors office that is cluttered with drug
  company promotions, it is reasonable for you to
  ask him if he/she had any financial relationship
  with the makers of drugs, devices or test
  advertised for. Moreover, seek independent
  perspectives and beware of statements like
  Experts believe , doctor think . Instead
• Give them a name, because vagueness is nameless!

20
ECRAN Rating criteria for healthcare journalists
4

• Criterion will not be fulfilled, in case of
• Failing to identify the source of the story (news
  release, journal article, editorial, scientific
  meeting presentation, etc.) and failing to make
  obvious the extent to which that source is likely
  to be conflicted (e.g. A PR consultant working
  for the company said or Dr. Smith, who
  received a company grant to perform the study,
  said)
• Failing to include the input of a trusted
  independent source with expertise on the subject
  to comment on the claims made (ideally, more than
  one such independent source would be used).
• Question to clear at the end of the day
• To what extent are the information independent
  and how plausible is the expertise?

21
ECRAN Rating criteria for healthcare journalists
5

• Is the story solely or largely based on
• news release?
• It is simply not sound journalistic practice to
  lift material directly from a news release
  without stating that this is the case. If we know
  that a news release was sent out and if we can
  get a copy of that release, we will review it
  along with the story.
• Many news organizations facing financial
  pressures may be lead to become more open to
  using news releases as a way to publish content
  while cutting costs. Although news releases can
  be valid sources of some information, journalism
  is being asked to collect independent information
  with the consequence that it is unacceptable to
  rely on a news release as the sole source of
  information.

22
ECRAN Rating criteria for healthcare
journalists
5

• Particularly in health care, there are many
  vested interests trying to influence consumer
  choices. Journalists are expected to use
  independent verification relying neither on news
  releases nor on company spokesmen!
• A team from Dartmouth Medical School and the
  Veterans Affairs Outcomes Group published in 2009
  an analysis on the quality of press releases by
  medical centers concluding that Press releases
  from academic medical centers often promote
  research that has uncertain relevance to human
  health and do not provide key facts or
  acknowledge important limitations.

23
ECRAN Rating criteria for healthcare journalists
5

• Criterion will not be fulfilled, in case of
• There is clear evidence that the exact same
  wording from a news release appears in a story.
• Lacking evidence of a press release and the worst
  a story can be rated is not applicable.
• Criterion will be fulfilled, in case of
• A story has several independent sources and
  proves NOT to rely only on a news release.
• Question to clear at the end of the day
• Is the news based only on a press release?

24
ECRAN Rating criteria for healthcare journalists
6

• Establishing true novelty of approach
• Many alledgedly new treatments, tests, products
  or procedures are not really novel. The agent may
  be another member of a well-established
  therapeutic class of drug. Even if it represents
  a new class, it may offer no more than the drugs
  that are widely available. In the press release
  for a new drug this essential information may be
  lost in the hype and the drug is portrayed as
  novel in order to increase initial sales. So
  ask yourself
• Is it novel or simply new?
• Journalists reporting should be able to make
  these distinctions on the real novelty of a
  product by researching or asking independent
  sources, e.g. specifiying that there are previous
  studies on this agent for the same indication.

25
ECRAN Rating criteria for healthcare
journalists
6

• In health care, newer isnt always better, the
  first in a new class of drugs may offer no more
  than the drugs that are widely available. Drug
  companies are very good in promoting the alledged
  novelty of a drug in order to increase initial
  sales. Moreover, some claims for new drugs are
  made for off-label use, i.e. for not officially
  approved uses and under doctors careful
  monitoring. Off-label uses might also result in
  wasting money and harming patients and the
  respective costs may not be covered by the state
  or private insurance. By focusing on a new idea
  journalists should tell how well it works putting
  it into the context of existing alternatives.

26
ECRAN Rating criteria for healthcare
journalists
6

• Mostly it is rare that a study appears out of
  nowhere and the big picture, where the
  novelty comes from, is often missing. To find
  it out journalists should search for the opinion
  of independent experts or reliable scientific
  sources like.

27
ECRAN Rating criteria for healthcare
journalists
6

• Criterion is will not be fulfilled, in case of
• Failing to accurately describe the new
  product/intervention and, as a consequence of
  this, misleading the consumer.
• Failing to explain the real novelty of the
  product/intervention (preferably comparing it to
  existing alternatives on the market).
• Failing to mention that the described new
  product/intervention is, in fact, for off-label
  uses.
• Question to clear at the end of the day
• Is the product/intervention a real novelty?

28
ECRAN Rating criteria for healthcare
journalists
7

• Comparing new approach with existing alternatives
• It is mandatory for a healthcare journalist to
  put the new treatment into the context of
  existing alternatives that have a longer, more
  proven track record. Whatever the new approach,
  the report should try to put it into the context
  of already available interventions, in order to
  compare the effectiveness of the various options.
  News stories on new treatments, tests, products
  and procedures should also take into
  consideration the value for money ratio.
• In case a treatment should have been developed
  from scratch, the journalist is expected to
  explain that there are no alternative treatments
  or only supportive or soothing remedies.

29
ECRAN Rating criteria for healthcare
journalists
7

• To wait and see is also a plausible alternative
  to the proposed treatment/test/product/interventio
  n. Active surveillance, declining immediate
  aggressive therapy, is a viable and rationale
  option.
• It is wise for patients to know their treatment
  options. They may also consider to make lifestyle
  changes rather than taking a drug, or else they
  may pursue a drug or non-surgical option rather
  than having surgery.
• Promotions of screening tests should also
  consider the option for individuals to choose NOT
  to be screened.

30
ECRAN Rating criteria for healthcare
journalists
7

• Criterion will not be fulfilled, in case of
• Discussing a surgical approach without mentioning
  non-surgical alternative approaches.
• Discussing a new test without mentioning other
  tests that are available, including the option of
  not being screened in the case of a screening
  test.
• Failing to discuss the advantages/disadvantages
  of the new idea compared with existing
  approaches.
• Failing to discuss how the new treatment, test,
  product or procedure fits into the realm of
  existing alternatives.
• Question to clear at the end of the day
• Are there any other treatment options besides
  this procedure/product?

31
ECRAN Rating criteria for healthcare journalists
8

• Establishing availability of
• treatment/test/product/procedure
• This criterion shall mostly be applied to
  coverage about a new drug in clinical trials (or
  in a trial on the new use of an established
  drug). A journalist is expected to tell his/her
  readers/public, if the drug/agent is available at
  the chemists (or may be soon) or if it is really
  only in an early-phase trial (e.g. pilot study or
  stage before approval).
• It is important for a journalist to know that
  there may be a lot of marketing hype that
  suggests that something is well-studied and ready
  for use.
• Questions should also be raised by the fact that
  something should have been approved in the U.S.
  and not or not yet in Europe.

32
ECRAN Rating criteria for healthcare
journalists
8

• Some early studies focus on safety and not on how
  well something works. Some other focus on early
  efficacy and may not be large enough or designed
  to show safety of a drug/treatment in the long
  term.
• Predictions about something to be approved by
  in a certain period should also be taken with a
  grain of salt, they usually come from someone who
  stands to benefit.
• New approaches, devices or operations should be
  scrutinized for their availability (no one wants
  to be patient number one with a new
  device/operation).
• Finally, have the personnel required to deliver
  the new approach properly been trained and are
  there enough of them? What is the learning curve
  for this?

33
ECRAN Rating criteria for healthcare
journalists
8

• Criterion will not be fulfilled, in case of
• Making not clear that the drug is not available
  now.
• The approval of a drug is treated as a closed
  issue.
• Something expected to be approved by and its
  availability on the market sounds more like a
  crystal ball prediction.
• Time predictions about the market availability of
  the drug are solely based on declarations sourced
  to a company spokesperson.
• Failing to describe to what extent a treatment is
  available (in every hospital or only in specialty
  clinics).
• Promoting a special intervention without
  providing a sense of how many trained doctors
  there are in the field (e.g. a hi-tech surgical
  intervention).
• Question to clear at the end of the day
• Is this item available now or not?

34
ECRAN Rating criteria for healthcare journalists
9

• Adequate estimate of costs of intervention
• Journalists are expected to appropriately report
  on intervention/product costs, though it may be
  difficult to estimate costs of an experimental
  approach early in its development. Even if the
  real cost is probably known at the time of market
  entry, health reporters could at least cite costs
  of existing alternatives. Journalists should be
  able to get a reasonable estimate from their
  source/experts on the topic.
• The rule of thumb is
• If it is not too early to talk about
• how well something might work,
• then it is not too early
• to start discussing what it may cost

35
ECRAN Rating criteria for healthcare journalists
9

• A large amount of healthcare stories do not
  appropriately report on the cost issue.
  Healthcare takes a large part of the Gross
  Domestic Product in developed countries and we
  all better start thinking about costs of
  treatments, tests, products and procedures, we
  all pay for.
• So, it is not enough to say
• The cost is much lower than
• but you should better ask
• What does this actually cost?
• How much lower?
• What about the cost-effectiveness, if compared to
  ?
• Journalists should also consider additional
  expenses for psychologists, dieticians,
  physiotherapists involved in the intervention or
  that the convalescence and rehabilitation may be
  more time-consuming.
• Moreover, they should investigate, if insurances
  cover the intervention/drug or it is affordable
  for the patients needing them.

36
ECRAN Rating criteria for healthcare journalists
9

• Criterion will not be fulfilled, in case of
• Costs of an intervention are not mentioned at
  all.
• Costs are played down without giving a
  satisfactory explanation therefore.
• Not comparing the costs of a procedure with other
  available alternatives (more or less expensive
  ones).
• Not mentioning, if the new procedure/drug will be
  covered by public/private health insurance.
• Criterion will not apply, in case of
• An experimental procedure in early development
  with no mention of the related costs.
• An expert claiming that costs are quite
  unimportant (e.g. speaking of well known products
  like aspirin or weight-loss products).
• Question to clear at the end of the day
• Is it covered by health insurance or must I pay
  out of my pocket?

37
ECRAN Rating criteria for healthcare journalists
10

• Is there potential for disease mongering?
• This criterion assesses whether a condition is
  exaggerated or oversold. There are different
  forms of mongering
• turning risk factors into diseases (e.g. low
  bone mineral density becomes osteoporosis)
• misrepresentation or inaccurate description of
  the natural history and/or severity of a disease
  (e.g. early-stage low-grade prostate cancer)
• medicalization of minor or transient variations
  in function (e.g. temporary erectile dysfunction
  or female sexual dysfunction)
• medicalization of normal states (baldness,
  wrinkles, shyness, menopause)
• exaggeration of the prevalence of a
  disease/disorder (e.g. using rating scales to
  diagnose erectile dysfunction).

38
ECRAN Rating criteria for healthcare journalists
10

• Some statements are trying to make all of us feel
  like theres something wrong with us even if
  there isnt. Thats what we call
  disease-mongering. This is dangerous because it
  is an unnecessary emotional and financial burden
  for society and may distract us and precious
  public resources from important healthcare issues
  that should be addressed. Also turning a normal
  state or variation of health into a medical
  problem is disease mongering, like baldness,
  wrinkles, shyness or menopause. Pseudodiseases,
  that lower the threshold for common diseases but,
  in fact, only open new markets, may turn millions
  of people overnight into patients needed to be
  treated with drugs or vitamins against high
  cholesterol or osteoporosis. Often, women are
  targeted because of premenstrual dysphoric
  disorders, female sexual dysfunctions, overactive
  bladders or menopause men equally get their
  share being hammered because of low testosterone
  levels, balding etc.

39
ECRAN Rating criteria for healthcare journalists
10

• Identifying disease mongering is a matter of
  judgment. Sometimes it is obvious. Sometimes
  theres a fine line about whether a statement on
  irritable bowel disorder, erectile dysfunction,
  restless leg syndrome or osteoporosis (all of
  which can be serious for some sufferers) is
  misrepresenting the condition to the public.
  Sometimes there may be multiple mongering with
  many hyperboles for the same issue.

40
ECRAN Rating criteria for healthcare journalists
10

• Criterion will not be fulfilled, in case of
• Spurious statistics are presented. Question
  prevalence estimates!
• Statistics inflate the seriousness of a condition
  (e.g. millions of people suffering in silence
  with toenail fungus)
• Someone selling sickness wants to scare you
  into testing or treatment.
• Risk factors are treated as diseases to convince
  us that we should be concerned about a score
  (i.e. surrogate markers, e.g. changed blood
  parameters) instead of outcomes like illness or
  death.
• Holding up worst-case patients experiences as
  representative examples of all with this
  conditions.

41
ECRAN Rating criteria for healthcare journalists
10

• Criterion will not apply, in case of
• A therapy/procedure/test/drug possibly being
  overestimated. There is no treatment mongering
  as such. This item should be best scrutinized
  under criterion Quantification of benefit or
  Proper evaluation of quality of evidence.
• Question to clear at the end of the day
• Should I really be concerned about this
  disease/circumstance?

42
ECRAN General journalistic criteria
11

• Choosing the right issue
• Is it a current, relevant and original
  contribution?
• Topicality of an issue may be related to the
  event and the frequency of medium publication,
  the importance or the originality (i.e.
  curiosity) of its contents. Relevance may occur
  for instance, if the issue directly or indirectly
  regards a large part of the population, if a
  therapy determines high costs for the health
  system, if a new development has an ethical
  dimension, if the topic has political importance
  or an exemplary nature. If not relevant, an issue
  may instead be curios, humorous, surprising etc.
  Novelty of an issue may be characterized by
  latest data published e.g. today, yesterday, this
  week in a scientific journal, or outcomes of
  recently held conferences.
• The rule of thumb is
• If the issue should not be current, then it must
  be relevant. If not relevant, then it must be at
  least curios/original!

43
ECRAN General journalistic criteria
12

• Delivery of message
• Has the issue been successfully delivered or
• does it even have model character for its format?
• A successful message must be delivered in an
  intelligible language, with a plain structure
  avoiding long and complicated sentences.
  Specialist or technical terms should only be used
  in exceptional cases and, anyway, accordingly
  explained by means of context or examples.
• Formal factors also play an important role the
  story must follow dramaturgical principles
  (personalisation, narrative elements, relation to
  everyday life) form and content must harmonise
  abstract thinking must be explained by pictures,
  illustrations, clear specimen texts good sound
  clips in a radio report should invite to listen
  to medical issues. In case of reports, WH
  questions should all be answered.
• Rule of thumb delivery standard is conform with
  the format

44
ECRAN General journalistic criteria
13

• Factual accuracy
• Does the story report essential facts?
• Factual accuracy means reporting historical facts
  without manifest or essential errors, e.g.
  accurate reporting of study conclusions or data,
  correct description of study design (i.e. number
  of participants enrolled/allocated/followed
  up/analysed in clinical trials). Accuracy is
  neither synonymous with objective truth nor
  completeness, e.g. a report must not comprise
  each and every aspect of the study but the
  described facts must be true. A single name error
  is not enough to invalidate this criterion, but
  many little mistakes may create the impression of
  a real sloppy work!
• Apart from purely data errors or hyperboles in
  the headlines or teaser beware of surrogate
  markers, association vs. correlation, absolute
  vs. relative risks, NNT, off-label use, phases of
  clinical trials, confidence interval etc ...

45
ECRAN Tips for understanding studies Common
pitfalls

• Surrogate markers
• A surrogate marker is an event or a laboratory
  value that researchers hope can serve as a
  reliable substitute for an actual disease. A
  common example is blood cholesterol levels which
  are surrogates, or substitutes, for heart
  disease.
• As a matter of fact, surrogate studies are
  indirect by definition and a journalist should be
  skeptical before celebrating the last medical
  breakthrough, because what is sound medical dogma
  today may fade over time. Surrogate research is
  an important research tool, that can open
  important therapeutic avenues and stimulate
  additional research. Those who earn their living
  and reputation from medical research may have a
  different view on surrogacy, but it is important
  to understand that exaggerations may damage
  patient care by highlighting surrogate outcomes
  instead of important results such as death,
  quality of life or functional capacity.

46
ECRAN Tips for understanding studies Common
pitfalls

• Association vs correlation and language use
• Look at this wording describing the results of an
  observational study
• Frequent fish consumption was associated with a
  50 reduction in the relative risk of dying from
  a heart attack.
• And look at the editors streamline adaption
  detrimental to meaning
• Women who ate fish five times a week cut their
  risk of dying later from a heart attack by half.
• The new wording suggests that the subjects fish
  consumption is responsible for their dying less
  frequently from heart attacks, but the original
  study does not support a conclusion of cause and
  effect.
• Epidemiologic or observational studies
  examine the association between what is known in
  epidemiologic jargon as an exposure (e.g. a food,
  environmental influences, lifestyle) and an
  outcome (often a disease or death). Because of
  all the other exposures occurring simultaneously
  in the complex lives of free-living humans that
  can never be completely accounted for, such
  studies cannot provide evidence of cause and
  effect they can only provide evidence of some
  relationship (between exposure and outcome) that
  a stronger design could explore further.

47
ECRAN Tips for understanding studies Common
pitfalls

• The only study design involving humans that does
  rise to the level of demonstrating cause and
  effect is a randomized trial. In this design,
  study subjects are assigned an exposure (or a
  control condition) at random, irrespective of any
  other exposures in their lives, and all such
  other exposures are then assumed to even out
  between the treated group and the control group
  of subjects (and this can be demonstrated). As a
  result, the only difference between the groups is
  whether they receive the exposure under study or
  the control condition. This approach is a true
  experiment. Any difference in outcome seen
  between the control and the experimental group
  should be due to the one factor or variable that
  differs.
• A subtle trap occurs in the transition from the
  cautious, nondirectional, noncausal, passive
  scientifical language in reporting the results of
  observational studies to the active language used
  in mass media. Active language is fine in general
  (who wants to write like a scientist?), but
  problems can arise when the use of causal
  language is not justified by the study design. In
  practice, a shift to causal language can occur at
  any stage writing, editing, or headline
  composing, with similar effects on meaning, that
  could lead readers to overestimate the meaning of
  a given study and possibly even make life choices
  that the evidence does not warrant. Sometimes
  even scientists and press-release writers slide
  into causal language in expressing results of
  observational studies.  

48
ECRAN Tips for understanding studies Common
pitfalls

• Absolute vs relative risks
• Researchers, clinicians or journalists often
  report only relative differences in making claims
  about a new idea, but they should tell the rest
  of the story!
• Absolute differences make the real sense out of
  such claims.
• Look at this example about blindness in diabetic
  patients in a 5-year-period
• If the risk for blindness is 2 in 100 (2) in a
  group of patients treated conventionally and 1 in
  100 (1) in patients treated with a new drug, the
  absolute difference is derived by simply
  subtracting the two risks 2 1 1.
• Expressed as an absolute difference, the new drug
  reduces the 5-year risk for blindness by 1.
• The relative difference is the ratio of the two
  risks. Given the data above, the relative
  difference is 1 2 50
• Expressed as a relative difference, the new drug
  reduces the risk for blindness by half.
• Data are accurate in both cases, but if you are
  a marketing manager for the new drug, you are
  likely to only use the relative risk reduction
  if you are a journalist, you would better do your
  job by citing the rest of the story (absolute
  risk reduction) often missing in news releases
  and direct-to-consumer prescription drug ads.

49
ECRAN Tips for understanding studies Common
pitfalls

• Number needed to treat (NNT)
• The NNT is defined as the inverse of the absolute
  risk reduction or the number of patients who need
  to be treated to prevent one additional bad
  outcome, calculated as 1/absolute risk reduction.
  
• Applied to our hypothetical diabetes blindness
  drug example
• the risk for blindness in a patient with diabetes
  over a 5-year period is 2 in 100 (2) in a group
  of patients treated conventionally and 1 in 100
  (1) in patients treated with a new drug. So the
  absolute difference is derived by simply
  subtracting the two risks 2 1 1.
• the number needed to treat would be 1 / 1 (or
  .01) 100.
• so you would need to treat 100 people with
  diabetes for 5 years in order to prevent one case
  of blindness.
• You can see that this is an important way to look
  at new claims about new drugs.

50
ECRAN Tips for understanding studies Common
pitfalls

• Off-label use
• Prescription drugs are officially approved by
  regulatory agencies for specific indications to
  treat specific conditions or diseases. Physicians
  may prescribe a drug for a use that is not
  described in the approved labeling if it seems
  reasonable or appropriate to them. This is whats
  called off-label use.
• For example it is not uncommon for physicians to
  prescribe low doses of beta blocker drugs to help
  people overcome jitters before public speaking.
  Beta blockers are not formally approved for this
  use, but in such circumstances doctors are
  advised that they have the responsibility to be
  well informed about the product, to base its use
  on firm scientific rationale and on sound medical
  evidence, to inform the patients and get its
  consent for the use of the drug.
• A study in the Archives of Internal Medicine
  found that about 21 of all estimated uses for
  commonly prescribed medications were off-label,
  and that 15 of all estimated uses lacked
  scientific evidence of any useful or beneficial
  effect. Authors of the study recommended more
  careful monitoring of off-label uses that might
  waste money and harm patients. Another problem of
  the off-label prescribing is that its cost may
  not be reimbursed by public or private health
  insurances.
• Finally, whilst off-label use is permitted,
  marketing of a drug for an off-label use is not.

51
ECRAN Tips for understanding studies Common
pitfalls

• Phases of clinical trials
• Journalists who report on drugs while they are
  still in clinical trials need to understand the
  distinction between phases I, II, and III of drug
  trials. It is misleading to report bold or
  conclusive statements about how well a drug works
  when it is only in phase I trials, since the
  primary goal of phase I trials is to evaluate how
  safe a drug is, not how well it works. Many times
  journalists report on early phase drug trials as
  if all the evidence is in hand.
• Advice for healthcare journalists
• Give accurate portrayals of the status of
  investigational drugs, devices and procedures,
  including significant caveats and explanations of
  hurdles, unknown and potential problems. 
• Without such caveats consumers would be better
  advised to have doubts about the accuracy and
  balance of the story.

52
ECRAN Tips for understanding studies Common
pitfalls

• Confidence interval
• In the interpretation of trial results scientists
  divide errors into two classes a type I error is
  the mistake of thinking something is true when it
  is not (also known as a false positive) a type
  II error is thinking something is not true when
  in fact it is (a false negative). When testing a
  specific hypothesis, scientists run statistical
  checks to work out how likely it would be for
  data which seem to support the idea to have come
  about simply by chance. If the likelihood of such
  a false positive conclusion is less than 5, they
  deem the evidence that the hypothesis is truly
  statistically significant. They are thus
  accepting that one result in 20 will be falsely
  positive but one in 20 seems a satisfactory low
  rate.

53
ECRAN EBM guide for healthcare journalists

• Valuable papers fur further reading
• This list has been updated at November 2013
• To find further contents, visit ECRAN website
  www.ecranproject.eu
• - Opportunities for research and NIH
• http//www.sciencemag.org/content/327/5961/36.summ
  ary
• - Demystifying trial networks and network
  meta-analysis
• http//www.bmj.com/content/346/bmj.f2914
• - Independent drug testing to ensure drug safety
  and efficacy
• http//papers.ssrn.com/sol3/papers.cfm?abstract_id
  2328348
• - Editorial policies for clinical trials and the
  continued changes in medical journalism
• http//jama.jamanetwork.com/article.aspx?articleid
  1699471
• - Informing the uninformed optimizing the
  consent message using a fractional factorial
  design
• http//www.ncbi.nlm.nih.gov/pubmed/23700028

54
ECRAN EBM guide for healthcare journalists

• - The Womens Health InitiativeA victory for
  women and their health
• http//jama.jamanetwork.com/article.aspx?articleID
  1745653
• - Better reporting of scientific studies why it
  matters
• http//www.plosmedicine.org/article/infodoi/10.13
  71/journal.pmed.1001504
• - Quantification of harms in cancer screening
  trials literature preview
• http//www.bmj.com/content/347/bmj.f5334
• - Risks (and benefits) in comparative
  effectiveness research trials
• http//www.nejm.org/doi/full/10.1056/NEJMp1309322
• - The right to participate in high-risk research
• http//philpapers.org/rec/SHATRT-2
• - Whos afraid of peer review?
• http//www.sciencemag.org/content/342/6154/60.full
  
• - Why we cant trust clinical guidelines
• http//www.bmj.com/content/346/bmj.f3830
• - Practices and impact of primary outcome
  adjustment in randomized controlled trials
  meta-epidemiologic study
• http//www.bmj.com/content/347/bmj.f4313

55
ECRAN EBM guide for healthcare journalists

• - US study criticized for experimentation with
  premature infants
• http//www.bmj.com/content/347/bmj.f4198
• - Do clinical trials work?
• http//www.nytimes.com/2013/07/14/opinion/sunday/d
  o-clinical-trials-work.html?pagewantedall_r2
• - Understanding clinical trials appreciating
  medical heroes
• http//doc.mediaplanet.com/all_projects/4921.pdf
• - Dispelling the many myths about clinical trials
• http//www.newswise.com/articles/dispelling-the-ma
  ny-myths-about-clinical-trials