APPLICATION OF THE CLOZE PROCEDURE TO

1
APPLICATION OF THE CLOZE PROCEDURE TO EVALUATE
COMPREHENSION AND IMPROVE READABILITY OF WRITTEN
PHARMACY MATERIALS
Miller MJ, DeWitt JE, McCleeary EM, OKeefe
KJ Drake University College of Pharmacy and
Health Sciences

• Background
• Functional health literacy is
• the constellation of skills, including the
  ability to perform basic reading and numerical
  tasks required to function in the health care
  environment1
• measured using a variety of techniques with each
  addressing different aspects of the reading
  process including reading, comprehension, and
  action (See Figure 1)
• The 2003 National Assessment of Adult Literacy2
  estimated
• 14 of adults had below basic health literacy
  skills
• 22 of adults had basic health literacy skills
• Among its recommendations, the Institute of
  Medicine has called for development and use of
  culturally appropriate measures of health
  literacy3

• Methods
• Design
• A descriptive, cross-sectional, in-person
  interview was used
• Data Collection Instruments
• Background characteristics
• Reading comprehension component of the Short Test
  of Functional Health Literacy (i.e., S-TOFHLA)6
• Cloze procedure using a 400-word excerpt from a
  pharmacy-relevant educational brochure titled
  How To Take Your Drugs Safely - Be an Informed
  Consumer.7 (See sample below)
• Bracketed items represent cloze units
  selected for deletion
• Comprehension measured by the percentage of cloze
  units successfully completed using exact word
  matches
• Unfortunately, drug errors happen. They happen
  in hospitals, in pharmacies, or even at home.
  And sometimes people get hurt because of these
  errors.
• The more information you have, the better
  able you are to prevent errors and to take
  care of yourself. You have to ask your
  pharmacists, doctors and nurses about your
  drugs, and you have to expect answers.
• Subject Selection

• Results
• Sample
• Of the 162 subjects, 8 did not meet the
  eligibility criteria
• Final analytical sample was 154 subjects
• Sample was predominantly female (71.4), white
  (93.5), and well educated (93.5 high school
  graduate or above).
• Scoring
• Mean score on the S-TOFHLA was 92.1 (sd13.4)
• 147 (95.5) of the subjects demonstrated adequate
  functional health literacy as measured by the
  S-TOFHLA
• Mean score on the cloze procedure was 53.3
  (sd15.9)
• After applying standard scoring thresholds, only
  69 subjects (44.8) achieved the 60 threshold
  score, demonstrating understanding of the written
  information
• The internal consistencies of the S-TOFHLA and
  the cloze procedure were 0.92 and 0.90
• Scores on the S-TOFHLA and cloze procedure were
  highly correlated (r0.71, plt0.0001)

• Discussion and Conclusions
• First research to report use of the cloze
  procedure to assess comprehension of written
  pharmacy-relevant educational materials and to
  guide revision of the educational materials
• The pharmacy-relevant comprehension assessment
  demonstrated concurrent validity with the
  S-TOFHLA
• Performances on the S-TOFHLA and the pharmacy
  relevant comprehension assessment were highly
  correlated, but not as strong as the relationship
  previously reported between the S-TOFHLA and the
  REALM, a word recognition test
• This study suggests the cloze procedure may more
  effectively discriminate among those with reading
  comprehension problems than the S-TOFHLA
• A majority of respondents failed to demonstrate
  understanding of the pharmacy-relevant
  educational pamphlet, suggesting a need for
  revision
• Subjects responses to individual cloze units
  can be used to enhance the quality of written
  health education materials
• Responses to cloze units provide clues to what
  the reader is thinking in terms of total context
  of the educational material and therefore can
  assist with improving both content and context of
  written educational materials
• Future work should apply the cloze procedure to
  the revised educational materials to reassess
  improvements in comprehension after document
  revision

Figure 1. The Reading Process4

• Review and Revision Process
• Overall, 60 (33/55) of the cloze units had a
  majority of correct responses
• Of the 55 cloze units, 25 were identified as
  consisting of a content word (e.g. home, doctors,
  nurses, answers) and 30 as syntax words (e.g. in,
  the, to, your)
• Of the content words, 40 (10/25) were correctly
  answered by the majority of subjects
• Of the syntax words, 73 (23/30) were correctly
  answered by the majority of subjects
• When a cloze unit had a majority of incorrect
  responses, an appropriate text revision or word
  substitution was considered using reported
  responses as a guide.
• The text was reviewed for clarity and intent.
• Example
• Original text Unfortunately, drug errors
  happen. They happen in hospitals, in
  pharmacies, or even at home. And sometimes
  people get hurt because of these errors.
• Results for completion of the first cloze unit
  in
  Correct 2 1.3
  
  Incorrect 152 98.7
• In this sentence, the word, in, is a
  preposition. While grammatically correct, and
  consistent with the previous phrase, only 7
  people attempted to fill in the blank with a
  preposition (at, within), an adverb (also), and
  conjunction (or). In contrast, 89 (57.8) filled
  in the blank with some variation of doctors
  office 18 others listed another reasonable
  location where errors might occur.
• Possible revisions
• Delete word They happen in hospitals,
  pharmacies, or even at home.
• OR
• Insert words most often chosen They happen
  in hospitals, doctors offices, pharmacies, or
  even at home.

Oral Language and other pre-requisite skills
Reading Tests - WRAT - REALM
Decoding

• References
• American Medical Association, Council on
  Scientific Affairs, Ad Hoc Committee on Health
  Literacy. Health Literacy Report of the Council
  on Scientific Affairs. JAMA. 1999 281(6)
  552-557.
• Kutner M, Greenberg E, Jin Y, Paulsen C. The
  health literacy of Americas adults results from
  the 2003 national assessment of adult literacy
  (NCES 2006-483). US Department of Education.
  Washington, DC National Center for Education
  Statistics 200610-12.
• Nielson-Bohlman L, Panzer AM, Kindig DA
  (Editors). (2004) Health Literacy A Prescription
  to End Confusion. Washington, DC National
  Academies Press.
• Doak CC, Doak LG, and Root JH. (1996). Teaching
  Patients with Low Literacy Skills, Second
  Edition. Philadelphia, PA JB Lippincott Company.
• Taylor WL. Cloze Procedure A new tool for
  measuring readability. Journalism Quarterly.1953
  Fall415-433.
• Nurss JR Parker RM, Baker DW. (2001). TOFHLA.
  Test of Functional Health Literacy in Adults,
  Second Edition. Snow Camp, NC, Peppercorn Books
  and Press.
• Institute for Safe Medication Practices. How to
  take your medications safely. Be an informed
  consumer. http//www.ismp.org/Consumer/Brochure.ht
  ml. Accessed 5/10/2005.
• StataCorp. (2003) Stata Statistical Software
  Release 8.0. College Station, TX Stata
  Corporation.
• Plain Language.gov, Improving Communication from
  the Federal Government to the Public. What is
  Plain Language? http//www.plainlanguage.gov/whati
  sPL/index.cfm. Accessed 10/4/2007.
• Funding Support Provided by
• Bayer Healthcare AFPE - AACP Pharmacy Faculty
  New Investigator Grant
• Drake University Faculty Development Grant

Comprehension Tests - Cloze Procedure -
Listening
Experience Logic Language
Literal Comprehension
Critical Thinking
Inference
Observed Action
Problem Solving Cognitive-affective Response

• Statement of the Problem
• Many previous health literacy research and
  practice efforts have focused on document
  readability and word recognition (i.e. reading
  tests) rather than comprehension
• The cloze procedure4,5 can be used to measure
  comprehension
• cloze unit- a successful attempt to accurately
  reproduce a randomly deleted part of a
  message by deciding, from the context that
  remains, what the missing part should be
• Considers total language context, Osgoods
  learning theory of communication, and statistical
  random sampling
• Objectives
• Use the cloze procedure to assess reading
  comprehension of a pharmacy-relevant educational
  pamphlet
• Use the cloze procedure to guide rewriting of the
  educational pamphlet text to enhance readability