Results of TRACs Data Reliability

1
Results of TRACs Data Reliability Feedback
Supervision
VCT, PMTCT, ART

• Novotel, Kigali, Rwanda
• May 2nd, 2006

Scialfa, T. (Tulane University), Mukandori, D.
(TRAC/ME Unit), Hajabashi, J. (Consultant)
2
Part 1

• A component of quality assurance Access to
  services

Internal organization of the service Waiting time
conditions Variation in days/hours of service,
Hours of service vs the workday
3
Results Organization of services

• VCT and PMTCT
• Reception (some sites), group counseling (some
  sites), indiv. Counseling, blood sample,
  (referral if necessary), multiple samples to lab,
  lab results to counselors, client return for
  post-test counseling/results (referral if
  necessary)
• Same organization at all sites with slight
  variation
• group IEC or no group IEC
• Registration at reception before or after group
  IEC
• Most clients wait for several (up to 8) hours for
  results and have no sheltered place to sit
• Only Kabgaye limits to 15 clients/couselor/day
  because they stated this is the MINISANTE norm.
• There is no triage at the reception

• ART
• There is a lot of variation in the way ART
  services are organized
• Generally speaking, however, clients who are HIV
  are referred (sent) from the VCT service to ART.
  I f tested positive at a different site, they
  might be retested at that ART site. New visits do
  not need appointments, revisits are by
  appointment.
• Most sites have a reception and waiting area
• There is no triage at the reception

4
Results Days and hours of service

• VCT and PMTCT
• Very few sites displayed opening hours and even
  fewer their hours of service
• Inability of providers to distinguish hours of
  service with their working/opening hours of
  clinic
• Variation in true hours of service most sites
  start later than opening hours, but stay until
  all clients are seen even if this means skipping
  their lunch break and staying after hours
• Number of days/week
• Min1 max 5 Average 2
• PMTCT Sites offering service 3 days/wk usually
  see new and returning clients on separate days.

• ART
• Very few sites display their hours of service
• Variation in true hours of service most sites
  start slightly late, but stay until all clients
  are seen even if this means skipping their lunch
  break and staying after hours
• Number of days/week
• Min 1 (1/24) max 5 (22/24) (1/24 unknown)
• New clients generally seen any day returning
  clients seen by appointment unless an emergency.

5
Results visits to the site

• VCT and PMTCT
• Visits past 3 months (majority only 1 time) most
  visits are for supervision some courtesy visits
• about 50 visited by NGO/GF supporting
  institutions and District sanitaire.
• About 35 visited by TRAC
• Majority appreciate the visits but dont like
  that they are unannounced and not coordinated
  with each other
• Minority of sites state that the number of visits
  perturbs their work schedule (receiving clients)
• Majority only receive verbal feedback after these
  visits (never get hard copy results of the
  supervision neither for a single supervision
  visit nor for evolution over several visits)

• ART
• Visits past 3 months (majority 1-2 times) most
  visits are for supervision some courtesy visits
• Nearly all visited by NGO supporting agencies
• 50 visited by TRAC
• 25 by district sanitaire
• Majority appreciate the visits but dont like
  that they are unannounced and not coordinated
• Minority state that the number of visits perturbs
  their work schedule (receiving clients)
• Majority only receive verbal feedback after these
  visits (never get hard copy results of the
  supervision neither for a single supervision
  visit nor for evolution over several visits)

6
Recommendations Part 1

• First and foremost, publicly thank all of the
  service providers for their willingness to serve
  their clients as evidenced by the vast majority
  skipping lunches and working overtime to
  accommodate all clients on the day that they
  arrive.
• PMTCT explore if it would be useful and possible
  to receive all types of clients (new visit,
  revisit, follow-up) all days of service with or
  without initiating a triage at the reception
  (this improves quality by increasing access to
  services)
• VCT explore possibility of initiating a triage
  for couples/partners
• PMTCT and VCT train providers in sites without a
  reception area (typically without group
  counseling also) to code the variable number of
  clients received as not applicable when
  preparing their monthly reports (rather than
  repeating the same number as the variable number
  of clients counseled.
• Standardize the sites so all are registering
  clients BEFORE the group counseling session.

7
Recommendations Part 1 cont.

• Review the MINISANTE norm of 10
  clients/counselor/day is this an absolute/day or
  average/day? Is this only urban and peri-urban
  centers or rural as well? (can this norm work in
  rural areas what are the consequences of
  enforcing? Of changing?)
• Orient all providers to the difference between
  the hours in a workday and hours of (different)
  services and then, based on reality of set-up
  time, provider meetings in morning, etc,
  establish true service hours for each service
  - at each site (may vary) and post these.
• Mobilize resources to build/create a waiting room
  and then standardize that there is (or is not)
  registration group counseling at all sites.
• Install TV and reading material in waiting room
  or a paillote/shelter near the clinic, where
  clients can sit comfortably and wait to be seen
  (and for test results)
• Create a joint supervision/visit plan to
  coordinate and announce visits, conduct together
  those which are similar

8
Part 2

• Completing the
• different data sources

9
Types number of sources of data

• ART
• Due to time constraints and the number and
  variation of data sources, we cannot present the
  same analysis as done for VCT and PMTCT.
• Nonetheless, one can categorize ART sites as
  those that have - or have not - computerized
  their records Those with computerized systems,
  typically have significantly fewer types and
  numbers of data sources used and needed for
  monthly reporting (but still maintain many
  different paper-based types and sources of data)
• We recorded 23 different types of data sources in
  use in Rwanda (but, some differ only in name or
  slightly in content) for reporting, the Min. 3
  and Max 9 (this is not based on complete data,)

• VCT
• 6 different TYPES of data sources in use by sites
  in Rwanda
• Min 3 Max 6
• Majority use 3 or 4 types
• Total of sources at given site
• Min 3 Max 15
• Majority have 4-6 sources.
• PMTCT (only for CPN component)
• 7 different TYPES of data sources in use by sites
  in Rwanda
• Min 3 Max 7
• Majority use 3-6 types
• Total of sources at a given site
• Min 3 Max 12
• Majority have 4-6 sources

10
Different types of data sources bytype of
service
11
Data Maintenance Storage

• VCT PMTCT (PNC only)
• VCT In about ½ the sites, counseling registers
  are maintained by 1 counselor in the other ½,
  they are maintained by several. In sites with a
  reception registry, gt ½ are maintained by a one
  individual
• PMTCT in the vast majority of sites, maintaining
  the reception registry (46/60) and the counseling
  registries (49/60) are shared among several
  people.
• VCT PMTCT approx. ½ of the sites store their
  data sources where maintained at the other
  sites, some or all of the sources are collected
  and stored together in a different location.
• VCT PMTCT in about 2/3 of sites, providers
  feel completing these data sources is easy in
  1/3 they feel it is difficult

• ART
• In 23/24 sites, all or most of the data sources
  are completed by several people (who are doing
  rotation or whom are available..
• 21/24 of the data sources are not moved from the
  locations where they are maintained.
• In approx. ½ of the sites, providers said they
  have difficulty completing the different data
  sources
• Some partners maintain databases but do not give
  access to the databases at the site (they are not
  on-site).

12
Part 3

• Format and content of data sources internal and
  external consistency

13
Format and content of data sources

• VCT PMTCT
• In nearly all sites, the providers must draw the
  columns and rows in each page of their (blank)
  registers and then write in the column headings
  this takes time and leaves opportunity for errors
  (as observed)
• In 2/3 of the sites, each page of a particular
  data source was drawn consistently in the same
  way different data sources of the same type were
  identical (internal consistency). In the other
  1/3, internal inconsistencies were identified
• There is a general consistency between sites in
  the format and content of types of data sources
  used (external consistency). In other words, most
  of the variation is due to a small minority of
  sites being different from the rest in this
  respect.
• Much more variation was noted during the Kigali
  pretest (internal external)

• ART
• In nearly all sites, the providers must draw the
  columns and rows in each page of their (blank)
  registers and then write in the column headings
  sometimes they do not have the rulers to draw the
  lines this takes time and leaves opportunity for
  errors
• Because there is generally only one data source
  for each type, there is good internal consistency
  
• There is much more variation in format and
  content of the different types of data sources
  between sites. (lack of external consistency with
  exception of the individual client record)

14
Part 4

• Preparing
• Monthly reports

15
Completing monthly reports VCT PMTCT

• In the majority of sites, the person responsible
  for the service one of the counselors who
  completes the monthly report (usually the same
  person, there is no rotation nor done as a team)
• The vast majority of sites use more sources of
  data for reporting than necessary typically this
  is using the lab registry (that they do this,
  demonstrates their desire to provide reliable
  data)
• Some sites did not use all sources necessary
  (usually they ignore reception register)
• Perception of providers vast majority dont feel
  they have a problem completing the monthly
  reports with exception of completing the
  variables couples tested/partners tested
  (note that we only explored CPN for PMTCT)
• But, that completing these takes a lot of time
  and not done as a team so others dont understand
  how much time it takes.
• The protocol intended to assess total reporting
  burden but this was not clear. What we got is
  still useful in a minority of sites (less than
  20 VCT and 30 PMTCT) the person responsible for
  completing the (TRAC) monthly report is required
  to complete other reports for that service
  (project/partner specific)

16
Completing monthly reports ART

• In 2/3 of the sites, the person responsible for
  the service completes the monthly report. In 5
  sites, its a different person, typically one of
  the counselors. In 2 sites, the international
  partner agency completes the reports from their
  database
• the majority of sites use many sources and all
  are necessary for the report a minority use less
  than necessary.
• More than ½ of ART providers said they have
  difficulty completing the monthly reports meaning
  it takes a lot of time due to the many different
  data sources and not, necessarily (we did not
  probe) due to lack of understanding of the
  variables.
• It was not percieved as difficult in sites with
  computerized client records although these
  computerized systems do not have a TRAC monthly
  report as standard outputs, most info needed for
  reporting is obtained from these and these
  databases are updated regularly (not just at end
  of month) with data from different sources.
• NB we are not referring to TRACnet which is used
  for transmission of the report after the
  paper-based report is compiled from all the data
  sources.
• The protocol intended to assess total reporting
  burden but this was not clear. What we got is
  still useful in ½ of the ART sites the person
  responsible for completing the (TRAC) monthly
  report is required to complete other ART reports
  (project/partner specific)

17
Recommendations Parts 2,3,4

• To reduce burden of reporting, free time for
  providers to see clients and increase data
  reliability
• VCT and PMTCT, follow model at Dushishoze where
  daily counseling is aggregated in a master
  registry
• PMTCT and VCT, reduce number of individual
  counseling registers by maintaining a register
  for each counseling room (use is shared by
  whomever using that room rather than having a
  unique register for each counselor).
• Share responsibility of completing the monthly
  reports (rotate responsibility or do it together)
  so difficulty of preparing a report with
  incomplete and/or incorrect data is understood by
  all.
• All sites whose partners maintain a database
  should make that available to the sites so site
  personnel can complete their own report.

18
Recommendations Parts 2,3,4 cont.

• Equip the sites with desks or cabinets that have
  drawers with locks to keep their registers
• Explore possibility of consolidating the types
  of data sources especially at ART sites (at some
  sites, providers must use client records for
  monthly reporting.
• For ART, collect the many different types of data
  sources, get stakeholders to review and propose
  standardized for each type of data source
• Standardize the nomenclature of each type of data
  source and print this on the registries.
• Train sites as teams so they better understand
  certain variables such as couples/counseling

19
Example improved register formatting
indicated HIV positive - indicates
HIV negative
20
Recommendations Parts 2,3,4 cont.

• Standardize format and content of the registers
  at all sites PMTCT, VCT et ART (including
  standardizing - and introducing databases)
• Have the registers printed, bound and distributed
  for use.
• A space for page totals should be included
• Each new month should start at top of new page
• Training in use of the registers should be
  conducted with all staff.
• No site should be forced or voluntarily change
  the format and content without some type of
  central level approval being obtained but,
• Leave a limited amount of flexibility in the
  content by leaving some columns (variables) blank
  which could be used for new variables if approval
  to add is obtained
• Standardize the age groups in the registries. If
  a project/agency wants to use a different age
  group, they should get the raw data from TRAC and
  analyze it themselves
• Use both birthdate and AGE in years and
• Create and distribute a table of age by birth
  year to easily determine age update regularly

21
Part 5

• Providing
• Feedback

22
Providing Feedback

• ARV
• We used graphs from TRACNet often the feedback
  session was held with only one person because of
  availability service hours
• Only 4/24 sites prepare their own graphs. That
  is, the vast majority dont use/benefit from
  their own data
• At all sites, this was the 1st time that they
  seen the TRACNet graphs
• Of those to whom we could submit the graphs (11)
  and had discussion, 10 said they appreciated it
  (one did not accept the data because it was
  different than their own)
• 10/11 sites with whom held discussion were
  capable of reading and interpreting the graphs
  without assistance. 1/11 could not even after
  providing assistance.
• For only in ½ of the ART sites did we obtain
  graphs from TRACnet because the ME Unit did not
  have access to it.

• PMTCT and VCT
• We used tables and graphs from TRACs PMTCT and
  VCT databases. On average, 3-4 providers were
  present during the feedback session.
• Less than 20 of the sites prepare their own
  charts/graphs (or even tables). That is, the vast
  majority dont use/benefit from their own data
• This was the 1st time any of these sites had seen
  these tables and graphs.
• nearly 70 with whom we held discussions could
  read and interpreting the graphs without
  explanation/help
• Another 10 could read and interpret these with
  very little help/explanation.
• Thus, only 10 could not use the graphs after a
  brief explanation .(10 unknown because didnt
  have graphs.

NB some site personnel (PMTCT, ART, VCT)
mentioned that they had training from QAP
project so this trained them to read and
interpret the data
23
Providing Feedback cont.

• VCT PMTCT
• Nearly all providers who could read/ interpret
  (with/without help) could also explain the cause
  of changes in the evolution of their indicators
• Nearly all providers appreciated this discussion
  around their data and, at the PMTCT sites, they
  specifically asked this be done regularly
• VCT less than 10 of the providers receive this
  type of feedback
• PMTCT about 20 received this type of feedback
• Among those 10 VCT and 20 PMTCT, sometimes only
  tables of data (no charts) are used and the
  institution providing feedback might not even
  leave a hardcopy
• Sites do not generally see the usefulness of
  these data but do understand that its useful for
  district and national level archives
  prevalence estimates.
• In about 20 of the VCT - and 10 of the PMTCT
  sites, The data from the TRAC database was
  different (discordance) from one of the monthly
  reports they had submitted.

• ARV
• 9/10 sites that could read and interpret a graph
  could explain the evolution of their data
• 10/11 sites appreciated the discussion and
  assistance (1 did not because of data errors in
  graphs)
• Only 4 sites have received feedback on their data
  other than that received at this visit. Of
  those, 2 receive data in tabular form (no
  charts/graphs)
• 5/11 sites there was a discordance between the
  site monthly reports (they kept) and with TRACnet
  outputs (graphs)

24
Part 6

• Data reliability

25
Results Data Reliability

• VCT 1st group variables
• -3 1site
• -2 1 site
• -1 3 sites
• 0 33 sites
• 1 19 sites
• 2 3 sites
• 3 2 sites

• VCT 2nd group variables
• -3 0 sites
• -2 1 site
• -1 5 sites
• 0 27 sites
• 1 10 sites
• 2 7 sites
• 3 3 sites

• PMTCT 1st group variables
• -3 1 site
• -2 1 site
• -1 3 sites
• 0 43 sites
• 1 10 sites
• 2 2 sites
• 3 0 sites

• PMTCT 2nd group variables
• -3 0 sites
• -2 2 sites
• -1 6 sites
• 0 41 sites
• 1 8 sites
• 2 2 sites
• 2 1 site

• ART 1st group variables
• -3 0 site
• -2 0 site
• -1 3 sites
• 0 13 sites
• 1 3 sites
• 2 0 site
• 3 3 sites
• /-3 2 sites

• ART 2nd group variables
• -3 0 site
• -2 1 site
• -1 0 site
• 0 9 sites
• 1 5 sites
• 2 0 site
• 3 5 sites
• /-3 4
  sites

26
Results Reliability cont.

• Data reliability in about ½ of the VCT sites and
  2/3 of the PMTCT sites was excellent based on
  this (small) sample of monthly reports there was
  no error detected between the report we compiled
  from all data sources and the report the site
  compiled and submitted.
• VCT
• most of the error found was in the /-1 range
• There appears to be some systematic errors
  resulting in a tendency to slightly overestimate
  one or more of the variables for both the 1st and
  2nd group
• PMTCT
• most of the error found was in the /-1 range
• There appears to be some systematic errors
  resulting in a tendency to slightly overestimate
  one or more of the variables in the 1st
• There appears to be random error in the 2nd group
• NB this is not to be interpreted statistically
  as done with a quantitative study with on PPS
  sampling.

• Data reliability in 9/24 sites was excellent
  based on this (small) sample of monthly reports
  there was no error detected between the report we
  compiled from all data sources and the report the
  site compiled and submitted
• Of the sites with errors, there were
  proportionally more falling into the /- 3 range
  than with VCT or PMTCT
• There appears to be some systematic errors
  resulting in a tendency to slightly overestimate
  one or more of the variables in the 2nd group
• There appears to be random error in the 1st group
• NB The presence of the /- 3 range indicates
  that one or more of the variables in that group
  were 3 and at the same time one or more
  variables was -3.

27
Explanation of errors

• PMTCT and VCT
• The reasons for the errors are due to a variety
  of factors varying per site
• Arithmetic errors
• Registers (data sources) not completed correctly
  or incompletely (including not having all
  variables.)
• Misunderstanding in how a variables is counted
• Short cuts used in reporting
• To serve clients on saturdays, some clients go
  directly to the laboratory for the test without
  passing through the counseling service (rare 2
  cases)
• In summary if data sources are simplified,
  standardized and pre-printed, nearly all sources
  of error will be eliminated and data reliability
  will be better,

• ART
• The reasons for the errors are due to a variety
  of factors varying per site
• Arithmetic errors
• Registers (data sources) not completed correctly
  or incompletely (including not having all
  variables.)

28
Recommendations Part 5 6

• Given that most of the providers with whom we
  shared information in the form of graphs are
  already capable of reading and interpreting
  graphs, supervisors at TRAC, the district and
  partner institutions should integrate and
  mainstream reviewing data from a site as a
  component of performance- based, quality
  assurance in their regularly scheduled
  supervision visits.
• To do this, we need to train supervisors in the
  use and explanation of data
• Supervisors should work with ME teams to prepare
  for their visits the ME team can help prepare
  the outputs for feedback discussion sessions and
  help them review and interpret the graphs before
  the supervisors go out.
• Train the (few) providers who cannot read
  interpret graphs and train as teams
• All supervision visits should be coordinated and
  bring copies of graphs to sites if not mailed or
  available through internet (for those with
  connections. And budget for them to be
  distributed in color (or rework the color and
  patterns of graphical outputs so they are easily
  read in black and white.
• Give more people access to TRACnet
• Improve the verification at TRAC when entering
  data from monthly reports (at both primary
  data-entry and verification after entry
• Improve the data-entry and verification within
  TRACnet
• Verify that there are no programming errors in
  TRAC databases/TRACNet which might explain these
  errors.