Diapositiva 1

1
European Conference on Quality in Official
Statistics - Q2008
Participants to research group De Vitiis, DAlò,
Di Consiglio, P.D. Falorsi (chief), S. Falorsi,
Orsini, Pallara, Russo, Seeber, Tuoto
Sampling and non sampling errors in the Italian
Television Audience Measurement system
Speaker Alessandro Pallara Istituto Nazionale
di Statistica
Rome, 9-11 July 2008
2
Outline of the talk

• Television Audience Measurement (TAM) and the
  meter panel
• Survey parameters and sampling design
• Estimation of sampling error
• Sources of bias in TAM estimates
• Measurement errors EI
• Panel attrition and conditioning
• Comments and concluding remarks

European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
3
Television Audience Measurement

• Television Audience Measurement (TAM) data have
  a high social and economic impact.
• Essential information to
• Broadcasters, for programming policy and
  programme scheduling
• Broadcasters and advertising agencies, for
  agreeing upon the price of commercial air-time
  and advertising campaigns

European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
4
Context and purposes of the research
The context for this research is the agreement
signed in 2006 between Italian NSI and the
Italian Communications Regulatory Authority
(Agcom), so that Istat has been appointed for
carrying out a study on the statistical
methodology behind the national TAM system.

• Purposes (and Research reports)
• review current estimation procedures for
  estimating daily ratings and associated sampling
  errors (released June 07)
• 2) evaluate accuracy of the survey estimates
  with respect to the various sources of non
  sampling errors (Dec. 07)
• 3) put forward tools and recommendations for
  checking statistical quality (both sampling and
  non sampling errors) of the output of TAM survey
  (under release, July 08)

European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
5
Standard TAM methodology

• Current worldwide standard in TAM methodology has
  two basic features
• a viewing household panel sample (the People
  meter panel) selected according to certain
  household demographic characteristics (age of the
  householder, number of components, city size,
  geographical region)
• a measurement device (the people meter) that
  register (a) TV set status (i.e. which channel is
  being tuned to with certainty) and (b) viewer
  presence, which is quite demanding on panelists
  (i.e. pressing their remote control button each
  time they enter or leave a television viewing
  session)

European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
6
Survey Parameters
Let r denotes a generic TV channel and T a
given time interval (daypart, day, week) Main
Parameters
The Audience is the average number of individuals
(homes or target groups) viewing a TV channel
over a given time interval (e.g. programme,
daypart).
European Conference on Quality in Official
Statistics - Q2008
The Reach (or cover/cume) is the cumulative
percentage or total (usually expressed in
thousands) of a population that has been counted
as viewers at least once during a specified
interval.
Rome, 10 July 2008
7
Survey Parameters (cont.ed)
The Share (of Audience) is defined as the percent
of Households Using Television (HUT) or Persons
Viewing Television (PVT) which are tuned to a
specific program or station at a specific time.
European Conference on Quality in Official
Statistics - Q2008
The Rating is the size of television audience
relative to the total universe, expressed as a
percentage
Rome, 10 July 2008
8
Survey population, statistical units, data
analyzed

• Survey population members of household aged 4
  or more
• Survey estimates refer to in-home TV viewing
  (persons and households, including viewing of
  guests of the sample households), of total
  population and selected target subpopulations
• Elementary data used for estimating parameters
• Individual viewing statement meter records (raw
  data) converted after data processing into
  summary statements of individual viewing over
  time (each minute). Each statement contains
  information concerning (a) Start and end time of
  the viewing session (b) identification of signal
  source and TV set being viewed (c) identity of
  viewer
• Data analyzed
• Raw and validated panel meter micro-data (daily
  data for 4 weeks between Sept. 05 through June
  06)
• Population total of auxiliary variables and
  sampling weights used in the estimation procedure

European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
9
Sampling design of most TAM survey

• Two phases in TAM sampling strategy
• In the first phase, a face-to-face interview (the
  Establishment Survey, ES) is carried out each
  year, based (in 2006) on a sample of
  approximately 30,000 households and using a
  two-stage stratified sample
• ? provides certain universe estimates (in terms
  both of individuals and household) which will be
  used in the TAM estimation procedure, such as
  education attainment, socio-economic status or
  number of children per household,
• ? provides a database of potential households
  for recruitment in the second phase sampling
• In the second phase a panel of about 5100
  households, is broadly randomly selected
  (within control strata) from ES respondents (the
  people meter panel sample).

European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
10
The Meter panel sample
Meter panel selected characteristics (used
for panel turnover control), Active vs. Lost
Panelists, Compared to Total population benchmarks
European Conference on Quality in Official
Statistics - Q2008
Estimated through ES
Rome, 10 July 2008
11
Problems with TAM sampling design in Italy

• quota sampling
• unknown selection probability of units from the
  recruitment households database (originating
  from different ESs)
• rules for field substitution of non responding
  households different contact rates between basic
  households and substitutes, interviewer may
  influence substitutions
• very high total (non response to ES refusal to
  panel recruitment) non response rate gt90
• Non respondents may be different as for the
  amount of television viewed light viewers out of
  home a lot and less available for interview,
  light viewers may feel their cooperation less
  important

European Conference on Quality in Official
Statistics - Q2008
.
Rome, 10 July 2008
12
Approach to measuring accuracy of TAM estimates

• MSE of an estimator of an unknown
  population parameter
• Approach to quality assessment
• Direct (smooth) estimators of the sampling
  variance
• (Indirect) indicators of the Bias

European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
13
Estimation of sampling variance
European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
14
Variance estimation an example
Estimates of Audience (each minute) and
Coefficient of Variation (CV) for a large channel
of the public network 4 Sept., 2005
European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
15
Variance estimation an example (cont.d)
Scatter plot of CV by Audience Size (minutes
and dayparts)
European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
16
Sources of bias in TAM estimates

• Potential sources of bias in meter panel sample
• coverage errors (e.g. non-TV homes not included
  in estimates, 1,500,000 est. persons in Italy)
• (wave) non responses
• model assumptions errors
• measurement errors
• attrition and panel conditioning

European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
17
Measurement Errors in meter panel data

• measurement errors mis-match between a signal
  source of a TV set being viewed and a person
  registered as a viewer through the people meter

• Main sources of measurement errors (data
  gathering and editing phases)
• Meter statements indicating that the TV set is
  switched on, but without any persons registered
  as present (uncovered viewing).
• Long viewing session without any change in
  registered set use or viewer presence (signing
  on/off of viewing individuals, channel switching,
  long/constant viewing)
• TV OFF viewing
• Same individual registered as a viewer for two or
  more TV sets at the same time (concurrent
  viewing)
• Undue or wrong re-assignment of uncovered viewing
  to a household component (processing errors)

European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
18
Processing TAM data EI

• Editing checks
• rejection of certain panel households from the
  daily reporting samples because of suspected
  faulty compliance by panelists excess (24 hours)
  viewing, long/constant viewing above set
  threshold values
• records of individual viewing are canceled out
  (concurrent viewing, overnight constant viewing,
  unassigned uncovered viewing)
• records of individual viewing are edited in
  (uncovered viewing assigned to viewer)

European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
19

Processing data - Editing and Imputation
Percent variation of audience estimates
(unweighted) from raw to validated data resulting
from treatment of uncovered viewing
European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
20
Processing data - Editing and Imputation
Percent variation of audience estimates using
different cut-off values and criteria for
deletion of records with long constant viewing
European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
21
Panel attrition
European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
22
Attrition rates by subgroup of population
European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
23

• Panel attrition and conditioning

Months-in-sample percent distribution of
households panel sample
European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
24
Age effects
Daily estimates of audience (thous. of
individuals) of satellite TV channels by some
dayparts (4 weeks between Sept. 05 through June
06) households below and above median of
time-in-sample
European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
25
Comments and concluding remarks - 1

• Sampling errors
• The CV decreases as the estimation increases.
• The larger estimates (major networks) quite
  reliable
• The lower estimates (local networks) quite
  unreliable
• The CV slowly decreases as the size of time
  interval of estimates increases

• Non sampling errors
• coverage errors related to list problems (non-TV
  homes, non-voting resident households, ..)
• non standardized criteria for substitution of
  non responding households to ES may lead to
  select in the panel heavy viewer households
• some evidences of the presence of an upper bias
  in survey estimates editing checks seems to be
  unbalanced towards editing viewing statements in
  rather than out, threshold values for considering
  long viewing as unrealistic result in canceling
  out viewing statements only in the case of
  overnight viewing
• the lack of an upper limit to time-in-sample for
  households in the panel suggests the presence of
  panel attrition and conditioning because of
  modifications in panelist viewing behavior and
  compliance with the measurement device during
  their presence in sample

European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
26
Comments and concluding remarks 2

• Recommendations for Improving quality
• coincidental surveys on a regular basis to check
  real viewing status of panelists vs. registered
  meter data
• occasional surveys of non respondents to analyze
  independence of response mechanism from viewing
  behavior
• introducing a method for panel rotation, with an
  upper limit to time-in-sample of panel households

European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008
27

• Thank you for your attention!

European Conference on Quality in Official
Statistics - Q2008
Rome, 10 July 2008