COI and the role for evaluation

1
(No Transcript)
2
COI and the role for evaluation
Presented by Marc Michaels, Director of Direct
and Relationship Marketing
3
What is the COI?

• The Central Office of Information (COI) is the
  Government's centre of excellence for marketing
  and communications
• Established in 1946 after the wartime Ministry of
  Information
• COI works with government departments and the
  public sector to produce information campaigns on
  issues that affect the lives of every citizen
• Combining expertise in marketing communications
  with an understanding of government systems and
  procurement policies
• COI provides consultancy, procurement, project
  management and evaluation in every area of
  communications and strives to achieve best value
  for government
• COI's Chief Executive Mark Lund reports to the
  Cabinet Office. COI is given annual ministerial
  targets to achieve

4
Who are the COI?
Client
Our current structure
New and PR
New and PR
News and PR
News and PR
NDS
MMU
Merchandising
Channel Integration Management
Interruptive
Direct and Relationship Marketing
5
What does the COI do?

• Most of the time we are in the game of targeting
  specific groups of people to get them TO DO or
  NOT DO things e.g.
• give up smoking
• dont take drugs
• eat more fruit
• pay tax on time
• take out a pension
• get actively involved in their children's
  education
• have career in the Forces, teaching, police, NHS
  etc.
• Spend time researching, profiling and pinpointing
  ways of reaching, acquiring, handling, and
  retaining them using the optimum combination of
  media tools at our disposal
• But how do we know it all works?

6
The need for evaluation

• Phillis Review, Engage, procurement initiatives
  and greater scrutiny of marketing budget gt need
  to build business cases and demonstrate positive
  ROI.
• After all, this is taxpayers money!
• Often difficult in the arena of Government
  communications as we are looking across a
  customer journey usually with a behavioural
  change outcome not sales.
• gt need for an independent, impartial centralised
  tool to help collate and benchmark and share
  information and findings in a commonly accessible
  language that is appropriate to Government

7
Part of our response COI Artemis

• Still a relatively new service (launched March
  07) designed to enable evidential based planning
• Response and conversion analysis by audience,
  media channel, vehicle of response, viewed
  alongside awareness
• Help facilitate better campaign planning
• - setting realistic objectives
• - task based budget setting
• - helping improve plans
• Forecasting likely demand for contact
  centres/websites
• Most importantly - an historical repository for
  the results of government campaign activity for
  shared learning and benchmarking

8
COI Artemis getting everything together in one
place
COI RESEARCH TRACKING DATA Direct link in from
COI research system.
EXTERNAL SOURCES Time series data (e.g. proxies)
to look at possible correlations
ABOVE THE LINE MEDIA COSTS VOLUMES Loaded
directly from the DDS system e.g. TV costs and
ratings
RESPONSE DATA FROM COI MANAGED CONTACT
CENTRES Time, data, type (interactive,
non-interactive) Channel (call, web etc), Media
(where captured) other data as required
BELOW THE LINE MEDIA COSTS VOLUMES Captured by
individual COI project Managers (e.g. DRM) to
give an equivalent media cost and volumes for
non-advertising activity such as Direct Mail,
outbound Telemarketing etc.
RESPONSE DATA FROM NON-COI MANAGED CONTACT
CENTRES As above, though harder to obtain /
specify as not directly under control.
COI ARTEMIS
DOOR DROPS COSTS VOLUMES Can be passed on
directly from the new DRM aggregated system at
the Leaflet Company

• CONVERSION DATA
• This is going to come from a variety of sources
  including
• ) CONTACT CENTRES who handle follow up data
• ) CENTRAL GOVT DEPT STATISTICS e.g. Army walk
  ins, recruits, people signed up to initiative
• ) ONS DATA e.g. smoking cessation figures and it
  will not be in a standard format. However it is
  likely to be very simple data sets of raw numbers
  over time.

• DIGITAL MEDIA response data.
• Traffic through websites either from
• ) COI central contact with I Level
• ) Other Department run individual campaign
  websites
• ) DirectGov/BusinessLink
• INTERACTIVE TV data
• Visits and responses (orders)

SPONSORSHIP/PR data equivalent Media values of PR
or Sponsorship and PR GNN
DIGITAL MEDIA COSTS VOLUMES can be passed on
directly from the aggregated Digital media
system with I level
COI ARTEMIS Graphical representation of feeds
from various sources that could be brought to
bear on understanding campaign costs, volumes and
success in terms of awareness, response and
conversion produced by Marc Michaels. Version 2
22nd Feb 06
9
A key element COI Artemis response hierarchy
the funnel
Meeting Public Sector needs

• The public sector is unique - not always
  comparable to commercial sector
• Looking across a customer journey usually with a
  behavioural change
• Not all actions along the journey are of equal
  value
• Media and channels vary in responsiveness and in
  their ability to convert against the right often
  hard to reach audiences
• Long lead times
• Balance campaign optimisation with social
  inclusivity
• Standardising the vocabulary
• Each stage in the funnel represents a step in the
  customer journey
• Allows for cross-campaign and cross-client
  comparison

10
COI Artemis a practical tool yielding many
benefits

• 43 end of campaign analyses already in the pool
  including
• Royal Navy 2008/9
• Royal Marines 2008/9
• Harmful Drinkers East Midlands Pilot 2009
• Hepatitis C 2009
• Tobacco Control Jan March 09
• DH Change for Life (Reframing the Issue) 09
• DH Change for Life (How are the kids?) 09
• HSE Agricultural Workers
• Social Care 09
• Tobacco Healthcare Professionals 09
• Learndirect (Summer 08, Autumn 08 and January 09)
  
• FRANK 2008/9
• FSA MoneyMadeClear Interim Evaluation 09
• gt we have shown that applying learning from
  COI Artemis to subsequent phases or to similar
  projects (by topic or audience) we are

11
COI Artemis a practical tool yielding many
benefits

• Providing a clear evidence base of what is
  working against previous phases and/or a pool
  of similar activity benchmarking norms
• Making recommendations as to what needs improving
  (creative, message, call to action etc)
• Providing more accurate forecasting of likely
  demand/response using historical data helping
  resource allocation and manage expectations
• Helping guide media deployment (evidence based
  scenario planning), improve accountability and
  rational for budget and objective setting and
  channel strategy to focus on the best performing
  media
• Reducing cost per acquisition/conversion
• Highlighting targeting opportunities and
  encouraging testing
• Understanding who is responding by incorporating
  Acorn/Health Acorn profiling
• Improving the customer journey by understanding
  dropout
• Understanding costs/investment needed

12
We are establishing rules of thumb for
relationships between reach response and
conversion for forecasting

• Straight averages can be misleading and this
  picture is changing all the time. Starting to
  look at ranges around the mean.
• Brief rule of thumb for every 100 people who
  come through, 74 will be valid, of those 62 will
  get some information, of those 19 may become
  active and 5 of those take a step towards
  conversion.
• Will vary by campaign and by genre and can be
  particularly sensitive to the media channels used
  and the vehicle of conversion (e.g. phone
  converts better than web).

13
We can group campaigns and look at the funnel
results by genre

• Recruitment generates a lot of interest
  However the response funnel narrows considerably
  and by the interview appointment stage
  (Intermediate conversion) Recruitment slips to
  last place
• Tobacco sees the best transition to Intermediate
  conversion (joining one of the schemes LSSS or
  Together) however C4L has beaten this with a
  one stage process.

14
We are building a picture of overall media
performance across the piece

• In general, online sources deliver excellent
  Total CPR, however TV DM deliver a better
  conversion rates bringing Active CPR into a
  similar range as Online

15
We are noting the impact of creative
A poorly performing Maltese Cross format door
drop excluded
16
HOOKED CAMPAIGN Highest spontaneous awareness
BUT
Lowest prompt to action
We can track campaign activity against awareness,
claimed behaviour or other research tracking
17
We can track changes in usage of vehicle of
response
Note when Freeview responses were included this
became as high as web visits and wasnt being
tracked!
18
and the ability of that vehicle to convert
through to positive action
of total response (web or calls) proportion
of web visits resulting in active
response/intermediate conversion is low c.f. calls

• High volume of web responses ? low Total/Valid
  CPR
• Web does not convert ? high Active/Intermediate
  Conversion CPR. In this instance the web
  tells but doesnt sell

19
We are noting the drop out between different
levels of response of response between different
phases of campaigns/creative
96
97
98
10
11
11
6
7
11

• Are people more likely to take action in January
  (New Years Resolutions), whilst in June they
  were gathering information in preparation for
  legislation change?

20
We can avoid expensive mistakes
Data not being captured correctly on website made
online look very bad indeed
Send Off campaign performing poorly could have
been ascribed to poor creative but actually a
function of the media it ran in
21
by making sure were in the right media.
Send Off (4 Jun 16 Jul 07) 1.6mil
Hook (01 Jan 28 Feb 07) 4.1mil
Send Off (01 Jan 28 Feb 07) 1.4mil

• Send Off on TV was much more effective than the
  January activity, and generated a much greater
  volume of response and conversion than when on
  press, radio and outdoor

22
We can take the evaluation down to a granular
level individual stations and spots

• Satellite channels deliver most cost effective
  CPR for tobacco campaigns, information which is
  looped back into planning process

23
Weve started creating dashboards for continuous
evaluation/monitoring and easier scenario planning
24
By applying Time Series Regression we can split
out the drivers of web visits where we have
little other data
Base
Online
TV
Press

• Web visits are often allocated to Online media
  This over states the effectiveness of online
  media. TV and Press now get there fair share.

25
We can see whether the campaign brought in the
right target audience
Responders
Strivers
Inner Health
Outer Health
Correlation Coefficient
0.9139
0.9340
0.9599

• We generally target lower socio economic groups,
  BUT we do index well against Educated Urbanites
  and Prosperous Professionals in campaigns. Are
  they just more prone to respond or is our
  advertising sometimes a bit too subtle/middle
  class

26
We can see who are the most responsive people
and look at regional variations

• In this instance, parents of younger children
  appear to be most responsive to the Childcare
  mailing and London has the best performance.

27
We can make projectionsto help set
targets/expectationsfor new activity
Source COI Artemisresponse and CRM
sign-upprojections for Obesity campaign based on
pool of similar campaigns
28
We can highlight potential media synergy effects
Tobacco Door Drop Areas covered by door drop
showed a 50 higher active response rate compared
to regions not covered

• For Childcare phase 2 we saw the same thing
  Response Rates are higher in dropped areas
  (average Total RR 0.0627) than areas without
  activity (Total RR 0.0454).

29
We can look for correlations between
different/external data sets to establish
relationships

• Continues to be a very strong relationship
  between the number of NRT products prescribed and
  response via the phone or web.
• Indicates that campaign response may be a
  significant factor in encouraging people to take
  action to quit smoking, as well as just
  generating response to our mechanisms.

30
Case Study DCSF Childcare
2nd phase media plan on DCFS childcare using 1st
phase Artemis findings delivered more efficient
response at all levels and efficiencies generated
savings of 428k at active level. (Using Burst 1
media plan, would have needed to spend extra
827k to generate the total response in phase 2)
Artemis improved the customer journey too.
Automated phone message was too long gt high
levels of drop-out, difficulties capturing media
without interaction and problems with CIS not
accepting transfers. Proposed using Live Agent
interactive phoneline 2nd phase greater
investment but gt 57 of all callers were
transferred to their CIS as opposed to 39
Mixing Artemis findings with follow up research
we tracked respondents to full conversion
average cost 115 to get someone to take up
formal childcare.
31
Case Study JCP to lone parents

• COI Artemis was used to identify the key drivers
  of responsiveness in Q1 2007 JCP campaign gt a
  segmentation which ranked lone parents by
  responsiveness and applied to target the next
  round of activity length of unemployment was
  key factor, not age of children as had been
  thought
• In Q3 attention was switched to understanding
  media synergies Regional testing programme
  combined with media matching capability
  established a media hierarchy with Direct mail
  proving most cost effective whilst uplift from
  radio was positive but not sufficient to pay for
  itself
• Recommendations from this approach identified
  savings of 25 (c. 200k)

32
Case Study Change4Life (How are the Kids?)
COI Artemis was used to forecast response levels
for our integrated How are the Kids?
Campaign. Using targeted media, we focussed on
the high risk cluster groups i.e. Clusters 1, 2,
3 and 5
Clusters 5 (Traditional) highest responders -
they dont encourage fussy eating and are more
receptive to advice
but segment 3 (Treaters) did not engage so
well.
232,000 people signed up to C4L CRM during the
peak campaign period. The original forecasted
volume (across all channels) was 111,797 partly
because of success of postal mechanism (72), the
piggy back and direct media and the survey opt-in
design.
Best performing media - Healthy Start piggy back
and Fruit and Veg Boxes for Schools
33
What next plans and challenges

• Incorporate research data on more consistent
  basis accounting for people who do not respond
  through central channels but are affected by the
  campaign becoming a core part of the building
  blocks of holistic evaluation.
• Look at multi-media uplift and synergy not just
  last media quoted time series regression,
  econometrics.
• Incorporate PR and Search uplifts and offline
  effect on online.

34
What next plans and challenges

• Developing a pan-Govt. evaluation capability to
  define, manage and coordinate holistic evaluation
  solutions.
• Drawing together evaluation expertise from across
  the Depts, COI and agency partners.
• Integrating output from all evaluation strands
  into one place
• Delivering a complete perspective on campaign
  performance.

35

• QUESTIONS?