NATIONAL STATISTICS OFFICES AND THE PROSUMER CHALLENGE

1
NATIONAL STATISTICS OFFICES AND THE PROSUMER
CHALLENGE

• New Techniques and Technologies for Statistics
  (NTTS) Seminar
• Brussels, 18-29 February 2009
• Space-Time Research
• John Ellenberger

2
Competing forces data availability
Statistical tug-of-war
3
The information revolution
4
Where do people look for information?

• 10,403,951 July Estimate
• Total Population?
• Include foreign workers?
• Guests, tourists etc?
• Different users are interested in different
  things..

5
What do users want?
Coverage
Timeliness
Robustness
6
The perfect survey?
Thank you nurse. my work is done.
Doctor, that was a perfect operation.
Pity the patient died
7
The Unbiased Statistician

• By collecting certain information you bias what
  is monitored.
• Is GDP a better measure of a governments
  performance than citizen happiness or well being?
• By only releasing some of the information you
  choose what is available for analysis.
• Aggregated output is a small fraction of what is
  collected.

This can be addressed
8
User Personas who are Prosumers?
Statisticians
Internal users
Subject Matter Experts
Analysts and Statisticians from Business
Government
External users
Knowledge workers Use data as part of their
decision making process
Public Looking for general information
9
Producers and consumers
Information Producers
Statisticians
Internal users
Subject Matter Experts
Analysts and Statisticians from Business
Government
Prosumer
External users
Knowledge workers Use data as part of their
decision making process
Information Consumers
Public Looking for general information
10
Tools for different users
Self-Service Analytics
Information Producers
Statisticians
Internal users
Subject Matter Experts
Self-Service BI
Analysts and Statisticians from Business
Government
Prosumer
External users
Knowledge workers Use data as part of their
decision making process
Self-Service Visualisations
Information Consumers
Public Looking for general information
11
Prosumers tell stories for others
If you are going to travel on the
Titanic.. Always travel 1st class, be young and
be female.. Whatever you do, dont be a 2nd
class, male, adult passenger
12
Prosumers generate debate...
13
Needs

• What Prosumers want
• Ability to ask any question
• All variables
• All level of detail
• Easy to navigate interfaces. They are not
  necessarily experts
• High speed access
• Extract the data for further analysis
• No cost

• What Statisticians must do
• Preserve confidentiality
• Protect the user from mistakes (a duty of care)
• Reduce costs

14
Can everyone be satisfied?

• Easy way to navigate micro data
• Topic Base
• Full micro database
• Fast aggregation engine
• Cope with large tables (1,000,000 cells)
• Efficient download
• SDMX
• In built duty of care
• On the fly confidentiality
• Suppression
• Modification/obfuscation
• Delivery through the web
• Marginal cost zero

15
Architecture diagram servicing a Prosumer
16
Topic based or full microdata
17
Choose any variables, group as desired
18
Run tabulation
19
On the fly confidentialisation - disturbance
20
On the fly confidentialisation - suppression
21
A duty of care simple example
22
Big tables. Big downloads
23
Big tables. Big downloads
24
The will to service Prosumers

• The technology is available to make microdata
  available to end users safely and affordably over
  the Internet.
• . For those willing to have a go

25
Merci!Danke!Grazie!Thank you