NATIONAL STATISTICS OFFICES AND THE PROSUMER CHALLENGE New Techniques and Technologies for Statistics (NTTS) Seminar Brussels, February 2009 Space-Time Research John Ellenberger

Competing forces – data availability 1980’s1990’s2000’s Privacy protection Revenue earning Error prevention Statistical purity Less data available Govt. policy Transparency ROI for statistics User expectations Technology Information explosion More data available Statistical tug-of-war 2010 ?

The information revolution 1980’s1990’s2000’s Privacy protection Revenue earning Error prevention Statistical purity Less data available Govt. policy Transparency ROI for statistics User expectations Technology Information explosion More data available 2010 ?

Where do people look for information? 10,403,951 July Estimate - Total Population? - Include foreign workers? - Guests, tourists etc? Different users are interested in different things….. 10,403,951 July Estimate - Total Population? - Include foreign workers? - Guests, tourists etc? Different users are interested in different things…..

What do users want? Coverage TimelinessRobustness National Statistics Office Users of Statistics

The perfect survey? Doctor, that was a perfect operation…. Thank you nurse…. my work is done. Pity the patient died…

The Unbiased Statistician I never comment on the results, just report the facts. By collecting certain information you bias what is monitored. Is GDP a better measure of a governments performance than citizen happiness or well being? By only releasing some of the information you choose what is available for analysis. Aggregated output is a small fraction of what is collected. This can be addressed

User Personas – who are Prosumers? Internal users External users Statisticians Subject Matter Experts Public Looking for general information Analysts and Statisticians from Business & Government Knowledge workers Use data as part of their decision making process

Producers and consumers Internal users External users Information Consumers Information Producers Statisticians Subject Matter Experts Prosumer Public Looking for general information Analysts and Statisticians from Business & Government Knowledge workers Use data as part of their decision making process

Tools for different users Internal users External users Information Consumers Information Producers Statisticians Subject Matter Experts Self-ServiceAnalytics Self-ServiceVisualisations Self-Service BI Prosumer Public Looking for general information Analysts and Statisticians from Business & Government Knowledge workers Use data as part of their decision making process

Prosumers tell stories for others If you are going to travel on the Titanic….. Always travel 1 st class, be young and be female….. Whatever you do, don’t be a 2 nd class, male, adult passenger… If you are going to travel on the Titanic….. Always travel 1 st class, be young and be female….. Whatever you do, don’t be a 2 nd class, male, adult passenger…

Prosumers generate debate...

Needs What Prosumers want Ability to ask any question All variables All level of detail Easy to navigate interfaces. They are not necessarily experts High speed access Extract the data for further analysis No cost Page 13NTTS Feb 2009National Statistics Offices and the Prosumer Challenge What Statisticians must do Preserve confidentiality Protect the user from mistakes (a duty of care) Reduce costs

Can everyone be satisfied? Easy way to navigate micro data Topic Base Full micro database Fast aggregation engine Cope with large tables (1,000,000 cells) Efficient download SDMX In built duty of care On the fly confidentiality Suppression Modification/obfuscation Delivery through the web Marginal cost = zero Page 14NTTS Feb 2009National Statistics Offices and the Prosumer Challenge

Architecture diagram – servicing a Prosumer Page 15NTTS Feb 2009National Statistics Offices and the Prosumer Challenge

Topic based or full microdata Page 16NTTS Feb 2009National Statistics Offices and the Prosumer Challenge Topic Microdata Full Microdata

Choose any variables, group as desired Page 17NTTS Feb 2009National Statistics Offices and the Prosumer Challenge

Run tabulation Page 18NTTS Feb 2009National Statistics Offices and the Prosumer Challenge

On the fly confidentialisation - disturbance Page 19NTTS Feb 2009National Statistics Offices and the Prosumer Challenge

On the fly confidentialisation - suppression Page 20NTTS Feb 2009National Statistics Offices and the Prosumer Challenge

A duty of care – simple example Page 21NTTS Feb 2009National Statistics Offices and the Prosumer Challenge A total of 5,759 Separate Dwellings - not 10,862 5,103 Separate Dwellings that contain a male Counting dwellings

Big tables…. Big downloads Page 22NTTS Feb 2009National Statistics Offices and the Prosumer Challenge

Big tables…. Big downloads Page 23NTTS Feb 2009National Statistics Offices and the Prosumer Challenge

The will to service Prosumers The technology is available to make microdata available to end users safely and affordably over the Internet…. …. For those willing to ‘have a go’ Page 24NTTS Feb 2009National Statistics Offices and the Prosumer Challenge

Merci! Danke! Grazie! Thank you…