Olav ten Bosch MSIS, Dublin, April 2014 On the use of internet robots for official statistics

Overview – Why internet as a data source (IAD)? – Internet robots, how do they work? – Applications: Airline tickets Housing market Clothing Robot assisted data collection – Conclusion

Why IAD? (1) Administrative sources – Tax, social security services – Municipalities/ Provinces – Supermarkets Surveys Internet sources Less!!! Faster, better, more efficient New indicators

4

Which content is original, reliable, stable, representative and accessible? Internet sources Why IAD? (2) – Internet prices for CPI ? – Real estate sites for housing statistics ? – Internet vacancies for job statistics ? – Social media sentiment for consumer confidence ? – Trade in second-hand goods as economic indicators ? – Travel activity for tourism statistics ?

Robots / crawlers / bots / spiders / scrapers: how do they work? (1) Browser Website Internet Requests code, images, style, data, etc. Graphical markup You Commands

Robots / crawlers / bots / spiders / scrapers: how do they work? (2) Robot/ spider/ crawler Website Internet Requests Navigation code, images, style, data, etc. Data You

Robots / crawlers / bots / spiders / scrapers: how do they work? (3) Robot/ spider/ crawler Website Internet Requests Navigation code, images, style, data, etc. Data Monitor actively Generic software for: - site navigation - product details - monitoring Data

Airline tickets (1) Robot collection versus manual collection

Airline tickets (2) Price of a ticket over time

Housing Market (1)

Housing market (2) Dynamics of the database behind becomes visible

Clothing (1):

2 sites: very volatile data Clothing (2): Challenges: -from volatile data to stable statistics -how to classify multiple less structured data sources Seasonal pattern

Robot-assisted data collection (1) – Use case: few price observations on many sites – Example: price of a cinema ticket – Robot tool to automatically check if prices are changed

Robot-assisted data collection (2) 16

Conclusion – Using internet as a datasource we can measure statistical phenomena in a completely different way – It is powerful to combine fast internet data with reliable (but slower) administrative data – We should redesign statistics with the possibilities of internet data in mind Challenges: – Legal framework – The internet changes continuously: how to turn volatile data sources into reliable statistics? – We need advanced statistical methods, processes and IT