datalibweb – Stata module to access micro data The World Bank The Poverty and Equity Global Practice Global Solutions Group on Welfare Measurement and Statistical Capacity July 2017

Motivation Improve access to data databases version control Organize micro-data in a logical and structured way version control Keep track of corrections, modifications or updates of the micro-data Ability to replicate numbers calculated with different versions from the current one. knowledge sharing Ensure knowledge is sharing and results can be replicated users working with micro-data at the same time Ensure that all the users are using the same data source

What is datalibweb? It is a global data system and improved version of datalib/datalib2 WEB: User registration and subscription and Management System, Web API Stata: Core encrypted functions to authenticate registered WB users, identifies and grants access to the microdata from the server based on users’ profiles of survey subscriptions Stata: Support to modular plugins (i.e. other related functions to process the retrieved data within user’s local machine [such as metadata on the surveys, convert to PPP, merge with different modules and raw data, etc])

Benefits of datalibweb Accessibility All users have access to the same micro-data and use by default the latest versions When available, quick access to metadata such as questionnaires, interviewers manual, reports, among others (through Microdata Library) Institutional memory Ensures that all results can be replicated Saves hard-drive space Nobody needs to have the micro-data on their computer’s hard-drive It is easy for quality control programing All do-files that call –datalibweb- might be shared and run among users directly No changes or modifications needed No micro-data is lost Faster communication between different units within the WB

Benefits of datalibweb It is IT-independent IT does not have to provide permissions to new users. Users need to register and subscribe to raw/original and harmonized collections they want (one time) Increase level of security Users can access the microdata based on their registration and subscriptions to surveys from multiple regional and GP network drives (must access through Stata) It works in a modular way Each region, Global Practice is able to contribute their own collections/modules that adapt to specific needs of use Easy management and reporting Better management of users (monitoring the usages, and informing users about new and updated database). Better integration of the system with other Bank applications/databases such as Microdata Library, the IW portal Regions and GPs maintain their own collection of databases Each region maintains and curates their databases, and ensure that users always get the latest data (real time).

Adoption of datalibweb is growing

Adoption of datalibweb is growing

How it works Go to FURL: datalibweb Follow instruction to install the Datalibweb package Subscribe to the surveys/collections you want to have access to. Poverty GP colleagues by default have access to the Global Poverty Working Group Database (GPWG-DB). GPWG-DB is the first global database of 1100 microdata that was used to produce the international poverty indicators and Shared Prosperity, it is maintained by the GPWG with contributions. Raw/original and harmonized data: Currently we have about 18 collections (both raw and harmonized), maintained by each regional statistical teams and the global team. Highly interactive (no code) or deeply nested code Users can use interactive interface with adaptive guided information or go straight to the code. Working with vintages/version controls Better control of vintages for users based on unique Survey Identifier. Survey identification enables the user to call a specific version or vintage of the survey of choice. For example, TZA_2011_HBS_v01_M refers to “Tanzania - Household Budget Survey 2011“

Installing datalibweb Manual installation: Get the file from this link: http://eca/povdata/datalibweb/_ado/datalibweb.zip  Copy with replacement all the files into c:/ado/plus, without changing the folder structure Enter this line in Stata net install datalibweb, all replace force from("http://eca/povdata/datalibweb/_ado/") datalibweb, update(ado)

Adaptive guided information for datalibweb users

datalibweb users can also use script to access data Typing multiple entries in “country” option produces an appended dataset of selected countries and year(s) of the chosen collection. datalibweb, country(ALB ARM) year(2008) type(GPWG) clear Survey ID: ALB_2008_LSMS_v01_M Title: Living Standards Measurement Survey 2008 Link to DDI: Survey metadata (microdata portal)   Survey ID: ARM_2008_ILCS_v01_M Title: Integrated Living Conditions Survey 2008 When raw data is requested, user has an option to choose amongst individual files within the survey. datalibweb, country(ALB) year(2012) type(ECARAW) clear Survey ID: ALB_2012_LSMS_v01_M Title: Living Standards Measurement Survey 2012 Link to DDI: Survey metadata (microdata portal) 1. [datalibweb, country(ALB) year(2012) type(ECARAW) surveyid(ALB_2012_LSMS_v01_M) filename(bookbread.dta)] ...   41. [datalibweb, country(ALB) year(2012) type(ECARAW) surveyid(ALB_2012_LSMS_v01_M) filename(weights_identification.dta)]

Future features of datalibweb User web interface for subscriptions and usage Regional admin can upload subscription and maintain their data repositories Much better performance Local option/getfile to save and reuse the data in your machine while on mission Interactive explore of the data availability (raw and harmonized data, documentations and dofile) API for querying different types of files (data, dofile, questionnaire) Graphical User Interface

datalibweb performance We care a lot about performance of the system and the secured environment of your data! Getting the catalog and a specific file each might take about 3-7 seconds, conditional on the network location, traffic usage and machine. We are trying to reduce the time with some caching listings at the cost of real-time listings.

Flexible subscription mechanism User subscriptions will be based on “Types”, which can be: Data Documentation Dofile Or anything Within each Type, for example, you can have access to different folders of data. For example under the Data type, you can have “Data\Stata” or “Data\Base” folders, and each folder contains different stages of data harmonization. Similarly, within Type “Documentation”, users can have access to Doc/Questionnaire, Doc/Technical, or Doc/Reports Or you can create your own “Type”

Web interface for user’s subscription and usage

datalibweb from your laptop Users can run datalibweb while on mission or on planes (no need to have WB intranet network) Users need to use datalibweb to save the files into your machines – option “getfile” Users use the “local” option – just add the “local” to the end of your normal code The system can remember the query “getfile” and update or remove the files at the user’s request

datalibweb interactivity Users can explore the availability of data and documentation for both Raw/original and harmonization, and their own subscription.

datalibweb interactivity – view by vintage Users can explore the availability of data and documentation for both Raw/original and harmonization, and their own subscription.

datalibweb interactivity – view by available modules Users can explore the availability of data and documentation for both Raw/original and harmonization, and their own subscription.

datalibweb interactivity – view by available modules Users can explore the availability of data and documentation for both Raw/original and harmonization, and their own subscription.

datalibweb interactivity – view by available modules Users can explore the availability of data and documentation for both Raw/original and harmonization, and their own subscription.

datalibweb Stata GUI – Country view The graphical user interface will be replacement of Windows Explorer to see what surveys we have across surveys and what type of documentations we can access organized by Types. It also links with user subscriptions and let users know what they subscribed to. Users can filter the surveys by various options such as “latest version”, “data only”, “subscription status” or by keywords matched.

datalibweb Stata GUI – collection view The collection view shows all the harmonized collection within a server. It is organized by the collection type and allow you to see all the countries and vintages within that collection.

datalibweb Stata GUI – export survey lists Users can export the list of surveys in the view to text or clipboard for sharing or to prepare the list of non-subscribed surveys by sub-menu options.

datalibweb Stata GUI – Subscription request email

SOL – Statistics online (future)

Please let us know if you have any question/suggestion. datalibweb@worldbank.org FURL: datalibweb