A Novel IT Architecture for Statistical Data Collection Using Big Data Technology Domenico Aprile, Lorenzo Di Gaetano, Guido Drovandi, Antonino Virgillito Bruxelles, March 15th 2017

Introduction The use of Big Data sources in the context of production of official statistics has been at the core of several initiatives at both national and international level in recent years. Among all the questions that were raised by the use of Big Data for statistics, a specific one is the use of the novel IT tools that are available for handling Big Data Capability of coping with high volume of data and variety of data formats (loosely-structured data) The use of Big Data tools is not necessarily tied to the handling of unusual datasets but they can prove their usefulness also to cover specific technical requirements. Can co-exist with traditional tools in a modern IT architecture This presentation focuses on the use of a NoSQL database in the context of a novel IT architecture for data collection. A Novel IT Architecture for Statistical Data Collection Using Big Data Technology

The problem Istat is currently developing a web-based tool for generalized collection of questionnaires. Each questionnaire has its own metadata structure which is prone to frequent changes along survey editions. This normally involves modifications that propagate through several connected software systems. A Novel IT Architecture for Statistical Data Collection Using Big Data Technology

The idea Use of a NoSQL database in the perspective of simplifying the overall architecture, by natively coping with heterogeneous and dynamically changing structures, and possibly also allows for a better performance over traditional tools. A Novel IT Architecture for Statistical Data Collection Using Big Data Technology

NoSQL databases The term “NoSQL” is used to indicate storage solutions that are not based on the traditional tabular data model of relational databases (RDBMS). NoSQL databases do not include typical features such as indexes, transactions etc. The principle is to trade the consistency guarantees of relational databases, that introduce an operational overhead and may not be necessary for all applications, for overall benefits in terms of enhanced query performance or support for loosely structured data. NoSQL DBs are not meant to replace RDBMSs but rather to cover different kinds of requirements. Several categories of NoSQL databases exist, that differs according to the kind of data model they use. Examples are document-based, that accept semi-structured content, graph, key-value and column-based. A Novel IT Architecture for Statistical Data Collection Using Big Data Technology

Column based NoSQL Key-value NoSQL databases are schemaless storage facilities in which records are simply organized into an identifier (key) and a value. The focus of such databases is on reaching a high throughput in operation, enabling random direct access to single rows in a table through the key. However, the lack of any form of structure leaves to the programmer all the responsibility of correcting, writing and interpreting the data stored in the value cell. Column based databases are an extension of the key-value store concept that accept the possibility of organizing the value part in a semi-structured schema, similar to a namespace organization, that still preserve some flexibility in the organization of data. A Novel IT Architecture for Statistical Data Collection Using Big Data Technology

Figure 1 : Data organization in Column-based NoSQL Databases It is articulated in a four-dimensional general schema, where the simple, flat value of the key-value stores is replaced by a set of column families, that are common to all the rows in a table and contain a set of column, that can vary from one row to another. Each row has also a timestamp that identify all the versions of the data (Figure 1). Column families Key Personal Data Professional Data   First Name Last Name Title Salary 1 John Doe Mr. 10000 2 Mary White Eng. 20000 3 Carl Green Phd. 35000 Column qualifiers Figure 1 : Data organization in Column-based NoSQL Databases A Novel IT Architecture for Statistical Data Collection Using Big Data Technology

Web Architecture for Data Collection Traditional architecture A different DB schema is required for each survey NoSQL based architecture All survey can be stored in a same table A Novel IT Architecture for Statistical Data Collection Using Big Data Technology

The proposed IT Architecture The idea is that the data collection software stores all the questionnaires into a column-based NoSQL database (NSDB), for all the surveys it manages. The data in the NSDB component can be then passed to successive phases of the survey, that, according to the specific survey, can be implemented with different technologies/tools, including RDBMS, SAS or other kinds of NoSQL databases such as MongoDB. A level of decoupling is then created, where the NSDB acts merely as a temporary storage area for backing up data collection and all the operations on microdata (check, editing, etc.) are carried out after data is extracted. This modular scheme represents a great simplification when applied to many different surveys, that are typically handled in heterogeneous ways in terms of software tools and methodology, and can facilitate the process of converging towards a unified architecture. A Novel IT Architecture for Statistical Data Collection Using Big Data Technology

Expected Benefits While we expect substantial benefits from this architecture regarding technical aspects such as the overall performance of data inserts, facilitated maintenance and more flexible and modular design, what we deem even more relevant is the possible impact of this solution over organization and production. The NoSQL storage, thanks to its virtually indefinite scalability in terms of size, may eventually become a unique centralized storage of raw survey microdata, that at the same time are always available for access and analysis. This organization easily enables cross-survey analysis of survey paradata, giving the possibility to extract novel in-depth insights over the collection process, that can help in assessing quality of the collection phase as a whole over different areas. A Novel IT Architecture for Statistical Data Collection Using Big Data Technology

Experiments We performed a comparison test between a key-value table using our traditional RDBMS architecture, using Oracle and Hibernate ORM, and an experimental HBase table. The test is based on a fixed amount of read/write cycles, to compare performance and throughput. We used a custom Java application for both Oracle and Hbase approaches. We also tested HBase scalability repeating read/write cycles on three tables of increasing sizes, to see how different table sizes impact on performance. A Novel IT Architecture for Statistical Data Collection Using Big Data Technology

Experimental Results RDBMS HBase Read/write avg. (ms) 31.65 7 10000 Read/write performance comparison RDBMS HBase Read/write avg. (ms) 31.65 7 Data access is much faster with NoSQL HBase scalability over increasing table size 10000 100000 500000 Write time avg (ms) 13.4 13.41 Read/write avg (ms) 7 10 Access times in NoSQL increase sub-linearly with table size A Novel IT Architecture for Statistical Data Collection Using Big Data Technology

Conclusions We showed that NoSQL databases represent a viable alternative to complement traditional data storage solutions Besides better performance of raw operations and a simplified programming model, NoSQL databases can enable interesting alternative ways of storing data For this reason, several possible use cases can be identified in statistical institutes around the idea of large-scale data repositories of non-uniform data A Novel IT Architecture for Statistical Data Collection Using Big Data Technology