Monitoring the acquisition process by web widgets Leonardo Tininini and Antonino Virgillito ISTAT Meeting on the Management of Statistical Information Systems (MSIS 2011) Luxembourg, May 2011

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS Outline Motivations The web widgets –The "General picture" widget –The "Regional thematic map" widget –The "Latest news histogram" widget Enhancing performances by materializing data The widgets' software architecture

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS Motivations The web is increasingly becoming a fundamental means for supporting several phases of the statistical data life cycle, particularly data acquisition and dissemination, however... little interest has been devoted so far to use the web for publicly monitoring the acquisition process Objectives: –providing, as far as possible, fresh data –do not disturb... (the acquisition process itself) –make the components “embeddable” in any web page

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS Web widgets A widget is a small web component that can be freely “embedded” into any web page (e.g. a blog), without requiring specific technical knowledge Widgets can be composed and organized in personalized dashboards according to the user's specific needs and preferences The widgets described here display some key data about the 2010 Italian Agriculture Census acquisition process

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS The “General picture” widget Shows the numeric values of 5 main aggregates: –the number of operators –the number of interviews completed by operators and recorded in the Census Management System –the number of questionnaires (fully) compiled by enterprises by using the web –the number of questionnaires returned (compiled) by operators by the web –the number of questionnaires under compilation (i.e. only partially compiled) on the web by enterprises. Accepts an additional parameter to select the territory of interest (Italy / a specific region) Refreshed every minute

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS The “General picture” widget (snapshot)

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS The “Regional thematic map” widget Same 5 aggregates as those in the “General picture” widget, each selectable by its own tab Aggregates are represented on the map of Italy using classes of values Exact aggregate values, along with names, can be obtained by rolling over the region on the map Each region's color is faded in proportion to the value of the specific aggregate for that region The maps are built using the freely-available Google Interactive Chart Tools API (specifically, the Geo Map component)

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS The “Regional thematic map” widget (snapshot)

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS The "Latest news histogram" widget Shows the time distribution of the questionnaires compiled over the last 24 hours Both enterprises and operators –the fraction corresponding to enterprises is depicted in darker green –the one corresponding to operators in lighter green Details can be obtained by rolling over some sections of the widget –the roll-over on each bar displays the details of the two component values (enterprises vs. operators) –the roll-over on the two legend items displays two different histograms, corresponding to questionnaires compiled only by enterprises or operators, respectively

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS The "Latest news histogram" widget (snapshot)

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS How to show the enterprise-only histogram

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS Enhancing performances by materializing data Data displayed by widgets are expensive to be computed (counts on several hundreds of thousands of records) Unpredictable number of users (due to free embedding) Potential performance degradation for both: –the operators and enterprises working on the Census Management and Acquisition System and –the widget users This potential degradation has been avoided by adapting two techniques, commonly known as view materialization and query rewriting, widely used in data warehousing environments –Displayed data are obtained by using (and possibly further processing) a few pre-computed data, instead of aggregating data from scratch

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS Materialized views and query rewriting

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS The widgets' software architecture Single Java web application, directly accessing the Census database High scalability: minimum (and bounded) overhead on the database against unpredictable request traffic Decoupling the access to the widgets made by the user from the access to the database (with aggregation) made by the widgets –One component (Update Job) computes the aggregates (storing them into the application memory) and is scheduled to automatically run every minute –The front-end component (widget in stricter terms) displays the aggregates stored in the application memory

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS High-level architecture of the web application

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS The widget embedding process Each widget is implemented as a separate JSP page The embedding starts form a single row of predefined HTML code that has to be included in the page (Embedding code) the code produces a call to a JavaScript fragment (Embedding script) that is stored on the server and is downloaded and executed on the client browser The execution of the Embedding script creates a frame within which the JSP page rendering the widget is actually loaded Through this technique, the widgets can be incorporated into virtually any web site or blog regardless of the underlying technology and without any programming knowledge

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS Conclusion and future works The proposed design can be considered as a general pattern for the acquisition process web monitoring: –easily embeddable “lightweight” components –high level of scalability, by... –decoupling the acquisition data from the monitoring data provided to users –price to pay: “freshness” of the information provided Next (more severe) testbed: monitoring the acquisition process of the Population Census –one/two order(s) of magnitude larger –the trade-off between freshness and overload will be much harder to achieve

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS Questions?