WHD Colloquium, March 27, Historical Data Integration based on Collective Intelligence Vladimir Zadorozhny Graduate Information Science and Technology Program School of Information Sciences University of Pittsburgh NADM Group V. Zadorozhny

Challenge Diverse, Heterogeneous, Semi-structured Data Sources WHD Data Integration Infrastructure Consolidated Structured Information 2V. Zadorozhny

Web of Data? Linked Data: using the Web to create typed links between data from different sources Linked Data uses RDF (Resource Description Framework) to make typed statements (triples) Expected result: Web of Data extending the Web with a global data space connecting diverse domains (people, companies, publications, etc.) In general, Web of Data has a potential (still questionable) to support loose data coupling that may facilitate more efficient data utilization  While WHD can utilize LD and related Web mashup technologies to some extent, it would be premature to rely upon the Linked Data infrastructure 3WHD Colloquium, March 27, 2012V. Zadorozhny

Dataverse Network? An open source application to publish, share, reference, extract and analyze research data that facilitates making data available to others "Dataverse owners can upload any file type and format (excel, txt,pdf, doc, etc.), and the files will be stored and made available in the original format“ ( Information consumers should further integrate data sources to perform analysis using multiple "dataverses".  While WHD aims to be a part of the Dataverse Network, it would not encourage users to contribute data in ANY format. Instead, users integrate their data into the WHD repository while submitting the data.  To summarize, WHD infrastructure crowdsourses the data integration task, not just data contribution task. 4WHD Colloquium, March 27, 2012V. Zadorozhny

Data Submission System Structured homogeneous historical data Structured homogeneous historical data Information Providers Annotated historical data Internal Data Reliability Assessment Fused historical data Fused historical data Information Consumers … Wrapper Heterogeneous historical data sources Wrapper Generation Wrapper Registration External Data Reliability Assessment Data Fusion General WHD Architecture 5WHD Colloquium, March 27, 2012V. Zadorozhny

According to the 2006 revision of the World Population Prospects the total population in the region of Liberia in 1950 was 824,000. The average population growth percent per year for the following ten years was 2.5. For Ivory Coast those numbers are 2,505,000 and 3.6 correspondingly Extendable Target Schema (relational is not mandatory): Source | Location | From | To | Population | Data Source: s1 (xl) Data Source: s2 (doc) Source|Location | From |To | Population| s2 | Liberia | 01/01/1950 | 12/31/1950| | s2 |Liberia | 01/01/1960 | 12/31/1960| 1,052,000 | s2 |Ivory Coals | 01/01/1950 | 12/31/1950| 2,505, 000 | s2 |Ivory Coast | 01/01/1950 | 12/31/1950| 3,692,000 | Materialize Data Keep Data Remotely select * from Population s1 |Mauritania | 01/01/1950 | 12/31/1950| 692,000 | s1 |Mauritania | 01/01/1960 | 12/31/1960| 892,000 | s1 | Senegal | 01/01/1950 | 12/31/1950| 2,543,000 | s1 | Senegal | 01/01/1960 |12/31/1960 | 3,277,000 | Simple Scenario Mapping: Territories -> Location Population -> Population Data Aggregation -> Total Year -> From,To Wrapper Mapping: region -> Location Population -> Population Data Aggregation -> Total Year -> From,To Wrapper WHD Infrastructure

Data Curation Data Collection Data Utilization Big Picture: continuously growing infrastructure (a la Wikipedia) 7WHD Colloquium, March 27, 2012V. Zadorozhny

Group of graduate IS students: special project in Advanced Data Management class (INFSCI2711) Content Management → Pligg ( Open Source Content Management System, Apache, PHP, and MySQL based) Data Integration Engine → Pentaho Kettle (Open Source Data Integration Engine, Java-based GUI and Command Line Tools, XML based data transformation file) Data providers  download Wrapper Generating Software  configure wrappers on their workstation ( using preconfigured templates)  register wrappers on WHD Server WHD Prototype 8WHD Colloquium, March 27, 2012V. Zadorozhny

Data Source Data Transformation Transformed Data XML Wrapper 10

11WHD Colloquium, March 27, 2012V. Zadorozhny

Data Reliability Assessment and Data Fusion The systems based on crowdsourcing require mechanisms to ensure data quality. WHD Infrastructure will support efficient data curation strategies based on advanced data reliability assessment and data fusion methods. As system continuously receives new historical reports, WHD estimates reliability of this data, which evolves with respect to new evidence. WHS uses a measure of inconsistency caused by a report to assess its internal reliability. WHD also allows users to submit their subjective feedback on reliability of data to assess external reliability. WHD utilizes subjective logic to combine internal and external reliability assessment 12

Historical Data: Redundancy t1 | source_ref1 | Measles | NYC |10/10/1900 | 10/10/1920 | 700 t2 | source_ref2 | Measles | NYC |10/20/1910 | 10/30/1930 | 300 Total number of Measles cases in New York City from 1900 to 1930: = 1000 ??? Temporal overlap between t1 and t Measles reports: Temporal Overlaps t5 | source_ref1 | Yellow fever | NY |10/10/1900 | 10/10/1920 | 700 t6 | source_ref2 | Hepatitis | NY|10/10/1900 | 10/10/1920 | 700 t7 | source_ref4 | Hepatitis B | NY| 10/20/1910 | 10/30/1930 | 300 Total number of Hepatitis cases in New York State from 1920 to 1930: =1700 ??? Naming overlap between t5, t6 and t7 Naming Overlaps Total number of Smallpox cases in New York State from 1900 to 1930: = 1100 ??? Spatial overlap between t3 and t4 Smallpox reports: 500 (NY)600 (NYC) t3 | source_ref1 | Smallpox | NY |10/20/1900 | 10/20/1920 | 500 t4 | source_ref1 | Smallpox | NYC |10/30/1920 | 10/30/1930 | 600 Spatial Overlaps

Historical Data: Inconsistency time Measles reports in NYC: ………. R1: R2: Redundant and Inconsistent : 14WHD Colloquium, March 27, 2012V. Zadorozhny

Information Consumer Toolset: Data Visualization Dashboard

ICTS: Map Exhibits and Timeline Widgets

CV ICTS: Motion Chart Animation

Conclusion We explore a novel approach to reliable, large-scale historical data integration based on collective intelligence We implement this approach in WHD infrastructure for consolidation heterogeneous historical data Major challenge: how to engage a large community of researchers to share their data and collectively resolve the data heterogeneities in a continuously growing large-scale distributed historical repository? – contributions from CHAI members (only a small fraction of Wikipedia users contributes information to ensure its growth) – as the infrastructure evolves users may become interested in “embedding” their data in a larger context to perform global analysis and to utilize WHD tools – open development platform (extendable data transformation library and toolsets) 18WHD Colloquium, March 27, 2012V. Zadorozhny

Acknowledgements Graduate IS Students (WHD system development team): Andrew Barnett (team leader) Andrew Entin Thomas Junker Jidapa Kraisangka Han Liao Eric Miller Ye Peng Evan Pulgino Henry Quattrone Mark Swartz Miao Tan Liu Yuchen Lihong Zhang Doctoral Students: Ying-Feng Hsu Julian Lee 19WHD Colloquium, March 27, 2012V. Zadorozhny