Mini-Project on Web Data Analysis DANIEL DEUTCH

Data Management “Data management is the development, execution and supervision of plans, policies, programs and practices that control, protect, deliver and enhance the value of data and information assets” (DAMA Data Management Body of Knowledge) A major success: the relational model of databases

Relational Databases Developed by Codd (1970), who won the Turing award for the model Huge success and impact: ‒The vast majority of organizational data today is stored in relational databases ‒Implementations include MS SQL Server, MS excel, Oracle DB, mySQL,… ‒2 Turing award winners (Edgar F. Codd and Jim Gray) Basic idea: data is organized in tables (=relations) Relations can be derived from other relations using a set of operations called the relational algebra ‒On which SQL is largely based

Research in Data(base) Management : Relational Databases (tables). ‒Indexing, Tuning, Query Languages, Optimizations, Expressive Power,…. ~20 years ago: Emergence of the Web and research on Web data ‒XML, text database, web graph…. ‒Google is a product of this research (by Stanford’s PhD students Brin and Page) Recent years: hot topics include distributed databases, data privacy, data integration, social networks, web applications, crowdsourcing, trust,… ‒Foundations taken from “classical” database research Theoretical foundations with practical impact

Web 2.0 “Old” web (“Web 1.0”): static pages – News, encyclopedic knowledge... – No, or very little, interactive process between the web-page and the user. Web 2.0: A term very broadly used for web-sites that use new technologies (Ajax, JS..), allowing interaction with the user. – “Network as platform" computing – The “participatory Web”

Web 2.0 “Old” web (“Web 1.0”): static pages – News, encyclopedic knowledge... – No, or very little, interactive process between the web-page and the user. Web 2.0: A term very broadly used for web-sites that use new technologies (Ajax, JS..), allowing interaction with the user. – “Network as platform" computing – The “participatory Web”

Online shopping

Advertisements

Social Networks

Crowd Sourcing

Data is all around Web graph “Social graph” Pictures, Videos, notifications, messages.. Data that the application processes Advertisments Even the application structure itself

(A small portion of) the web graph

Need to Analyze Huge amount of data out there – Est billion web-pages and counting – Half a billion tweets per day and counting An average user “sees” about 600 tweets per day Most of it is irrelevant for you, some is incorrect

Filter, Rank, Explain Filter – Select the portion of data that is relevant – Group similar results Rank – Rank data by trustworthiness, relevance, recency... – Present highest-rank first Explain – An explanation of why is the data considered relevant/highly-ranked – An explanation of how has the data propagated “Why do I see this?”

Main topics Analysis of Tables and Links on the Web Trust Management Explanation (Provenance) Information Extraction Social Networks Crowd-sourcing Distributed Query Evaluation

Approach Leverage knowledge from “classic” database research Account for the new challenges Do so in a generic manner Leverage unique features such as collaborative contribution, distribution, etc.

17 Students Physical Storage Indexing Distribution... Data modelQuery language Select… From… Where… Students Takes sid=sid sname name=“Mary ” cid=cid Courses

Foundations Model Query Language Query evaluation algorithms Prototype implementation and optimizations Getting Data and Testing

Project Requirements Read a paper (or a bunch of papers) in the area Likely to require that you follow citations and read earlier papers! Think of an application based on the paper ideas Does not have to be exactly the application described in the paper! E.g. you do not have to use relational databases Think of how would you get/generate data Implement, test Submit an application+ report

Report An integral part of the project submission Should include: A detailed description of the model and algorithms that you have implemented A detailed description of the application Code design Use cases Difficulties that you have encountered and how you addressed them

Timeline By 20/3 (1 week from now): send me an ordered list of 3 preferred papers title includes the words “mini-project” Body includes the names and IDs of the pair A bit after passover (date TBA): Each pair presents a 7-10 minutes presentation on the expected project A slide on each of the issues mentioned in the requirement slide 1 week before the last week of the semester: short project presentations (including screenshots or live demo)