Presentation is loading. Please wait.

Presentation is loading. Please wait.

RANKING THE INTERNETZ WITH MESHWORK Justin Cano Insight Data Engineering Fellow.

Published byMeredith Sharp Modified over 8 years ago

Similar presentations

Presentation on theme: "RANKING THE INTERNETZ WITH MESHWORK Justin Cano Insight Data Engineering Fellow."— Presentation transcript:

1 RANKING THE INTERNETZ WITH MESHWORK Justin Cano Insight Data Engineering Fellow

2 Motivation The internet is huge How does your page rank amongst others in your mesh? What is the reach of your website? Which pages are affecting your page rank? Data Source Common Crawl Organization More than 7 years of web page data, over 500TB CC April 2015 web corpus ~168TB Processed ~445GB for project Readily available in S3

3 Meshwork – your mesh in a network http://www.jcano.me/meshwork

4 Pipeline Data from S3 (source of truth) REST

5 Data Raw (WARC format) Extraction Edge List

6 … … Data Flow … … Link edge data (vertexId, pageRank)

7 Scaling up Page Rank job… spark-submit --class pageRank...

8 About Me Justin Cano UC Riverside BS Computer Engineering Previous work experience Software Engineer @ Hobbies I like building things! Hardware, software Learning and using new technologies Moviegoer Outdoor activities: biking, snowboarding Interests: design, app dev Favorite TV Shows: Futurama & The Daily Show Embedded Systems Developer @ Software Engineer Intern @

9

Download ppt "RANKING THE INTERNETZ WITH MESHWORK Justin Cano Insight Data Engineering Fellow."

Similar presentations

School of Engineering & Technology Computer Architecture Pipeline.

School of Engineering & Technology Computer Architecture Pipeline.
IRMA 20061 DOES OPTIONAL WEB- BASED SEMINAR SUPPORT BETTER (CONSTRUCTIVIST) LEARNING IN THE BASICS OF INFORMATICS? Pekka Makkonen Jyväskylä FINLAND.

IRMA DOES OPTIONAL WEB- BASED SEMINAR SUPPORT BETTER (CONSTRUCTIVIST) LEARNING IN THE BASICS OF INFORMATICS? Pekka Makkonen Jyväskylä FINLAND.
IT skills: IT concepts: Web client (browser), Web server, network connection, URL, mobile client, peer-to- peer application This work is licensed under.

IT skills: IT concepts: Web client (browser), Web server, network connection, URL, mobile client, peer-to- peer application This work is licensed under.
Introduction to Computers. Michael R. Izzo 401-942-3850.

Introduction to Computers. Michael R. Izzo
Problem Addressed Attempts to prove that Web Crawl is random & biased image of Web Graph and does not assert properties of Web Graph Understanding the.

Problem Addressed Attempts to prove that Web Crawl is random & biased image of Web Graph and does not assert properties of Web Graph Understanding the.
What do I (an ISE professional) need to be able to do-understand Need to understand the computer tools available to help me do my job better Need to understand.

What do I (an ISE professional) need to be able to do-understand Need to understand the computer tools available to help me do my job better Need to understand.
Key Traits of a Good Software Developer David Dunn Presented 4/20/10.

Key Traits of a Good Software Developer David Dunn Presented 4/20/10.
David Janzen Software Engineering II – Scratch Thursday, July 9, 2009.

David Janzen Software Engineering II – Scratch Thursday, July 9, 2009.
Thin and Thick Clients Figment Mark Hopf Chris Nassar.

Thin and Thick Clients Figment Mark Hopf Chris Nassar.
Internet Technologies Networking / Internet Protocols (TCP/IP) Server/Client Software Communication via Ports Web Page Technology Recipe of Web Page Development.

Internet Technologies Networking / Internet Protocols (TCP/IP) Server/Client Software Communication via Ports Web Page Technology Recipe of Web Page Development.
WEB SCIENCE: SEARCHING THE WEB. Basic Terms Search engine Software that finds information on the Internet or World Wide Web Web crawler An automated program.

WEB SCIENCE: SEARCHING THE WEB. Basic Terms Search engine Software that finds information on the Internet or World Wide Web Web crawler An automated program.
CS246 Search Engine Scale. Junghoo John Cho (UCLA Computer Science) 2 High-Level Architecture  Major modules for a search engine? 1. Crawler  Page.

CS246 Search Engine Scale. Junghoo "John" Cho (UCLA Computer Science) 2 High-Level Architecture  Major modules for a search engine? 1. Crawler  Page.
The Internet. What is the Internet?  The Internet is a network of networks.  It gives users access to a wide variety of information from millions of.

The Internet. What is the Internet?  The Internet is a network of networks.  It gives users access to a wide variety of information from millions of.
What IS the Web? Mrs. Wilson Internet Basics & Beyond.

What IS the Web? Mrs. Wilson Internet Basics & Beyond.
Use Links to Popularize Your Web Site Prepared by Milton Zlotnick SCORE 476 Staten Island, NY.

Use Links to Popularize Your Web Site Prepared by Milton Zlotnick SCORE 476 Staten Island, NY.
EMT1111 Logic and Problem Solving Dr. José M. Reyes Álamo Lecture 1.

EMT1111 Logic and Problem Solving Dr. José M. Reyes Álamo Lecture 1.
CS435/535, Computer Graphics Jingyuan(Alex) Zhang Office: 3413 SEC (Science and Engineering Complex) Phone: 348-9516

CS435/535, Computer Graphics Jingyuan(Alex) Zhang Office: 3413 SEC (Science and Engineering Complex) Phone:
Roshnika Fernando P AGE R ANK. W HY P AGE R ANK ?  The internet is a global system of networks linking to smaller networks.  This system keeps growing,

Roshnika Fernando P AGE R ANK. W HY P AGE R ANK ?  The internet is a global system of networks linking to smaller networks.  This system keeps growing,
Homework 4 Final homework Deadline: Sunday April 20, 11.59 PM In this homework you have to write a short essay on how Google can handle new types of data.

Homework 4 Final homework Deadline: Sunday April 20, PM In this homework you have to write a short essay on how Google can handle new types of data.
IST 441 Example Projects. Undergrad Project Find a customer – interest in xbox game forum Build a search engine for Xbox game forums etc. Compare two.

IST 441 Example Projects. Undergrad Project Find a customer – interest in xbox game forum Build a search engine for Xbox game forums etc. Compare two.

Similar presentations

Ads by Google