Presentation is loading. Please wait.

Presentation is loading. Please wait.

Building a real time Tweet map with Flink in six weeks

Published byBrianne Knight Modified over 6 years ago

Similar presentations

Presentation on theme: "Building a real time Tweet map with Flink in six weeks"— Presentation transcript:

1 Building a real time Tweet map with Flink in six weeks
OSTMap Fast poc development with flink

2 Proof of concept - an important tool in the industry
PoC often necessary to show feasibility to customers touch several topics: Scalability Stream processing Batch processing Storage and querying of data OSTMap as example PoC

3 Goals for OSTMap Increase trust into big data technologies on customer side It is easy to build an application with current technologies With almost no experience Teach students big data technologies Recruiting Bring big data to the university Build a real time application to view recent geotagged tweets on a map Search for terms and users, show these tweets on a map Analytics: First data science jobs …

4 Industry in practice: IT-Ringvorlesung 2016
Martin Grimmer A course at the University of Leipzig. work on projects of local companies six students over a period of 6 weeks - no full time invest Weekly meetings Github project: github.com/IIDP/OSTMap Matthias Kricke Michael Schmeißer Nico Graebling OSTMap Vincent Märkl Christopher Rost Christopher Schott Kevin Shrestha Hans Dieter Pogrzeba

5 mgm technology partners
We bring applications into production! Innovative software solution provider with application responsibility Specialist for highly scalable, transactional online applications Central lines of business: Insurance, E-Commerce, E-Government Founded in 1994 347 employees, 9 offices (2014) Revenue: 43,7 Mio € (2014) Part of Allgeier SE

6 ScaDS Competence center for scalable data services and solutions Dresden/Leipzig bundled Big Data research expertise of the TU Dresden and Leipzig University Drive Big Data innovations Bring industry and science together Knowledge exchange and transfer

7 Walking skeleton “A Walking Skeleton is a tiny implementation of the system that performs a small end-to-end function. It need not use the final architecture, but it should link together the main architectural components. The architecture and the functionality can then evolve in parallel.” - Alistair Cockburn gif from

8 Milestone 1 read stream, store data as json file, show tweets, read data from json files stream processing Ingest / Storage Querying visualisation

9 Milestone 2 write to and read from accumulo, show tweets on map, full table scans, slow visualization

10 Milestone 3 Term index, geotemporal index, ui improvements, clustering, …

11 OSTMap – stream, batch, storage and querying
a) stream processing geotagged tweets b) batch processing c) querying data webservice

12 Stream processing of incoming data – first version
data for GeoTweetSource DateExtraction KeyGeneration RawTweetSink time index This enabled us to build a slow term search and a slow map search via full table scans.

13 Stream processing of incoming data – final version
data for GeoTweetSource DateExtraction KeyGeneration RawTweetSink time index TermExtraction (tokenizing) TermIndexSink term index UserExtraction GeoTemporalIndexCreation GeoTemporalIndexSink geotemporal index Language Extraction 1 minute window sum by language LanguageFrequencySink language statistics Now we were able to build a faster term and map search and language frequency visualization.

14 Batch processing Initial creation of the term index and geotemporal index for already processed tweets Data export Other statistics like: Area/ tweet distance a user covers with his tweets

15 RawTweetData (TimeIndex)
Storage Accumulo table design Table Row Column Family Column Qualifier Value RawTweetData (TimeIndex) timestamp, hash 8b + 4b - raw tweet json TermIndex term field (user,text) RawTweetData key 12b LanguageFrequency time bucket YYYYMMDDhhmm language-tag tweet count 4b

16 Geotemporal Index for OSTMap Geo index
geohashes used as row keys in accumulo … 3z 6b 6c 6f 6q 9p 9r 9x 9z d0 d1 d2 d3 d4 d5 d6 … dg Row CF CQ geohash RawTweetKey - geo data 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g partitioned by geohash (z curve) function from 2d coordinate space to 1d key space

17 Geotemporal Index for OSTMap Geo index – querying?
Row CF CQ geohash RawTweetKey lat/lon 9z db dc df dg … 3z 6b 6c 6f 6q 9p 9r 9x 9z d0 d1 d2 d3 d4 d5 d6 … dg 9x d8 d9 dd de range: [9p] 9r d2 d3 d6 d7 bounding box range: [9r] accumulo iterators 9p d0 d1 d4 d5 accumulo iterators range: [d0,d1,d2,d3] result 3z 6b 6c 6f 6g partitioned by geohash accumulo iterators calculate coverage of bounding box calculate scan ranges from coverage

18 partitioned by geohash,
Geotemporal Index for OSTMap Add some time! … 13z 16b 16c 16f 16q 19p 19r 19x 19z 1d0 1d1 1d2 1d3 1d4 1d5 1d6 … 1dg … 23z 26b 26c 26f 26q 29p 29r 29x 29z 2d0 2d1 2d2 2d3 2d4 2d5 2d6 … 2dg Row CF CQ day, geohash RawTweetKey lat/lon 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g … day lon lat partitioned by geohash, with timebuckets day 1 day 2 day i …

19 partitioned by geohash,
Geotemporal Index for OSTMap What about Hotspots? … 13z 16b 16c 16f 16q 19p 19r 19x 19z 1d0 1d1 1d2 1d3 1d4 1d5 1d6 … 1dg … 23z 26b 26c 26f 26q 29p 29r 29x 29z 2d0 2d1 2d2 2d3 2d4 2d5 2d6 … 2dg Row CF CQ day, geohash RawTweetKey lat/lon 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g … day lon lat partitioned by geohash, with timebuckets

20 partitioned by geohash,
Geotemporal Index for OSTMap What about Hotspots? spreading byte Row CF CQ sb, day, geohash RawTweetKey lat/lon node 0 … 01d2 01d3 01d4 … … 02d2 02d3 02d4 … 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g … 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g 9z db dc df dg 9x d8 d9 dd de 9r d2 d3 d6 d7 9p d0 d1 d4 d5 3z 6b 6c 6f 6g spreading byte = hash(tweet) % 255 reproducable pre table splits in accumulo node 1 … 11d2 11d3 11d4 … … 12d2 12d3 12d4 … … node 2 … 21d2 21d3 21d4 … … 22d2 22d3 22d4 … day lon lat … partitioned by geohash, with timebuckets … node n

21 demo

22 Thank you www.scads.de www.mgm-tp.com
Martin Grimmer grimmer[at]informatik.uni-leipzig.de Matthias Kricke kricke[at]informatik.uni-leipzig.de Michael Schmeißer michael.schmeisser[at]mgm-tp.com

Download ppt "Building a real time Tweet map with Flink in six weeks"

Similar presentations

DIGIDOC A web based tool to Manage Documents. System Overview DigiDoc is a web-based customizable, integrated solution for Business Process Management.

DIGIDOC A web based tool to Manage Documents. System Overview DigiDoc is a web-based customizable, integrated solution for Business Process Management.
Page 1 GADD Software - An Introduction Public version, August 2014, gaddsoftware.com.

Page 1 GADD Software - An Introduction Public version, August 2014, gaddsoftware.com.
Supervised by Prof. LYU, Rung Tsong Michael Department of Computer Science & Engineering The Chinese University of Hong Kong Prepared by: Chan Pik Wah,

Supervised by Prof. LYU, Rung Tsong Michael Department of Computer Science & Engineering The Chinese University of Hong Kong Prepared by: Chan Pik Wah,
A NEW PLATFORM FOR A NEW ERA. 2 Pivotal Confidential–Internal Use Only 2 The Pivotal Big Data Suite.

A NEW PLATFORM FOR A NEW ERA. 2 Pivotal Confidential–Internal Use Only 2 The Pivotal Big Data Suite.
7/14/2015EECS 584, Fall 20111 MapReduce: Simplied Data Processing on Large Clusters Yunxing Dai, Huan Feng.

7/14/2015EECS 584, Fall MapReduce: Simplied Data Processing on Large Clusters Yunxing Dai, Huan Feng.
FACULTY OF COMPUTER SCIENCE OUTPUT DD  annual event from students for students with contact to industry (~800 visitors)  live demonstrations  research.

FACULTY OF COMPUTER SCIENCE OUTPUT DD  annual event from students for students with contact to industry (~800 visitors)  live demonstrations  research.
Hadoop Team: Role of Hadoop in the IDEAL Project ●Jose Cadena ●Chengyuan Wen ●Mengsu Chen CS5604 Spring 2015 Instructor: Dr. Edward Fox.

Hadoop Team: Role of Hadoop in the IDEAL Project ●Jose Cadena ●Chengyuan Wen ●Mengsu Chen CS5604 Spring 2015 Instructor: Dr. Edward Fox.
Analytics Map Reduce Query Insight Hive Pig Hadoop SQL Map Reduce Business Intelligence Predictive Operational Interactive Visualization Exploratory.

Analytics Map Reduce Query Insight Hive Pig Hadoop SQL Map Reduce Business Intelligence Predictive Operational Interactive Visualization Exploratory.
Committed to Deliver….  We are Leaders in Hadoop Ecosystem.  We support, maintain, monitor and provide services over Hadoop whether you run apache Hadoop,

Committed to Deliver….  We are Leaders in Hadoop Ecosystem.  We support, maintain, monitor and provide services over Hadoop whether you run apache Hadoop,
Overview of SQL Server Alka Arora.

Overview of SQL Server Alka Arora.
M i SMob i S Mob i Store - Mobile i nternet File Storage Platform Chetna Kaur.

M i SMob i S Mob i Store - Mobile i nternet File Storage Platform Chetna Kaur.
HBase A column-centered database 1. Overview An Apache project Influenced by Google’s BigTable Built on Hadoop ▫A distributed file system ▫Supports Map-Reduce.

HBase A column-centered database 1. Overview An Apache project Influenced by Google’s BigTable Built on Hadoop ▫A distributed file system ▫Supports Map-Reduce.
Project 2 Presentation & Demo Course: Distributed Systems By Pooja Singhal 11/22/2011 1.

Project 2 Presentation & Demo Course: Distributed Systems By Pooja Singhal 11/22/
Google’s Big Table 1 Source: Chang et al., 2006: Bigtable: A Distributed Storage System for Structured Data.

Google’s Big Table 1 Source: Chang et al., 2006: Bigtable: A Distributed Storage System for Structured Data.
BigTable and Accumulo CMSC 461 Michael Wilson. BigTable  This was Google’s original distributed data concept  Key value store  Meant to be scaled up.

BigTable and Accumulo CMSC 461 Michael Wilson. BigTable  This was Google’s original distributed data concept  Key value store  Meant to be scaled up.
Map-Reduce-Merge: Simplified Relational Data Processing on Large Clusters Hung-chih Yang(Yahoo!), Ali Dasdan(Yahoo!), Ruey-Lung Hsiao(UCLA), D. Stott Parker(UCLA)

Map-Reduce-Merge: Simplified Relational Data Processing on Large Clusters Hung-chih Yang(Yahoo!), Ali Dasdan(Yahoo!), Ruey-Lung Hsiao(UCLA), D. Stott Parker(UCLA)
Key/Value Stores CMSC 491 Hadoop-Based Distributed Computing Spring 2015 Adam Shook.

Key/Value Stores CMSC 491 Hadoop-Based Distributed Computing Spring 2015 Adam Shook.
G063 - Distributed Databases. Learning Objectives: By the end of this topic you should be able to: explain how databases may be stored in more than one.

G063 - Distributed Databases. Learning Objectives: By the end of this topic you should be able to: explain how databases may be stored in more than one.
© 2008 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP-ITU Innovation Center Dicle Erkul.

© 2008 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP-ITU Innovation Center Dicle Erkul.
Near Real-Time Verification At The Forecast Systems Laboratory: An Operational Perspective Michael P. Kay (CIRES/FSL/NOAA) Jennifer L. Mahoney (FSL/NOAA)

Near Real-Time Verification At The Forecast Systems Laboratory: An Operational Perspective Michael P. Kay (CIRES/FSL/NOAA) Jennifer L. Mahoney (FSL/NOAA)

Similar presentations

Ads by Google