Large-Scale Spatial Data Management on New Hardware: A Case Study Using Taxi Trip Data in NYC Jianting Zhang Department of Computer Science The City College of New York http://www-cs.ccny.cuny.edu/~jzhang/

Outline Introduction Spatial data, GIS, BigData, HPC, myself and my lab A loyal MapInfo user (from v4.0 to v11.0) Parallel spatial data management Taxi trip data in NYC Spatial query processing on GPUs Efficient shortest path computation and applications Ongoing research/development efforts ISP: In-memory Spatial Data Processing System Spatial Data Processing on Embedded Systems

Computational Geometry Geographical Information System GIS Environmental Modeling Air quality Remote Sensing Hydrology Ecology Social-Economic Modeling Urban planning Transportation Census/Taxation Social Studies Spatial Databases: data modeling, indexing, query processing Computational Geometry Computer Graphics Scientific Data/Information Visualization Statistics/Machine learning Image Processing/Computer Vision

Ecological Informatics Computer Science Spatial Databases Data Mining Geography GIS Applications Remote Sensing Environmental sciences Computer Science 4

Big Geospatial Data Challenges Event Locations, trajectories and O-D data E.g., Taxi trip records (GPS traces or O-D locations) 0.5 million in NYC (medallion taxi cab only) and 1.2 million in Beijing per day From O-D locations to trajectories to frequent patterns Satellite: e.g., from GOES to GOES-R (2015/2016) [$11B] http://www.goes-r.gov/downloads/GOES-R-Tri-10-06-09_v7.pdf Spectral (3X)*spatial (4X)* temporal (5X)=60X 2km*2km*5min*16bands(360*60)*(180*60)*(12*24)*16~ 1+ trillion pixels per day Derived thematic data products (vector) http://www.goes-r.gov/products/baseline.html http://www.goes-r.gov/products/option2.html Species distributions E.g. 400+ million occurrence records (GBIF) E.g. 717,057 polygons and 78,929,697 vertices for 4148 birds distribution data (NatureServe)

Cloud computing+MapReduce+Hadoop B C Thread Block CPU Host (CMP) Core Local Cache Shared Cache DRAM HDD SSD GPU SIMD PCI-E Ring Bus ... GDRAM MIC T0 T1 T2 T3 4-Threads In-Order 16 Intel Sandy Bridge CPU cores+ 128GB RAM + 8TB disk + GTX TITAN + Xeon Phi 3120A ~ $9,994

7.1 billion transistors (551mm²) 2,688 processors Feb. 2013 7.1 billion transistors (551mm²) 2,688 processors 4.5 TFLOPS SP and 1.3 TFLOPS DP Max bandwidth 288.4 GB/s PCI-E peripheral device 250 W (17.98 GFLOPS/W -SP) Suggested retail price: $999 ASCI Red: 1997 First 1 Teraflops (sustained) system with 9298 Intel Pentium II Xeon processors (in 72 Cabinets) This slides shows ASCI Red, the world’s first supercomputer with Teraflops double precision computing power in 1997 and the Nvidia GTX Titan available from the Market in 2013 for $1000 with even more computing power. Now the question is: Many years ago, some database researchers tried to develop specialized hardware for database applications. However, that did not generate the expected impacts from the industry due to high cost. Personally, I do not believe GPUs are perfect for data management. However, I do believe that it is the most cost-effective hardware for high-performance data management, specially for geospatial data which is both data intensive and computing intensive. The simple belief has motivated us to work continuously over the past four years and we are happy to receive an NSF grant to continue the research in the next four years. To our knowledge, Prof. Kenneth Ross’s group at Columbia is also working on GPU-based data management, but they are mostly focusing on relational data. What can we do today using a device that is more powerful than ASCI Red 16 years ago?

Affiliated Institutions Students: Simin You (Ph.D. 2009 -), Siyu Liao (Ph.D. 2014-), Costin Vicoveanu (Undergraduate, 2014-) Bharat Rosanlall (Undergraduate, 2014), Jay Yao (MS-thesis, 2011-2012), Chandrashekar Singh (MS 2013), Agniva Banerjee (MS, 2012), Roger King (MS, 2012), Wahyu Nugroho (MS, 2011), Xiao Quan Cen Feng (MS 2011), Chetram Dasrat (Undergraduate, 2008) Geospatial Technologies and Environmental CyberInfrastructure (GeoTECI) Lab Dr. Jianting Zhang Department of Computer Science The City College of New York Collaborating Institutions

GeoTECI@CCNY HIGHEST-DB Trajectory data management on GPUs $449,845/4yr (08/01/2013-07/31/2017) HIGHEST-DB HIgh-performance GrapHics units based Engine for Spatial-Temporal data Spatial and Spatiotemporal indexing, query processing and optimization Trajectory data management on GPUs Segmentation/simplification/compression/Aggregation/Warehousing Map matching with road networks Data mining (moving cluster, convoy, swarm...) … when yellow cabs, green cabs and MTA buses meet with multi-core CPUs, GPUs and MICs in NYC …

GeoTECI@CCNY … when GOES-R satellites, extratropical cyclones and hummingbirds meet with TITAN … High-resolution Satellite Imagery T In-situ Observation Sensor Data Global and Regional Climate Model Outputs Data Assimilation Zonal Statistics Ecological, environmental and administrative zones V Temporal Trends High-End Computing Facility A B C Thread Block ROIs

...building a highly-configurable experimental computing environment for innovative BigData technologies… GeoTECI@CCNY CCNY Computer Science LAN CUNY HPCC “Brawny” GPU cluster Web Server/ Linux App Server Windows App Server SGI Octane III Microway DIY Dell T5400 HP 8740w KVM HP 8740w Lenovo T400s Dual 8-core 128GB memory Nvidia GTX Titan Intel Xeon Phi 3120A 8 TB storage Dual Quadcore 48GB memory Nvidia C2050*2 8 TB storage *2 Dual-core 8GB memory Nvidia GTX Titan 3 TB storage Dual Quadcore 16GB memory Nvidia Quadro 6000 1.5 TB storage Quadcore 8 GB memory Nvidia Quadro 5000m “Wimmy” GPU cluster Dell T7500 Dell T7500 Dell T5400 DIY Dual 6-core 24 GB memory Nvidia Quadro 6000 Dual 6-core 24 GB memory Nvidia GTX 480 Dual Quadcore 16GB memory Nvidia FX3700*2 Quadcore (Haswell) 16 GB memory AMD/ATI 7970

Outline Introduction Spatial data, GIS, myself and my lab A loyal MapInfo user (from v4.0 to v11.0) Parallel spatial data management Taxi trip data in NYC Spatial query processing on GPUs Efficient shortest path computation and applications Ongoing research/development efforts ISP: In-memory Spatial Data Processing System Spatial Data Processing on Embedded Systems

Taxi trip data in NYC Taxicabs 13,000 Medallion taxi cabs License priced at $600, 000 in 2007 Car services and taxi services are separate Taxi trip records ~170 million trips (300 million passengers) in 2009 1/5 of that of subway riders and 1/3 of that of bus riders in NYC 13 13

Taxi trip data in NYC Over all distributions of trip distance, time, speed and fare (2009)

Other types of OD Data Social network activities Call Detail Record (CDR) 15 15

Taxi trip data in NYC How to manage taxi trip data? How good are they? Geographical Information System (GIS) Spatial Databases (SDB) Moving Object Databases (MOD) How good are they? Pretty good for small amount of data  But, rather poor for large-scale data 

Taxi trip data in NYC Can we do better? Example 1: Loading 170 million taxi pickup locations into PostgreSQL UPDATE t SET PUGeo = ST_SetSRID(ST_Point("PULong","PuLat"),4326); 105.8 hours! Example 2: Finding the nearest tax blocks for 170 million taxi pickup locations using open source libspatiaindex+GDAL 30.5 hours! Intel Xeon 2.26 GHz processors with 48G memory I do not have time to wait... Can we do better?

Outline Introduction Spatial data, GIS, myself and my lab A loyal MapInfo user (from v4.0 to v11.0) Parallel spatial data management Taxi trip data in NYC Spatial query processing on GPUs Efficient shortest path computation and applications Ongoing research/development efforts ISP: In-memory Spatial Data Processing System Spatial Data Processing on Embedded Systems

Spatial query processing on GPUs The goal is to design a data management system to efficiently manage large-scale taxi trip data on massively data parallel GPUs With the help of new data models, data structures and algorithms To cut the runtimes from hours to seconds on a single commodity GPU device And support interactive queries and visual explorations

Spatial query processing on GPUs Day Physical Data Layout Month Raw data Year Compression, aggregation and indexing Spatial Query and Data Mining (Zhang, Gong, Kamga and Gruenwald, 2012), (Zhang, You and Gruenwald, 2014a)

Spatial query processing on GPUs 8 9 7 Trip_Pickup_Location Trip_Dropoff_Location Start_Zip_Code End_Zip_Code time_between_service distance_between_service start_x start_y end_x end_y (local projection) 3 Start_Lon Start_Lat End_Lon End_Lat Trip_Pickup_DateTime Trip_Dropoff_DateTime 2 5 Passenger_Count Fare_Amt Tolls_Amt Tip_Amt Medallion# Shift# Trip# Trip_Time Trip_Distance 6 4 1 Payment_Type Surcharge Total_Amt Rate_Code 10 vendor_name date_loaded store_and_forward 11 (Zhang, Gong, Kamga and Gruenwald, 2012) (Zhang, You and Gruenwald, 2014a)

Spatial query processing on GPUs Year City Week of the Year Borough Top level grid Month Day of the Year Community District Tax Lot Census Tract Day of the Week Police Precinct Day Hour Level k grid Census Block Tax Block Peak/ off-peak 15/30-minutes Street Segment Level 0 grid Pickup/drop-off locations Pickup/drop-off timestamps (Zhang, You and Gruenwald, 2012a), (Zhang, You and Gruenwald, 2014a) NYC taxi trip records Auxiliary data (weather, events…)

Spatial query processing on GPUs P2P-T P2N-D P2P-D (Zhang, You and Gruenwald, 2012a)

Spatial query processing on GPUs (Zhang, Gong, Kamga and Gruenwald, 2012), (Zhang and You 2012), (You and Zhang 2013), (zhang, You and Gruenwald 2014a)

Spatial query processing on GPUs Single-Level Grid-File based Spatial Filtering (zhang, You and Gruenwald 2014a) Vertices (polygon/polyline) Points Perfect coalesced memory accesses Utilizing GPU floating point power Nested-Loop based Refinement (Zhang, You and Gruenwald, 2012a), (Zhang, You and Gruenwald, 2014a)

Spatial query processing on GPUs Data Taxi trip records: 300 million in two years (2008-2010), ~170 million in 2009 (~150 million in Manhattan) NYC DCPLION street network data: 147,011 street segments NYC Census 2000 blocks: 38,794 NYC MapPluto Tax blocks: 735,488 in four boroughs (excluding SI) and 43,252 in Manhattan Hardware Dell T5400 Dual Quadcore CPUs with 16 GB memory Nvidia Quadro 6000 with 448 cores and 6 GB memory

Spatial query processing on GPUs Top: grid size =256*256 resolution=128 feet Right: grid size =8192*8192 resolution=4 feet Spatial Aggregation 9,424 /326=30X (8192*8192) Temporal Aggregation 1709/198=8.6X (minute) 1598 /165 = 9.7X (hour) (Zhang, You and Gruenwald, 2012a)

Spatial query processing on GPUs 38,794 census blocks (470,941 points) 735,488 tax blocks (4,698,986 points) 147,011 street segments P2N-D P2P-T P2P-D P2N-D P2P-T P2P-D - 15.2 h 30.5 h 10.9 s 11.2 s 33.1 s 4,900X 3,200X Algorithmic improvement: 3.7X Using main-memory data structures: 37.4X GPU Acceleration: 24.3X CPU time GPU Time Speedup (Zhang, You and Gruenwald, 2012a), (Zhang and You 2012a) (Zhang and You 2012b)

Trip Purpose Analysis (Zhang, You and Gruenwald, 2014b)

Trip purposes? (Zhang, You and Gruenwald, 2014b)

Trip purposes? (Zhang, You and Gruenwald, 2014b)

Trajectory data management on GPUs (Zhang, You and Gruenwald, 2012b) T-Drive (Microsoft Research Asia): 10,357 taxis in a week in Beijing (15 million points) Complete GPS traces GeoLife (Microsoft Research Asia) 178 users, 17,621trajectories, 23,667,828 points “Walk” mode in Beijing: 3,245 trajectories with 1,440,823 points After removing “too large” trajectories 2341 trajectories Hausdorff Distance between two point sets Results: GPU Implementation using Thrust library (47.25 ms) CPU Implementation using STL library (4,110ms)87X speedup Results: 25-40X overall speedups

Outline Introduction Spatial data, GIS, myself and my lab A loyal MapInfo user (from v4.0 to v11.0) Parallel spatial data management Taxi trip data in NYC Spatial query processing on GPUs Efficient shortest path computation and applications Ongoing research/development efforts ISP: In-memory Spatial Data Processing System Spatial Data Processing on Embedded Systems

SP computation and applications Shortest path computation Dijkstra and A* New generation algorithms Contraction Hierarchy (CH) based (Sommer 2012) Regular edge 61 57 22 Shortcut 61 23 Contracted node Original: 34,36,57,61,22,23,53,50,11,60 (9 hops) Using shortcuts: 34,36,57,23,53,11,60 (6 hops)

SP computation and applications Network Centrality (Brandes, 2008) Node based Edge based Can be easily derived after shortest paths are computed Mapping node/edge between centrality can reveal the connection strengths among different parts of cities

SP computation and applications Mapping Betweenness Centralities 166 million trips 25 million unique after matching point locations to road segments/intersections Shortest path computation completes in less than 2 hours (5,952 seconds) on a single CPU core (2.0 GHZ) 4200 pairs computation per second 3 orders of magnitude faster than ArcGIS NA Mapping of Computed Shortest Paths Overlaid with NAVSTREET road network and NYC Community Districts Map

SP computation and applications Mapping Betweenness Centralities (All hours)

SP computation and applications 00H 02H 04H 06H 08H 10H 12H 14H 16H 18H 20H 22H Legend: Mapping Betweenness Centralities (bi-hourly)

SP computation and applications The data is not as clean as we had thought…outlier detection 8 9 7 Trip_Pickup_Location Trip_Dropoff_Location Start_Zip_Code End_Zip_Code time_between_service distance_between_service start_x start_y end_x end_y (local projection) 3 Start_Lon Start_Lat End_Lon End_Lat Trip_Pickup_DateTime Trip_Dropoff_DateTime 2 5 Passenger_Count Fare_Amt Tolls_Amt Tip_Amt Medallion# Shift# Trip# Trip_Time Trip_Distance 6 4 1 Meshed up on purpose due to privacy concerns Payment_Type Surcharge Total_Amt Rate_Code 10 vendor_name date_loaded store_and_forward 11

SP computation and applications The data is not as clean as we had thought…outlier detection In addition: Some of the data fields are empty Pickup and drop-off locations can be in Hudson River The recorded trip distance/duration can be unreasonable ... Outlier detections for data cleaning are needed

SP computation and applications Outlier Detection Existing approaches for outlier detection for urban computing Thresholding: e.g. 200m < dist < 30km Locating in unusual ranges of distributions Spatial analysis: within a region or a land use type Matching trajectory with road segments – treat unmatched ones as outliers Some techniques require complete GPS traces while we only have O-D locations Large-scale shortest path computing has not been used for outlier detection

SP computation and applications Raw Taxi trip data Type I outlier (spatial analysis) Match pickup/drop-off point locations to street segments within Distance D0 Successful? Aggregate unique (sid,tid) pairs Assign pickup/drop-off nodes by picking closer ones Type II outlier (network analysis) Compute shortest path CD >D1 AND CD>W*RD? CD: Compute shortest distance RD: Recorded trip distance Update centrality measurements

SP computation and applications -Discussion on outlier detection The approach is approximate in nature Taxi drivers do not always follow shortest path Especially for short trips and heavily congested areas But we only care about aggregated results and the errors have a chance to be cancelled out by each other Increasing D0 will reduce # of type I outliners, but the locations might be mismatched with segments Reducing D1 and/or W will increase # of type II outliers but may generate false positives.

SP computation and applications -Results of outlier detection Examples of Detected Type II Outliers D0=200 feet, D1=3 miles, W=2 ~2.5 millions (1.5%) type I outliers ~ 18,000 type II outliers

Outline Introduction Spatial data, GIS, myself and my lab A loyal MapInfo user (from v4.0 to v11.0) Parallel spatial data management Taxi trip data in NYC Spatial query processing on GPUs Efficient shortest path computation and applications Ongoing research/development efforts ISP: In-memory Spatial Data Processing System Spatial Data Processing on Embedded Systems

ISP: Large-Scale In-memory Spatial Data Processing https://amplab.cs.berkeley.edu/benchmark/ http://blog.cloudera.com/blog/2014/05/new-sql-choices-in-the-apache-hadoop-ecosystem-why-impala-continues-to-lead/ Preliminary experiment results: 6.4X speedup over PostgreSQL/PostGIS on an in-house 2-nonde mini-cluster (SGI Octane III) 10X speedup over Hadoop-GIS (based on Hadoop/Hive) on a 10-node Amazon EC2 cluster

Tiny GPU Cluster for BigData 12V/5A (MAX), typically <5W http://elinux.org/Jetson/Jetson_TK1_Power Google I/O 2014 Project Tango (minutes 6-18) https://www.youtube.com/watch?v=cV8JDSO1NS8 Tiny GPU Cluster for BigData Real-time 3D Spatial Indexing and Query processing on mobile platforms (with Prof. Zhigang Zhu @CCNY)

Some (hopefully not so) random thoughts on Spatial Computing for IoT Cell-phone based sensing and querying 3D world (personal navigation) Crowd-sourcing 3D urban infrastructure/traffic monitoring using RGB-D videos http://www.sensor-networks.org/ ORNL: ees/etsd/btric Building Information System and energy control Ontology-driven emergency response and disaster relief

Q&A http://www-cs.ccny.cuny.edu/~jzhang/ jzhang@cs.ccny.cuny.edu