1. Large-Scale Spatial Data Management on New Hardware: A Case Study Using Taxi Trip Data in NYC
Jianting Zhang
Department of Computer Science
The City College of New York

2. Outline
Introduction
Spatial data, GIS, BigData, HPC, myself and my lab
A loyal MapInfo user (from v4.0 to v11.0)
Parallel spatial data management
Taxi trip data in NYC
Spatial query processing on GPUs
Efficient shortest path computation and applications
Ongoing research/development efforts
ISP: In-memory Spatial Data Processing System
Spatial Data Processing on Embedded Systems

3. Computational Geometry
Geographical Information System
GIS
Environmental Modeling
Air quality
Remote Sensing
Hydrology
Ecology
Social-Economic Modeling
Urban planning
Transportation
Census/Taxation
Social Studies
Spatial Databases: data modeling, indexing, query processing
Computational Geometry
Computer Graphics
Scientific Data/Information Visualization
Statistics/Machine learning
Image Processing/Computer Vision

4. Ecological Informatics
Computer Science
Spatial Databases
Data Mining
Geography
GIS Applications
Remote Sensing
Environmental sciences
Computer Science 4

5. Big Geospatial Data Challenges
Event Locations, trajectories and O-D data
E.g., Taxi trip records (GPS traces or O-D locations)
0.5 million in NYC (medallion taxi cab only) and 1.2 million in Beijing per day
From O-D locations to trajectories to frequent patterns
Satellite: e.g., from GOES to GOES-R (2015/2016) [$11B]
Spectral (3X)*spatial (4X)* temporal (5X)=60X
2km*2km*5min*16bands(360*60)*(180*60)*(12*24)*16~ 1+ trillion pixels per day
Derived thematic data products (vector)
Species distributions
E.g million occurrence records (GBIF)
E.g. 717,057 polygons and 78,929,697 vertices for 4148 birds distribution data (NatureServe)

6. Cloud computing+MapReduce+HadoopB C
Thread Block
CPU Host (CMP)
Core
Local Cache
Shared Cache
DRAM
HDD
SSD
GPU
SIMD
PCI-E
Ring Bus
...
GDRAM
MIC T0 T1 T2 T3
4-Threads
In-Order
16 Intel Sandy Bridge CPU cores+ 128GB RAM + 8TB disk + GTX TITAN + Xeon Phi 3120A ~ $9,994

7. 7.1 billion transistors (551mm²) 2,688 processors
Feb. 2013
7.1 billion transistors (551mm²)
2,688 processors
4.5 TFLOPS SP and 1.3 TFLOPS DP
Max bandwidth GB/s
PCI-E peripheral device
250 W (17.98 GFLOPS/W -SP)
Suggested retail price: $999
ASCI Red: 1997 First 1 Teraflops (sustained) system with Intel Pentium II Xeon processors (in 72 Cabinets)
This slides shows ASCI Red, the world’s first supercomputer with Teraflops double precision computing power in 1997 and the Nvidia GTX Titan available from the Market in 2013 for $1000 with even more computing power. Now the question is:
Many years ago, some database researchers tried to develop specialized hardware for database applications. However, that did not generate the expected impacts from the industry due to high cost. Personally, I do not believe GPUs are perfect for data management. However, I do believe that it is the most cost-effective hardware for high-performance data management, specially for geospatial data which is both data intensive and computing intensive. The simple belief has motivated us to work continuously over the past four years and we are happy to receive an NSF grant to continue the research in the next four years. To our knowledge, Prof. Kenneth Ross’s group at Columbia is also working on GPU-based data management, but they are mostly focusing on relational data.
What can we do today using a device that is more powerful than ASCI Red 16 years ago?

8. Affiliated Institutions
Students: Simin You (Ph.D ), Siyu Liao (Ph.D ), Costin Vicoveanu (Undergraduate, 2014-)
Bharat Rosanlall (Undergraduate, 2014), Jay Yao (MS-thesis, ), Chandrashekar Singh (MS 2013), Agniva Banerjee (MS, 2012), Roger King (MS, 2012), Wahyu Nugroho (MS, 2011), Xiao Quan Cen Feng (MS 2011), Chetram Dasrat (Undergraduate, 2008)
Geospatial Technologies and Environmental CyberInfrastructure (GeoTECI) Lab
Dr. Jianting Zhang
Department of Computer Science
The City College of New York
Collaborating Institutions

9. GeoTECI@CCNY HIGHEST-DB Trajectory data management on GPUs
$449,845/4yr (08/01/ /31/2017)
HIGHEST-DB
HIgh-performance GrapHics units based Engine for Spatial-Temporal data
Spatial and Spatiotemporal indexing, query processing and optimization
Trajectory data management on GPUs
Segmentation/simplification/compression/Aggregation/Warehousing
Map matching with road networks
Data mining (moving cluster, convoy, swarm...)
… when yellow cabs, green cabs and MTA buses meet with multi-core CPUs, GPUs and MICs in NYC …

10.
… when GOES-R satellites, extratropical cyclones and hummingbirds meet with TITAN …
High-resolution Satellite Imagery T
In-situ Observation Sensor Data
Global and Regional Climate Model Outputs
Data Assimilation
Zonal
Statistics
Ecological, environmental and administrative zones V
Temporal Trends
High-End Computing Facility A B C
Thread Block
ROIs

11. ...building a highly-configurable experimental computing environment for innovative BigData technologies…
CCNY
Computer Science LAN
CUNY HPCC
“Brawny” GPU cluster
Web Server/
Linux App Server
Windows App Server
SGI Octane III
Microway
DIY
Dell T5400
HP 8740w
KVM
HP 8740w
Lenovo T400s
Dual 8-core
128GB memory
Nvidia GTX Titan
Intel Xeon Phi 3120A
8 TB storage
Dual Quadcore
48GB memory
Nvidia C2050*2
8 TB storage *2
Dual-core
8GB memory
Nvidia GTX Titan
3 TB storage
Dual Quadcore
16GB memory
Nvidia Quadro 6000
1.5 TB storage
Quadcore
8 GB memory
Nvidia Quadro 5000m
“Wimmy” GPU cluster
Dell T7500
Dell T7500
Dell T5400
DIY
Dual 6-core
24 GB memory
Nvidia Quadro 6000
Dual 6-core
24 GB memory
Nvidia GTX 480
Dual Quadcore
16GB memory
Nvidia FX3700*2
Quadcore (Haswell)
16 GB memory
AMD/ATI 7970

12. Outline Introduction Spatial data, GIS, myself and my lab
A loyal MapInfo user (from v4.0 to v11.0)
Parallel spatial data management
Taxi trip data in NYC
Spatial query processing on GPUs
Efficient shortest path computation and applications
Ongoing research/development efforts
ISP: In-memory Spatial Data Processing System
Spatial Data Processing on Embedded Systems

13. Taxi trip data in NYC Taxicabs 13,000 Medallion taxi cabs
License priced at $600, 000 in 2007
Car services and taxi services are separate
Taxi trip records
~170 million trips (300 million passengers) in 2009
1/5 of that of subway riders and 1/3 of that of bus riders in NYC 13 13

14. Taxi trip data in NYC
Over all distributions of trip distance, time, speed and fare (2009)

15. Other types of OD Data Social network activities
Call Detail Record (CDR) 15 15

16. Taxi trip data in NYC How to manage taxi trip data? How good are they?
Geographical Information System (GIS)
Spatial Databases (SDB)
Moving Object Databases (MOD)
How good are they?
Pretty good for small amount of data 
But, rather poor for large-scale data 

17. Taxi trip data in NYC Can we do better? Example 1:
Loading 170 million taxi pickup locations into PostgreSQL
UPDATE t SET PUGeo = ST_SetSRID(ST_Point("PULong","PuLat"),4326);
105.8 hours!
Example 2:
Finding the nearest tax blocks for 170 million taxi pickup locations using open source libspatiaindex+GDAL
30.5 hours!
Intel Xeon 2.26 GHz processors with 48G memory
I do not have time to wait...
Can we do better?

18. Outline Introduction Spatial data, GIS, myself and my lab
A loyal MapInfo user (from v4.0 to v11.0)
Parallel spatial data management
Taxi trip data in NYC
Spatial query processing on GPUs
Efficient shortest path computation and applications
Ongoing research/development efforts
ISP: In-memory Spatial Data Processing System
Spatial Data Processing on Embedded Systems

19. Spatial query processing on GPUs
The goal is to design a data management system to efficiently manage large-scale taxi trip data on massively data parallel GPUs
With the help of new data models, data structures and algorithms
To cut the runtimes from hours to seconds on a single commodity GPU device
And support interactive queries and visual explorations

20. Spatial query processing on GPUs
Day
Physical Data Layout
Month
Raw data
Year
Compression, aggregation and indexing
Spatial Query and Data Mining
(Zhang, Gong, Kamga and Gruenwald, 2012), (Zhang, You and Gruenwald, 2014a)

21. Spatial query processing on GPUs8 9 7
Trip_Pickup_Location
Trip_Dropoff_Location
Start_Zip_Code
End_Zip_Code
time_between_service
distance_between_service
start_x
start_y
end_x
end_y
(local projection) 3
Start_Lon
Start_Lat End_Lon
End_Lat
Trip_Pickup_DateTime
Trip_Dropoff_DateTime 2 5
Passenger_Count
Fare_Amt
Tolls_Amt
Tip_Amt
Medallion#
Shift#
Trip#
Trip_Time
Trip_Distance 6 4 1
Payment_Type
Surcharge
Total_Amt
Rate_Code 10
vendor_name
date_loaded
store_and_forward 11
(Zhang, Gong, Kamga and Gruenwald, 2012)
(Zhang, You and Gruenwald, 2014a)

22. Spatial query processing on GPUs
Year
City
Week of the Year
Borough
Top level grid
Month
Day of the Year
Community District
Tax Lot
Census Tract
Day of the Week
Police Precinct
Day
Hour
Level k grid
Census Block
Tax Block
Peak/
off-peak
15/30-minutes
Street Segment
Level 0 grid
Pickup/drop-off locations
Pickup/drop-off timestamps
(Zhang, You and Gruenwald, 2012a), (Zhang, You and Gruenwald, 2014a)
NYC taxi trip records
Auxiliary data (weather, events…)

23. Spatial query processing on GPUs
P2P-T
P2N-D
P2P-D
(Zhang, You and Gruenwald, 2012a)

24. Spatial query processing on GPUs
(Zhang, Gong, Kamga and Gruenwald, 2012), (Zhang and You 2012), (You and Zhang 2013), (zhang, You and Gruenwald 2014a)

25. Spatial query processing on GPUs
Single-Level Grid-File based Spatial Filtering
(zhang, You and Gruenwald 2014a)
Vertices (polygon/polyline)
Points
Perfect coalesced memory accesses
Utilizing GPU floating point power
Nested-Loop based Refinement
(Zhang, You and Gruenwald, 2012a), (Zhang, You and Gruenwald, 2014a)

26. Spatial query processing on GPUs
Data
Taxi trip records: 300 million in two years ( ), ~170 million in 2009 (~150 million in Manhattan)
NYC DCPLION street network data: 147,011 street segments
NYC Census 2000 blocks: 38,794
NYC MapPluto Tax blocks: 735,488 in four boroughs (excluding SI) and 43,252 in Manhattan
Hardware
Dell T5400 Dual Quadcore CPUs with 16 GB memory
Nvidia Quadro 6000 with 448 cores and 6 GB memory

27. Spatial query processing on GPUs
Top: grid size =256*256
resolution=128 feet
Right: grid size =8192*8192
resolution=4 feet
Spatial Aggregation
9,424 /326=30X (8192*8192)
Temporal Aggregation
1709/198=8.6X (minute)
1598 /165 = 9.7X (hour)
(Zhang, You and Gruenwald, 2012a)

28. Spatial query processing on GPUs
38,794 census blocks (470,941 points)
735,488 tax blocks (4,698,986 points)
147,011 street segments
P2N-D
P2P-T
P2P-D
P2N-D
P2P-T
P2P-D -
15.2 h
30.5 h
10.9 s
11.2 s
33.1 s
4,900X
3,200X
Algorithmic improvement: 3.7X
Using main-memory data structures: 37.4X
GPU Acceleration: 24.3X
CPU time
GPU Time
Speedup
(Zhang, You and Gruenwald, 2012a), (Zhang and You 2012a) (Zhang and You 2012b)

29. Trip Purpose Analysis
(Zhang, You and Gruenwald, 2014b)

30. Trip purposes?
(Zhang, You and Gruenwald, 2014b)

31. Trip purposes?
(Zhang, You and Gruenwald, 2014b)

32. Trajectory data management on GPUs
(Zhang, You and Gruenwald, 2012b)
T-Drive (Microsoft Research Asia):
10,357 taxis in a week in Beijing (15 million points)
Complete GPS traces
GeoLife (Microsoft Research Asia)
178 users, 17,621trajectories, 23,667,828 points
“Walk” mode in Beijing: 3,245 trajectories with 1,440,823 points
After removing “too large” trajectories 2341 trajectories
Hausdorff Distance between two point sets
Results:
GPU Implementation using Thrust library (47.25 ms)
CPU Implementation using STL library (4,110ms)87X speedup
Results: 25-40X overall speedups

33. Outline Introduction Spatial data, GIS, myself and my lab
A loyal MapInfo user (from v4.0 to v11.0)
Parallel spatial data management
Taxi trip data in NYC
Spatial query processing on GPUs
Efficient shortest path computation and applications
Ongoing research/development efforts
ISP: In-memory Spatial Data Processing System
Spatial Data Processing on Embedded Systems

34. SP computation and applications
Shortest path computation
Dijkstra and A*
New generation algorithms
Contraction Hierarchy (CH) based
(Sommer 2012)
Regular edge 61 57 22
Shortcut 61 23
Contracted node
Original: 34,36,57,61,22,23,53,50,11,60 (9 hops)
Using shortcuts: 34,36,57,23,53,11,60 (6 hops)

35. SP computation and applications
Network Centrality (Brandes, 2008)
Node based
Edge based
Can be easily derived after shortest paths are computed
Mapping node/edge between centrality can reveal the connection strengths among different parts of cities

36. SP computation and applications Mapping Betweenness Centralities
166 million trips
25 million unique after matching point locations to road segments/intersections
Shortest path computation completes in less than 2 hours (5,952 seconds) on a single CPU core (2.0 GHZ)
4200 pairs computation per second
3 orders of magnitude faster than ArcGIS NA
Mapping of Computed Shortest Paths Overlaid with NAVSTREET road network and NYC Community Districts Map

37. SP computation and applications
Mapping Betweenness Centralities (All hours)

38. SP computation and applications
00H
02H
04H
06H
08H
10H
12H
14H
16H
18H
20H
22H
Legend:
Mapping Betweenness Centralities (bi-hourly)

39. SP computation and applications The data is not as clean as we had thought…outlier detection8 9 7
Trip_Pickup_Location
Trip_Dropoff_Location
Start_Zip_Code
End_Zip_Code
time_between_service
distance_between_service
start_x
start_y
end_x
end_y (local projection) 3
Start_Lon
Start_Lat End_Lon
End_Lat
Trip_Pickup_DateTime
Trip_Dropoff_DateTime 2 5
Passenger_Count
Fare_Amt
Tolls_Amt
Tip_Amt
Medallion#
Shift#
Trip#
Trip_Time
Trip_Distance 6 4 1
Meshed up on purpose due to privacy concerns
Payment_Type
Surcharge
Total_Amt
Rate_Code 10
vendor_name
date_loaded
store_and_forward 11

40. SP computation and applications The data is not as clean as we had thought…outlier detection
In addition:
Some of the data fields are empty
Pickup and drop-off locations can be in Hudson River
The recorded trip distance/duration can be unreasonable
...
Outlier detections for data cleaning are needed

41. SP computation and applications Outlier Detection
Existing approaches for outlier detection for urban computing
Thresholding: e.g. 200m < dist < 30km
Locating in unusual ranges of distributions
Spatial analysis: within a region or a land use type
Matching trajectory with road segments – treat unmatched ones as outliers
Some techniques require complete GPS traces while we only have O-D locations
Large-scale shortest path computing has not been used for outlier detection

42. SP computation and applications
Raw Taxi trip data
Type I outlier
(spatial analysis)
Match pickup/drop-off point locations to street segments within Distance D0
Successful?
Aggregate unique (sid,tid) pairs
Assign pickup/drop-off nodes by picking closer ones
Type II outlier
(network analysis)
Compute shortest path
CD >D1 AND CD>W*RD?
CD: Compute shortest distance
RD: Recorded trip distance
Update centrality measurements

43. SP computation and applications -Discussion on outlier detection
The approach is approximate in nature
Taxi drivers do not always follow shortest path
Especially for short trips and heavily congested areas
But we only care about aggregated results and the errors have a chance to be cancelled out by each other
Increasing D0 will reduce # of type I outliners, but the locations might be mismatched with segments
Reducing D1 and/or W will increase # of type II outliers but may generate false positives.

44. SP computation and applications -Results of outlier detection
Examples of Detected Type II Outliers
D0=200 feet, D1=3 miles, W=2
~2.5 millions (1.5%) type I outliers
~ 18,000 type II outliers

45. Outline Introduction Spatial data, GIS, myself and my lab
A loyal MapInfo user (from v4.0 to v11.0)
Parallel spatial data management
Taxi trip data in NYC
Spatial query processing on GPUs
Efficient shortest path computation and applications
Ongoing research/development efforts
ISP: In-memory Spatial Data Processing System
Spatial Data Processing on Embedded Systems

46. ISP: Large-Scale In-memory Spatial Data Processing
Preliminary experiment results:
6.4X speedup over PostgreSQL/PostGIS on an in-house 2-nonde mini-cluster (SGI Octane III)
10X speedup over Hadoop-GIS (based on Hadoop/Hive) on a 10-node Amazon EC2 cluster

47. Tiny GPU Cluster for BigData
12V/5A (MAX), typically <5W
Google I/O 2014 Project Tango (minutes 6-18)
Tiny GPU Cluster for BigData
Real-time 3D Spatial Indexing and Query processing on mobile platforms (with Prof. Zhigang

48. Some (hopefully not so) random thoughts on Spatial Computing for IoT
Cell-phone based sensing and querying 3D world (personal navigation)
Crowd-sourcing 3D urban infrastructure/traffic monitoring using RGB-D videos
ORNL: ees/etsd/btric
Building Information System and energy control
Ontology-driven emergency response and disaster relief

49. Q&A