26SEP2007 FOSS4G2007 Database Shootout: Benchmarking spatial DMBSs Wim de Haas Wilko Quak

Delft University of Technology 26SEP2007 FOSS4G20072 You’re in the eye of the storm! Early morning: Brock Anderson about WMS/PostGIS/Shapefile performance This afternoon: Kevin Neufeld about tips for the PostGIS power user Now: Reflect on the factor 10

Delft University of Technology 26SEP2007 FOSS4G20073 Overview Introduction What are the problems A classification of Spatial DBMS users How can we help them Benchmark proposal First test results Next steps

Delft University of Technology 26SEP2007 FOSS4G20074 Introducing the Ministry of Transport, Public Works and Water Management Our core tasks are: to offer protection against floods to guarantee safe and reliable connections over land, water and through the air to ensure clean and sufficient water Rijkswaterstaat (RWS) is the executive branche of the Ministry of Transport

Delft University of Technology 26SEP2007 FOSS4G20075 Business drivers How to keep track of all these assets? How to ensure consistency & coherence in operations and change of Rijkswaterstaat? How to facilitate decisionmaking and communication Enter the Digitaal Topografisch Bestand –3D –1:1000 –EUR 60M

Delft University of Technology 26SEP2007 FOSS4G20076 DTB waterway and highway

Delft University of Technology 26SEP2007 FOSS4G20077 DTB Birds eye view

Delft University of Technology 26SEP2007 FOSS4G20078 DTB Amsterdam Airport

Delft University of Technology 26SEP2007 FOSS4G20079 Enter IVRI The new system for data acquisition and maintenance for the DTB Oracle 10g ArcGIS 9.2 Summit Evolution

Delft University of Technology 26SEP2007 FOSS4G First comment on Murphy’s Law Murphy was an optimist Oracle and ESRI in a blame game Took extra time in the project Triggered to be less dependent on Oracle Oracle Spatial is not cheap, so can we use PostGIS as the main datastore for Spatial data?

Delft University of Technology 26SEP2007 FOSS4G Why bother … Stonebraker2007: Where to find dramatic differences in Spatial DBMSs? We define “dramatically outperform” to mean at least a factor 10 advantage […then] customers will be inclined to try the new architecture

Delft University of Technology 26SEP2007 FOSS4G Where to expect Dramatic differences? Linux vs Windows. (No) MySQL vs PostGIS (Yes) Choice of FileSystem (Maybe) Functionality Difference (Yes) Choice of Parameters (Maybe)

Delft University of Technology 26SEP2007 FOSS4G Problems with testing DBMS vendors do not want published results –Oracle explicitly forbids publishing benchmark results Hardware –Moore’s Law –I/O Release Frequency of Software Spatial testing cannot be done on synthetic data Too many parameters Benchmark results are outdated before they are publised

Delft University of Technology 26SEP2007 FOSS4G Benchmark consideration: Weird Cases department diagonal query geometry flat query geometry

Delft University of Technology 26SEP2007 FOSS4G Benchmark consideration: Hot vs Cold

Delft University of Technology 26SEP2007 FOSS4G Solution Do not publish the result of the benchmark Publish a framework that lets people do their own benchmarking No “One size fits all”: Buyer’s guide Help different users to find best DBMS

Delft University of Technology 26SEP2007 FOSS4G Classification of spatial DMBS users Four classes: 1. Server Builders: publish spatial data via web server 2. GIS User: Load various datasets and perform complex analyses 3. Data Maintainer: Maintain one core dataset 4. Power Users: All of the above and more

Delft University of Technology 26SEP2007 FOSS4G Class 1: Web Server Builders You do not really need a DBMS for this (You use a fraction of DBMS functionality) Only one query counts: Find everything within BBOX

Delft University of Technology 26SEP2007 FOSS4G Class 2: GIS users Main interest is functionality Spend more time on loading data Need a good query optimiser Analysis

Delft University of Technology 26SEP2007 FOSS4G Class 3: Dataset Maintainers Limited number of queries Transactions are an issue Clustering of data after updates is interesting

Delft University of Technology 26SEP2007 FOSS4G Class 4: Power users Do their own testing Need a platform to discuss their findings

Delft University of Technology 26SEP2007 FOSS4G Benchmark components 1. Functionality test Literature review Factual testing 2. Very simple performance test script with few parameters BBOX Query Fixed Dataset (Propasal OpenStreetMap dataset) 3. Configurable test suite Full Suite that tests every corner of DBMS For specialists only

Delft University of Technology 26SEP2007 FOSS4G Test 1 – Functionality: MySQL vs PostGIS MySQL Almost all operations in MySQL return the same result as the corresponding MBR- based functions. PostGIS Full OpenGIS support. Functionality of MySQL is only suited for simple WMS support and no spatial operations are done on geometry

Delft University of Technology 26SEP2007 FOSS4G Test 2: simple BBOX select Write simple script that generates a lot of rectangle queries. Paremeter: DBMS size query box size experiment length

Delft University of Technology 26SEP2007 FOSS4G Test 2: grow DBMS size Question: Does query response time depend on DBMS size or on core memory? Experiment: Run same test on more an more copies of same database.

Delft University of Technology 26SEP2007 FOSS4G Test 2 – result: PostGIS vs MySQL

Delft University of Technology 26SEP2007 FOSS4G Test 2 – result: Conclusions As long as dataset fits in core memory differences are small. MySQL can do more with less memery. MySQL degrades faster if you run out of memory

Delft University of Technology 26SEP2007 FOSS4G Test 3: Comprehensive Test Suite Create set of killer polygons so that every line of source code will be touched by running operations. Test Query optimizer Test Join Operator –Must be done with Skewed Data

Delft University of Technology 26SEP2007 FOSS4G Conclusions (overall) This is a work in progress The factor 10 is not within reach

Delft University of Technology 26SEP2007 FOSS4G How to proceed Finish the work and publish these TU Delft wiki or osgeo.org wiki? Start a Special Interest Group a.k.a. Chapter?

Delft University of Technology 26SEP2007 FOSS4G Questions