1. 26SEP2007 FOSS4G2007 Database Shootout: Benchmarking spatial DMBSs Wim de Haas Wilko Quak

2. Delft University of Technology 26SEP2007 FOSS4G20072 You’re in the eye of the storm! Early morning: Brock Anderson about WMS/PostGIS/Shapefile performance This afternoon: Kevin Neufeld about tips for the PostGIS power user Now: Reflect on the factor 10

3. Delft University of Technology 26SEP2007 FOSS4G20073 Overview Introduction What are the problems A classification of Spatial DBMS users How can we help them Benchmark proposal First test results Next steps

4. Delft University of Technology 26SEP2007 FOSS4G20074 Introducing the Ministry of Transport, Public Works and Water Management Our core tasks are: to offer protection against floods to guarantee safe and reliable connections over land, water and through the air to ensure clean and sufficient water Rijkswaterstaat (RWS) is the executive branche of the Ministry of Transport

5. Delft University of Technology 26SEP2007 FOSS4G20075 Business drivers How to keep track of all these assets? How to ensure consistency & coherence in operations and change of Rijkswaterstaat? How to facilitate decisionmaking and communication Enter the Digitaal Topografisch Bestand –3D –1:1000 –EUR 60M

6. Delft University of Technology 26SEP2007 FOSS4G20076 DTB waterway and highway

7. Delft University of Technology 26SEP2007 FOSS4G20077 DTB Birds eye view

8. Delft University of Technology 26SEP2007 FOSS4G20078 DTB Amsterdam Airport

9. Delft University of Technology 26SEP2007 FOSS4G20079 Enter IVRI The new system for data acquisition and maintenance for the DTB Oracle 10g ArcGIS 9.2 Summit Evolution

10. Delft University of Technology 26SEP2007 FOSS4G200710 First comment on Murphy’s Law Murphy was an optimist Oracle and ESRI in a blame game Took extra time in the project Triggered to be less dependent on Oracle Oracle Spatial is not cheap, so can we use PostGIS as the main datastore for Spatial data?

11. Delft University of Technology 26SEP2007 FOSS4G200711 Why bother … Stonebraker2007: Where to find dramatic differences in Spatial DBMSs? We define “dramatically outperform” to mean at least a factor 10 advantage […then] customers will be inclined to try the new architecture

12. Delft University of Technology 26SEP2007 FOSS4G200712 Where to expect Dramatic differences? Linux vs Windows. (No) MySQL vs PostGIS (Yes) Choice of FileSystem (Maybe) Functionality Difference (Yes) Choice of Parameters (Maybe)

13. Delft University of Technology 26SEP2007 FOSS4G200713 Problems with testing DBMS vendors do not want published results –Oracle explicitly forbids publishing benchmark results Hardware –Moore’s Law –I/O Release Frequency of Software Spatial testing cannot be done on synthetic data Too many parameters Benchmark results are outdated before they are publised

14. Delft University of Technology 26SEP2007 FOSS4G200714 Benchmark consideration: Weird Cases department diagonal query geometry flat query geometry

15. Delft University of Technology 26SEP2007 FOSS4G200715 Benchmark consideration: Hot vs Cold

16. Delft University of Technology 26SEP2007 FOSS4G200716 Solution Do not publish the result of the benchmark Publish a framework that lets people do their own benchmarking No “One size fits all”: Buyer’s guide Help different users to find best DBMS

17. Delft University of Technology 26SEP2007 FOSS4G200717 Classification of spatial DMBS users Four classes: 1. Server Builders: publish spatial data via web server 2. GIS User: Load various datasets and perform complex analyses 3. Data Maintainer: Maintain one core dataset 4. Power Users: All of the above and more

18. Delft University of Technology 26SEP2007 FOSS4G200718 Class 1: Web Server Builders You do not really need a DBMS for this (You use a fraction of DBMS functionality) Only one query counts: Find everything within BBOX

19. Delft University of Technology 26SEP2007 FOSS4G200719 Class 2: GIS users Main interest is functionality Spend more time on loading data Need a good query optimiser Analysis

20. Delft University of Technology 26SEP2007 FOSS4G200720 Class 3: Dataset Maintainers Limited number of queries Transactions are an issue Clustering of data after updates is interesting

21. Delft University of Technology 26SEP2007 FOSS4G200721 Class 4: Power users Do their own testing Need a platform to discuss their findings

22. Delft University of Technology 26SEP2007 FOSS4G200722 Benchmark components 1. Functionality test Literature review Factual testing 2. Very simple performance test script with few parameters BBOX Query Fixed Dataset (Propasal OpenStreetMap dataset) 3. Configurable test suite Full Suite that tests every corner of DBMS For specialists only

23. Delft University of Technology 26SEP2007 FOSS4G200723 Test 1 – Functionality: MySQL vs PostGIS MySQL Almost all operations in MySQL return the same result as the corresponding MBR- based functions. PostGIS Full OpenGIS support. Functionality of MySQL is only suited for simple WMS support and no spatial operations are done on geometry

24. Delft University of Technology 26SEP2007 FOSS4G200724 Test 2: simple BBOX select Write simple script that generates a lot of rectangle queries. Paremeter: DBMS size query box size experiment length

25. Delft University of Technology 26SEP2007 FOSS4G200725 Test 2: grow DBMS size Question: Does query response time depend on DBMS size or on core memory? Experiment: Run same test on more an more copies of same database.

26. Delft University of Technology 26SEP2007 FOSS4G200726 Test 2 – result: PostGIS vs MySQL

27. Delft University of Technology 26SEP2007 FOSS4G200727 Test 2 – result: Conclusions As long as dataset fits in core memory differences are small. MySQL can do more with less memery. MySQL degrades faster if you run out of memory

28. Delft University of Technology 26SEP2007 FOSS4G200728 Test 3: Comprehensive Test Suite Create set of killer polygons so that every line of source code will be touched by running operations. Test Query optimizer Test Join Operator –Must be done with Skewed Data

29. Delft University of Technology 26SEP2007 FOSS4G200729 Conclusions (overall) This is a work in progress The factor 10 is not within reach

30. Delft University of Technology 26SEP2007 FOSS4G200730 How to proceed Finish the work and publish these TU Delft wiki or osgeo.org wiki? Start a Special Interest Group a.k.a. Chapter?

31. Delft University of Technology 26SEP2007 FOSS4G200731 Questions