1. Scaling PostgreSQL with GridSQL

2. Who Am I? Jim Mlodgenski – Co-organizer of NYCPUG – Founder of Cirrus Technologies – Former Chief Architect of EnterpriseDB

3. Agenda What is GridSQL? Architecture Query Flow Scaling Limitations

4. What is GridSQL? “Shared-Nothing”, distributed data architecture. – Leverage the power of multiple commodity servers while appearing as a single database to the application Essentially... Open Source Greenplum, Netezza or Teradata

5. GridSQL Details Designed for Parallel Querying Not just “Read-Only”, can execute UPDATE, DELETE Data Loader for parallel loading Standard connectivity via PostgreSQL compatible connectors: JDBC, ODBC, ADO.NET, libpq (psql)

6. What GridSQL is not? A replication solution like Slony or Bucardo A high availability solution like Streaming Replication in PostgreSQL 9.0 A scalable transactional solution like PostgresXC An elastic, eventually consistent NoSQL database

7. Configuration Can be configured for multiple logical “nodes” per physical server – Take advantage of multi-core processors Tables may be either replicated or partitioned Replicated tables for static lookup data or dimensions – Partitioned tables for large fact tables

8. Partitioning Tables may simultaneously use GridSQL Partitioning with Constraint Exclusion Partitioning – Large queries scan a much smaller subset of data by using subtables – Since each subtable is also partitioned across nodes, they are scanned in parallel – Queries execute much faster

9. Architecture Loosely coupled, shared-nothing architecture Data repositories – Metadata database – GridSQL database GridSQL processes – Central coordinator – Agents

10. Query Optimization Cost Based Optimizer – Takes into account Row Shipping (expensive) Looks for joins with replicated tables – Can be done locally – Looks for joins between tables on partitioned columns

11. Aggregation First set of aggregates done in parallel at the nodes Like groups of intermediate results shipped to same target node Second aggregation done in parallel Coordinator streams in node results, combining on the fly and sending to client result set, performing a merge sort if ORDER BY present

12. Two Phase Aggregation SUM – SUM(stat1) – SUM2(SUM(stat1) AVG – SUM(stat1) / COUNT(stat1) – SUM2 (SUM(stat1)) / SUM2 (COUNT(stat1))

13. Creating Tables Tables can be partitioned or replicated CREATE TABLE region (r_regionkey INTEGER NOT NULL, r_name CHAR(25) NOT NULL, r_comment VARCHAR(152)) REPLICATED;

14. Creating Tables CREATE TABLE orders ( o_orderkey INTEGER NOT NULL, o_custkey INTEGER NOT NULL, o_orderstatus CHAR(1) NOT NULL, o_totalprice DECIMAL(15,2) NOT NULL, o_orderdate DATE NOT NULL, o_orderpriority CHAR(15) NOT NULL, o_clerk CHAR(15) NOT NULL, o_shippriority INTEGER NOT NULL, o_comment VARCHAR(79) NOT NULL) PARTITIONING KEY o_orderkey ON ALL;

16. Query 1 – Execution (no Agents) Go to Animation Slide

18. Query 7 – Execution (with Agents) Go to Animation Slide

19. Scalability

20. A few DBT3 queries on Amazon EC2 – Using PostgreSQL 9.0

23. Limitations SQL Support – Uses its own parser and optimizer so: No Window Functions No Stored Procedures No Full Text Search No Spatial

24. Transaction Performance Single row Insert, Update, or Delete are slow compared to a single PostgreSQL instance – The data must make an additional network trip to be committed – All partitioned rows must be hashed to be mapped to the proper node – All replicated rows must be committed to all nodes Use “gs-loader” for bulk loading for better performance

25. High Availability No heartbeat or fail-over control in the coordinator – High Availability for each PostgreSQL node must be configured separately – Streaming replication can be ideal for this Getting a consistent backup of the entire GridSQL database is difficult – Must ensure there are no transaction are occurring – Backup each node separately

26. Adding Nodes Requires Downtime – Data must be manually reloaded to partition the data to the new node With planning, the process can be fast with no mapping of data – Run multiple PostgreSQL instances on each physical server and move the PostgreSQL instances to new hardware as needed

27. Interesting Side Note GridSQL scales well in a cloud environment The results are dependent on the cloud vendor

28. Summary GridSQL can improve performance tremendously of PostgreSQL queries GridSQL can scale linearly as more nodes are added GridSQL is open source so if the limitations are an issue, submit a patch

29. Download GridSQL at: http://sourceforge.net/projects/gridsql/ Jim Mlodgenski Email: jim@cirrusql.comjim@cirrusql.com Twitter: @jim_mlodgenski