sfdc_ppt_corp_template_01_01_2012.ppt Unlocking Proprietary Data with PostgreSQL Foreign Data Wrappers Pat Patterson Principal Developer

Agenda  Foreign Data Wrappers  Writing FDW’s in C  Multicorn  Database.com FDW for PostgreSQL  FDW in action

Why Foreign Data Wrappers?  External data sources look like local tables! –Other SQL database MySQL, Oracle, SQL Server, etc –NoSQL database CouchDB, Redis, etc –File –LDAP –Web services Twitter!

Why Foreign Data Wrappers?  Make the database do the work –SELECT syntax DISTINCT, ORDER BY etc –Functions COUNT(), MIN(), MAX() etc –JOIN external data to internal tables –Use standard apps, libraries for data analysis, reporting

Foreign Data Wrappers  SQL Management of External Data (SQL/MED)  2011 – PostgreSQL 9.1 implementation –Read-only –SELECT-clause optimization –WHERE-clause push-down Minimize data requested from external source  Future Improvements –JOIN push-down Where two foreign tables are in the same server –Support cursors

FDW’s in PostgreSQL  ‘Compiled language’ (C) interface  Implement a set of callbacks typedef struct FdwRoutine { NodeTag type; /* These functions are required. */ GetForeignRelSize_function GetForeignRelSize; GetForeignPaths_function GetForeignPaths; GetForeignPlan_function GetForeignPlan; ExplainForeignScan_function ExplainForeignScan; BeginForeignScan_function BeginForeignScan; IterateForeignScan_function IterateForeignScan; ReScanForeignScan_function ReScanForeignScan; EndForeignScan_function EndForeignScan; /* These functions are optional. */ AnalyzeForeignTable_function AnalyzeForeignTable; } FdwRoutine;

FDW’s in PostgreSQL  Much work! CouchDB FDW couchdb_fdw.c > 1700 LoC

Multicorn   PostgreSQL 9.1+ extension  Python framework for FDW’s  Implement two methods…

Multicorn from multicorn import ForeignDataWrapper class ConstantForeignDataWrapper(ForeignDataWrapper): def __init__(self, options, columns): super(ConstantForeignDataWrapper, self).__init__(options, columns) self.columns = columns def execute(self, quals, columns): for index in range(20): line = {} for column_name in self.columns: line[column_name] = '%s %s' % (column_name, index) yield line

Database.com FDW for PostgreSQL  OAuth login to Database.com / Force.com –Refresh on token expiry  Force.com REST API –SOQL query SELECT firstname, lastname FROM Contact  Request thread puts records in Queue, execute() method gets them from Queue  JSON parsing – skip embedded metadat  < 250 lines code

Demo

Conclusion  Foreign Data Wrappers make the whole world look like tables!  Writing FDW’s in C is hard! –Or, at least, time consuming!  Writing FDW’s in Python via Multicorn is easy! –Or, at least, quick!  Try it for yourself!

Resources     FDW-for-PostgreSQL FDW-for-PostgreSQL