External Data Access 5/29/08

Current Problems No way to load, process & analyze live Atlas data via critical analysis & programming tools (SAS, R, Perl) Direct PostgreSQL queries against Ontology Manager (OM), aka “The Blender”, are too difficult Accessing data by direct database connection is a potential security issue; need to minimize this OM is too slow for large datasets

Typical Programmer/Analyst Tasks: Requirements Require live, read-only access to all data user can view on Atlas from the tools she uses regularly Need ad hoc queries: easy to develop new queries and find the data of interest Need automated processes: e.g., nightly analysis of current data using a fixed query Need security: follow user’s Atlas permissions Key tools: SAS, R, Perl

Administrative Tasks: Requirements A few tasks require live, direct, read-only access to all data in the Atlas database using db admin tools, scripts, ODBC browsers, etc. Do not need to follow folder permissions since user can read all LabKey schemas Filtering by container, finding lists & objects, etc. should be reasonably easy Need queries with a small number of joins – Tabular OM data (lists, assays, samples) must be easier to query – Joins are okay… just reduce the complexity Key tools : pgAdmin III, DbVisualizer, EMS SQL Manager, Perl

Programmer/Analyst Tasks: Proposed Solution Provide their tools access to data via the LabKey “front door” as a wrapper on top of the query client API – Custom package for each tool: R, SAS, Perl (preferred solution) – ODBC wrapper (possible option) Usable from many tools (SAS EXEC SQL, RODBC) Clumsy, hard to develop, doesn’t help with rest of client API (insert/update? charts?) Encourage use of API-based solution to limit users who require direct database access Challenges – Add full SQL query support to API – Fix issues with current SQL syntax – Improve ease-of-use (build query in UI then “export to SAS/R script”) – Development effort to create wrappers for each language – SAS interface may be difficult due to proprietary nature of SAS

Administrative Tasks and Performance Issues: Proposed Solution Provide option to migrate tabular OM data from virtual to hard tables – User option for some types of data (e.g., lists) – Wholesale migration for others (e.g., datasets) – Continue to use OM for data stored as trees or graphs (e.g., experiment) Challenges – Naming hard tables in a reasonably discoverable way – Development and test effort