701 - Building with the Data Tools project John Graham Der Ping Chou

Data Tools Platform DTP New top-level project at Eclipse Managed by Sybase, IBM, and Actuate Additional contributions: –Versant –ObjectNation GmbH –Pervasive Software Broad community support

Who We Are John Graham DTP PMC Sybase Der Ping Chou WTP Committer DTP IBM

Outline Overview of DTP Model Base project detail Case study: Demonstration of extending the base models Questions

Purpose of DTP Provide a comprehensive set of extensible data frameworks exemplary tools for data-centric applications

Why DTP? Data is fundamental to software applications Known set of requirements based on years of experience Unify the approach for Eclipse Enable higher-level data applications

Scope of DTP PMC worked very hard on this section The Scope section was very difficult: –Data is used everywhere –How to draw the line between what naturally belongs in DTP and what does not? Settled on a test based on problem domain

Scope Test Data-centric applications –Associated with a data source –Having a problem domain no more specific than just “data” within the Eclipse ecosystem of projects Data-dependent applications –Built on data-centric frameworks and tools –Have a more specific problem domain in the Eclipse ecosystem of projects

Example: In Scope SQL Editor Associated with a data source –SQL statement itself –Database system using the statement Problem domain –Generic (relational) data operations –Not naturally in scope for other, existing Eclipse projects

Example: Out of Scope Java source code editor: Uses data (source code) Problem domain is Java tooling –Eclipse has the Java Development Tools (JDT) project –So, not in scope for DTP, because the more specific JDT problem domain includes it

DTP Structure As a top-level project at Eclipse: DTP project –(Sub)project Component DTP is managed by the PMC A project is managed by a project lead A component has a technical lead All volunteers, all open source

Current Data Tooling Web Tools Platform (WTP) project has an rdb component Business Intelligence and Reporting Tools Project (BIRT) Eclipse plug-ins, both commercial and open source

Consolidation in DTP rdb moving components to DTP BIRT providing Open Data Access (ODA) to DTP Sybase open sourcing connectivity and SQL tooling for DTP Donations from ObjectNation GmbH Other areas under discussion

Possible Future Components In support of Object/Relational Mapping (ORM): –Versant, Oracle have projects at Eclipse in this area –Working with them, and others, to insure that DTP meets the community’s data requirements XML/Relational Mapping Administration Extract-Transform-Load (ETL)

First DTP Release Concentrating on relational structures Milestones planned for –2 in 5Q4 –2 in 6Q1 Release in late 6Q1 Project plan draft posted on DTP site

Milestone Components

DTP Architecture

DTP Projects Model Base Connectivity SQL Development Tools

Model Base Project Provides foundational domain models for DTP and DTP consumers, based on Eclipse Modeling Framework (EMF) and using a model-driven development with UML

Model Base Benefits Support for change management Broad editing support: visual and otherwise Extensible, database agnostic models Compliance with latest standards, such as SQL Supporting JDBC and other connection standards

Model Base Components Driver Definition model SQL model (99/03) Database Definition model SQL Query model

Connectivity Project Provides foundational driver and connection support for data-driven application build in Eclipse.

Connectivity Benefits One place to define drivers One place to define (configure) connections Unified view of data sources Unified view of data

Connectivity Components Driver Management Connection Profiles Data Source Explorer JDBC Connection Support Open Data Access

Driver Management Allows for driver management in Eclipse Template-based Generic Extensible

Connection Profiles Abstraction of “connection” Configuration details Generic, not just for databases Extensible for –Types –Content

Data Source Explorer Exemplary code to demonstrating how to Populate the view from the SQL Model. Use the Navigator View framework to display database in Database Explorer (WTP)

Data Source Explorer cont.. Use the DDL Generator to generate DDL Extract from and load data into a relational table

Data Source Explorer cont.. Filter support on database elements – schema, table, view…

Data Source Explorer cont.. Connect, disconnect, execute/launch SQL Scrapbook and work offline

Data Source Explorer cont.. Sample, open/update, extract from and load data into a relational table

JDBC Connection Support Exemplary code that demonstrates Supporting a generic JDBC catalog loader Extending the wizard framework to support multiple drivers for a single db vendor Specialization of the catalog loading mechanism

Open Data Access (ODA) Previously in BIRT project Provides abstraction over data, like Connection Profile does for connections Define “data drivers” for data types Meta-data access support

SQL Dev Tools Provides a set of frameworks and exemplary tools for using SQL.

SQL Dev Tools Benefits How many text editors do you have to write? –How many SQL editors does Eclipse need? –How about visualization of SQL? Provide a common SQL Query Model framework –Extension of SQL Model –Detailed EMF model representation of SQL query Covers DML: Select, Insert, Update, Delete –Based on SQL with vendor-specific extensions –UML Model -> EMF Utilities -> Generated Code –Extensible SQL source generation Support a SQL DML parser –Based on LPG parser generator –LPG is also used to create the Eclipse Java parser –Targets SQL Query Model directly as parse result –Uses Eclipse extension-point mechanism to support multiple SQL dialects

SQL Query Model/Parser Architecture Client SQL Parser SQL Query Model

SQL Dev Tools Components Frameworks for Routines SQL Query parser SQL Execution Plan

SQL Dev Tools Components Exemplary tools SQL Editor Visual SQL Editor Results View Script History

Routines A “routine” is –Stored procedure –Function Generic, vendor-neutral –Creation, editing, saving –Execution –Debugging

SQL Query Parser Complies with SQL 03 Based on LPG parser generator –LPG is also used to create the Eclipse Java parser Targets SQL Query Model directly as parse result Uses Eclipse extension-point mechanism to support multiple SQL dialects

SQL Execution Plan Execution Plans important for –Understanding database performance –Tuning queries Framework enables –Generic access to execution plans –Ability to visualize in text and (later, perhaps) graphical means

SQL Editor Eclipse text editor for SQL –Syntax colorization –Code assist –Templates Syntax checks and feedback Dialect specialization through underlying SQL Query model

Visual SQL Editor Graphical representation of SQL statements Ability to construct statements without worrying (as much) about SQL syntax As with SQL Editor, dialect specialization supported Smaller brother of “Visual Query Builder” notion floated earlier in DTP

Results View Display results of SQL execution –Uses a Connection Profile instance –Uses a SQL statement Sorting, filtering of view Export of results

Script History Stores queries executed during an Eclipse session Allows for running queries multiple times Thinking about filters based on Connection Profile instances Thinking about persistence across platform instances

In Depth: The Model Base Project

Model Base Themes Using UML to model SQL Model and Database Definition Model, SQL Query Model to support EMF for DTP Published Model: del.htm DBDefinition.htm

Model Base Themes - 1 Tightly integrated and loosely coupled Support the latest SQL Standard

Model Base Themes - 2 Exemplary application built on top of the model to exhibit generic implementation vs. Vendor specific extension

Model Base Themes - 3 Minimum support vs. rich database element support Future model base upgrade

Case Study Supporting richer database features in DTP model framework

Case Study Adding new database element support to SQL Model –Change the SQLModel (Rose model) file to add your database element –Use the Eclipse framework to generate a new EMF model –Generate code to create a new Database Definition Model Generate db vendor document using the provided utility Provide databaseDefinition and uri_mapping extension points a database definition plugin Extend database catalog loader – –Adding class file to handle newly added database elements Extend database DDL Generator – Adding database UI plug-in to support Vendor specific JDBC connection –

Contacting DTP Web site: Newsgroup: news://news.eclipse.org/eclipse.dtp Mailing lists: Wiki : Conferences: –EclipseWorld –EclipseCon –…

Questions?