Data Virtualization & Information As A Service (IaaS) By Anil Allewar Senior Solutions Architect - Synerzip 1

About Me!! Confidential2 Anil Allewar Senior Solutions Synerzip Technology Evangelist & speaker Core interests: JEE, EAI, EII

Use cases Agenda What does it mean? Implementation Frameworks Demo Questions? 3 Architecture explained

Why it makes sense? Confidential4

Use Cases Data Warehouse ETL Financial Data OLTP Data ETL 3 rd Party Data Data Mart ETL Web Service 1 Web Service 2 Legacy Data Custom Program Excel files Confidential5

Traditional Data Integration Enterprise Information SystemETL Source System ETL Confidential6 Business Applications

Problems with ETL Confidential7 More than 1 copy of data for staging Intermediate data => Errors Lead time to add new source Domain knowledge for mapping Batch Process => No real time data

Problems with DBMS consolidation Confidential8 Alternate approach => Single EIS (say RDBMS) Extensive changes to existing apps Might not satisfy everyone’s requiremets

Use cases Agenda What does it mean? Implementation Frameworks Demo Questions? 9 Architecture explained

Data Virtualization & Federation Confidential10 Single API to access data Only metadata stored at virtualization layer Real time access without copying/moving data Federate data across hetero/homogenous sources

Data Virtualization Confidential11

Use cases Agenda What does it mean? Implementation Frameworks Demo Questions? 12 Architecture explained

Architecture Confidential13 User Application Common Access API Common Access API Connector 1 Connector 2 RUNTIME & QUERY ENGINE Virtual Database Virtual Database Translator 1 Translator 2

Use cases Agenda What does it mean? Implementation Frameworks Demo Questions? 14 Architecture explained

Vendors Commercial Products –Composite Software –Denodo –IBM –Informatica –Red Hat Open Source –Jboss Teiid Confidential15

Selected Platform – JBoss Teiid Confidential16 Open Source Number of relational/NoSQL/E RP/CRM data stores JEE standards Add custom EIS support using JEE components Active & responsive community Synerzip contribution: Defect discovery, root cause analysis, feature verification

Teiid Components Virtual Database –container for components used to integrate data from multiple data sources Source Models –structure and characteristics of physical data sources View Models –structure and characteristics of abstract structures you want to expose to your applications Teiid Designer –Eclipse based UI to dynamically discover data source objects and apply data federation –Generate virtual database from 1 or more sources Confidential17

Teiid Components Translator –Provides abstraction later between Teiid Query Engine and source system –Convert Teiid SQL commands to source specific execution commands –Convert result data from source system to Teiid specific format Resource Adapter –Provides connectivity to the physical data source –Integration provided through Java Connector Architecture (JCA) API Confidential18

Teiid – Supported EIS Amazon SimpleDB Apache Accumulo Apache SOLR Cassandra File Google Spreadsheet JPA LDAP Excel – as file SalesForce JDBC – MS access, DB2, derby, excel- odbc, greenplum, h2, hive(for accessing Hadoop), oracle, teradata and most RDBMS MongoDB Object OData OLAP Web Services SAP Netweaver Gateway Confidential19

Performance Characteristics Access same data using Oracle and Teiid drivers –Retrieval times comparable when accessing tables having no Blobs Confidential20 ms

Performance Characteristics –Teiid slower when accessing Blob data Can be tuned Confidential21

Use cases Agenda What does it mean? Implementation Frameworks Demo Questions? 22 Architecture explained

Demo Confidential23 JDBC Client JDBC API JDBC API RDBMS Resource Adapter MongoDB Resource Adapter TEIID RUNTIME & QUERY ENGINE Federated VDB mySQL Translator MongoDB Translator mySQL

Demo-Steps Pre-requisites –mySQL server 5.5+ installed –MongoDB 2.4.x+ installed Steps –Load the mySql and MongoDB database with sample data –Setup environment – JBoss, Eclipse –Create Teiid project in Eclipse using Teiid designer Import source model using JDBC Create the virtual model and federate data from the source model Create a virtual database (VDB) and deploy to JBoss –Access data using JDBC client or through browser using OData Confidential24

Demo – Scenario Confidential25 Federated Data Federated Data

Demo – Connection Profile Confidential26

Demo – Source Model Confidential27

Demo - Source Model Generation Confidential28

Demo – Map Source To View Confidential29

Demo - Association Confidential30

Demo – Data Federation Confidential31

Demo – Source Code Source code – Teiidhttps://github.com/Synerzip/JBoss- Teiid –Contains Configuration files Instructions “How-to” videos VDBs, source models and view models Confidential32

Conclusion Data Virtualization and Federation is a rapidly emerging technology that solves traditional BI/ETL problems. It provides lower time to market, distributes data across the enterprise as a service and provides real time access to enterprise data. Confidential33

Use cases Agenda What does it mean? Implementation Frameworks Demo Questions? 34 Architecture explained

Contact Me Confidential35

Questions? Hemant Elhence

Synerzip in a Nutshell 1.Software product development partner for small/mid-sized technology companies Exclusive focus on small/mid-sized technology companies, typically venture-backed companies in growth phase By definition, all Synerzip work is the IP of its respective clients Deep experience in full SDLC – design, dev, QA/testing, deployment 2.Dedicated team of high caliber software professionals for each client Seamlessly extends client’s local team, offering full transparency Stable teams with very low turn-over NOT just “staff augmentation”, but provide full mgmt support 3.Actually reduces risk of development/delivery Experienced team - uses appropriate level of engineering discipline Practices Agile development – responsive, yet disciplined 4.Reduces cost – dual-shore team, 50% cost advantage 5.Offers long term flexibility – allows (facilitates) taking offshore team captive – aka “BOT” option

Our Clients

Call Us for a Free Consultation! Hemant Elhence Thanks!