Welcome to the IBM IS Tech Talk Virtual Tables in Information Analyzer Oct 20, 2016 Welcome to the IBM IS Tech Talk Virtual Tables in Information Analyzer <TBD Add Kimberly’s talk here>

Oct 20 2016 Virtual Tables in Information Analyzer Suresh Tirumalasetti – Software Engineer Hello, my name is Suresh Tirumalasetti I am part of Information Analyzer Product Development team Today’s Tech Talk will focus on the Virtual Tables in Information Analyzer

When to use Virtual Tables ? Types of Virtual Tables aGENDA What is Virtual Table ? When to use Virtual Tables ? Types of Virtual Tables How to create Virtual Tables ? Demonstration Q&A <quickly and briefly touch on today’s agenda>

Analytics Platform Services - Disclaimer IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. Any mention today of plans, futures, etc. is subject to this disclaimer

How to ask questions in the chat 5 IBM Confidential How to ask questions in the chat Click the icon on the left shown in the red box At the bottom of the screen, type you question One of our SMEs will try to provide a reply ASAP

A virtual table is to segment data from one or more tables. 6 IBM Confidential What is a Virtual Table ? A virtual table is to segment data from one or more tables. Such segmentation can be Vertical (either a subset or superset of columns based on a selection of chosen columns) or Horizontal (an explicit set of rows or records based on a conditional expression) or Both. Others (Aggregations / Joins) In short, it helps to filter the data of user interest User can then run subsequent analysis (for example, column. cross-domain, data rule etc.) on this virtual table.

When to use Virtual Tables ? 7 IBM Confidential When to use Virtual Tables ? I like to filter my data using simple/complex queries before I really analyze/profile the data. I have enough Database resources. I already have SQL query based Rules, I need quickly migrate them to Information Analyzer and like to add more IA specific rules and integrate them.

Types of Virtual Tables 8 IBM Confidential Types of Virtual Tables Classical Virtual Table SQL Virtual Table Create using IA workbench / CLI / REST API CLI / REST API Available from 8.1.1 11.5 Scope One table Across many tables Complexity Limited (Select set of columns, Provide where condition) High (Any valid SQL that can be executed against repository)

Classical Virtual Table 9 IBM Confidential Classical Virtual Table Created by selecting rows using a simple criteria and selecting columns from the columns in the table Can filter the number of rows and select columns to produce a much smaller and focused table on which analysis and rules can be applied No column manipulation is available No complex SQL functions

The SQL can include multi-table joins and nearly any valid SQL 1010 IBM Confidential SQL Virtual Table A virtual table based on a customized SQL Query can be created from the command-line interface by using an IAAdmin script or the HTTP REST API Users can perform column analysis, key analysis, and data rules analysis etc. for a SQL Virtual table like as with any other data source. The SQL can include multi-table joins and nearly any valid SQL Users are responsible for the validation of the SQL The name and logical type of the columns expected in the result set must be known and specified when creating the SQL virtual table

SQL Virtual Table contd… 1111 IBM Confidential SQL Virtual Table contd… A SQL virtual table can dramatically simplify the logic required in InfoSphere Information Analyzer data rules SQL virtual tables can be used to implement SQL rules when the user already has a rule expressed as a SQL statement and want to execute the same in an IA rule but moving to the IA rule expression is not an option create a SQL virtual table with the SQL statement to execute

SQL Virtual Table contd… 1212 IBM Confidential SQL Virtual Table contd… A SQL VT is similar to a database view which enhance the select statements with the following addl. Capabilities: conditions: Select Distinct Joins Counts (*) Group By’s Dynamic SQL Queries or views based upon other values within another table Consider the following example: Select * from Portfolio_table where effective_date = (select run_date from batch_run_date_Table)

Create SQL Table from CLI 1313 IBM Confidential Create SQL Table from CLI The definition of SQL virtual table is pretty much similar to a classical virtual table in project xml input A SQL virtual table can be created using IAAdmin -create / - update commands Ex: /opt/IBM/InformationServer/ASBNode/bin/IAAdmin.sh -url https://server:port -user admin -password admin -create - projectContent /tmp/createSQLVT.xml ** encrypted password can be used instead of a plain text password

Input XML for Classical Virtual Table for REST API 1414 IBM Confidential Input XML for Classical Virtual Table for REST API <?xml version="1.0" encoding="UTF-8"?> <iaapi:Project xmlns:iaapi="http://www.ibm.com/investigate/api/iaapi" name="JK_BANK"> <description /> <DataSources> <DataSource> <Schema name="JK_BANK1"> <VirtualTable name="ClasssicalVT" baseTable="BANK_ACCOUNTS"> <description>My Classical VT</description> <WhereCondition>ACCOUNT_ID>200000</WhereCondition> <Column name="ACCOUNT_ID" /> <Column name="BANK_BALANCE" /> <Column name="ADDR2" /> <Column name="ADDR1" /> <Column name="ACCOUNT_TYPE" /> </VirtualTable> </Schema> </DataSource> </DataSources> </iaapi:Project>

Input XML for SQL Virtual Table for REST API 1515 IBM Confidential Input XML for SQL Virtual Table for REST API

Input XML for SQL Virtual Table for REST API 1616 IBM Confidential Input XML for SQL Virtual Table for REST API <?xml version="1.0" encoding="UTF-8"?> <iaapi:Project xmlns:iaapi="http://www.ibm.com/investigate/api/iaapi" name="JK_BANK"> <description /> <DataSources> <DataSource> <Schema name="JK_BANK1"> <VirtualTable name="SQLVT_2"> <SelectStatement><![CDATA[SELECT bb.branch_id, bb.branch_address, bb.branch_city, bb.branch_state, bc.ss_num, bc.NAME FROM jk_bank2.bank_branch bb INNER JOIN jk_bank2.bank_checking bc ON bb.branch_id = bc.account_branch]]></SelectStatement> <Column name="BRANCH_ID" length="10" type="int32" /> <Column name="BRANCH_ADDRESS“ length="128" type="string" /> <Column name="BRANCH_CITY" length="128" type="string" /> <Column name="BRANCH_STATE" length="5" type="string" /> <Column name="SS_NUM“ length="12" type="string" /> <Column name="NAME" length="128" type="string" /> </VirtualTable> </Schema> </DataSource> </DataSources> </iaapi:Project> Select Statement sent to the source Datatype and length are required for the columns mentioned in the select statement

Virtual Tables are applicable for Relational databases ONLY 1717 IBM Confidential Some facts… Virtual Tables are applicable for Relational databases ONLY Virtual Tables "are not shown or supported today" in the IA Thin Client

Live Demo Time for live demo I will be using the IA Workbench for the demo because, as I stated earlier, the SQL VTs are not supported today in the IA thin client

Q&A Open up to read and respond to ‘chat’ questions submitted during the talk

Thank You Thank you for your time and participation today!

Demo Backup Screens

Procedure to create Classical Virtual Table in Workbench 2222 IBM Confidential Procedure to create Classical Virtual Table in Workbench On the Column Analysis workspace of the console, select the base table from which to create virtual table and click Create Virtual Table. In the Name field, type a name for the virtual table. Optional: In the Short Description field, type a short description. For example, you might want to provide a reason for the specific view. Optional: In the Long Description field, type a longer description. Select one or more columns from the Available Columns list. Click the arrow button to move the selected columns into the Selected Columns list. Optional: Modify the order of the selected columns by using the up and down arrows. Optional: To define filters for the selected columns, click Add Condition to select the following 'where' clause parameters: open parenthesis, a column name, an operator, a value, close parenthesis, and an And or Or Boolean operator. Click Save.

Creation of Classical Virtual Table in Workbench

Creation of Classical Virtual Table in Workbench

Creation of Classical Virtual Table in Workbench

Creation of Classical Virtual Table in Workbench

Procedure to create a SQL based Data Rule 2727 IBM Confidential Procedure to create a SQL based Data Rule Define a dummy rule logic "col exists" Define a rule binding that dummy logic to the SQL virtual table Set the output of the rule to "Passing Records“ Add all the columns of the SQL VT in the rule output Now, you have an IA rule which can be scheduled and executed like a native IA rule but will execute the SQL Combining the advantages of the SQL and the IA rules results are kept in history, you can use user named tables, etc...

A tool to create input XML for SQL Virtual table 2828 IBM Confidential A tool to create input XML for SQL Virtual table Given that we need additional details like Datatype, Length attributes for each column in the datasource in input XML A simple customizable java program helps users to define such XML

IA SQLVT Helper

IA REST API

IA Project Properties

IA Column Analysis

IA Data Rule

IA Data Rule