Business Intelligence Methodology 1/3/2012

Public Information. Copyright 2012 – Instant Business Intelligence. 2 Introduction  History of the Methodology  Overview of the Methodology  Discussion points on the Methodology  Summary

Public Information. Copyright 2012 – Instant Business Intelligence. 3 History of the Methodology  First project Peter undertook in 1991 was a disaster  “Everything I know should stand me in good stead”. BIG mistake!  In 1993 introduced to the Metaphor Consulting Methodology  Based on a book called “The Consultants Methodology Handbook” or similar  This contained a lot of great ideas and pointers to great ideas  In 1995 asked to write a BI Methodology by SAS  In 1997/8 Exposed to the PwC BI Methodology  In 1999/00 Exposed to the Prism Iterations Methodology  In 2001/05 Exposed to the Sybase Systems Integrators Methodology  They all have strengths and weaknesses  Even with the advent of SeETL/BI4ALL in 2005/11 we have been working to these older techniques.  Our experience and tools give us a new opportunity to change

Public Information. Copyright 2012 – Instant Business Intelligence. 4 Overview of the Methodology  We have made some “radical changes” in processes  Our experience and tools allow us to work smarter  SeETL now allows us to map far more data than ever before  The data models allow us to design databases faster than ever before  The data warehouses are covering more areas than ever before  This is created a new problem…too much data to comprehend  We now focus on understanding data, all other issues are solved  Our experience also tells us what is most important in our major industry areas like telco and retail  We have moved understanding data to the front of the project  We have created a far greater focus on prototyping  We have moved requirements back in the process

Public Information. Copyright 2012 – Instant Business Intelligence. 5 Methodology Phases  Phase A - Prototype, Presentation and Proposal  Phase B - Tender Response  Phase C – IBI Internal Project Review  All Phases - Project Management  Phase 1 – Hardware/Software Installation for Pilot  Phase 2 – Source Data Cataloguing, Profiling, Analysis  Phase 3 - Requirements Gathering  Phase 4 - Extract Subsystem Design

Public Information. Copyright 2012 – Instant Business Intelligence. 6 Methodology Phases  Phase 5 - Detailed Data Warehouse Analytical Apps Design/Build  Phase 5A - Data Warehouse Database Design/Build  Phase 5B - Data Preparation and Loading Design/Build  Phase 5C Implement Initial End User Applications  Phase 5D – Pilot System Test  Phase 5E – Initial End User Training  Phase 6 – Pilot Implementation  Phase 6A – Pilot Implementation  Phase 6B – Pilot Review, Analysis, Updates  Phase 7 – ETL Migration and Testing

Public Information. Copyright 2012 – Instant Business Intelligence. 7 Methodology Phases  Phase 8 – Hardware/Software Installation for Rollout  Phase 9 – Scale Up – Data Warehouse Volumes  Phase 10 – End User Training  Phase 11 – Scale Up – Roll Out to End Users  Phase 12 – Data Warehouse Exploitation Projects  Phase 13 – IBI Client Project Review

Public Information. Copyright 2012 – Instant Business Intelligence. 8 Discussion Points  The number and diversity of fields is now the #1 problem  Integration of the data is the #2 problem  Fast databases like Netezza and Sysbase IQ mean that sample data can be loaded very fast and analysed very fast.  We have moved data analysis to the front of the process. Note.  Phase 1 – Hardware/Software Installation for Pilot  Phase 2 – Source Data Cataloguing, Profiling, Analysis  The idea is to get production level volumes of data into the staging area as soon as possible and run the new data profiling tools  Only after the data has been profiled and come to be understood do we go into requirements  Phase 3 - Requirements Gathering  In industries like telco, retail, web, media a great deal is already known as to industry standard requirements

Public Information. Copyright 2012 – Instant Business Intelligence. 9 Discussion Points  We now build the base layer of the data warehouse according to the BI4ALL data models as non lossy  We then build derivation fact tables as needed  “Everything is connected to everything” builds a “mesh” of joins between all the transaction level fact tables and many summaries too  We can start on this work prior to Requirements Gathering as well  The speed of mapping development and prototype development means that 4,000 or so data fields is quite feasible for a 1.0 DW  We propose Stored Procedures as optional extra to turn the data warehouse into a Q&A machine. SPs being able to be called from any tool and for the results to be reliable and consistent  We propose the business people are heavily involved in the prototyping stage and that full production volumes be used

Public Information. Copyright 2012 – Instant Business Intelligence. 10 Discussion Points  Once the prototypes are accepted and agreed to be deployed?  Then we migrate the ETL to the “standard” if SeETL is not to be used.  The report development can start as soon as early portions of the ETL and data model are delivered  It is far easier to accommodate change  Later releases are far easier to accommodate  The Metadata dictionary is used throughout for control and analysis  Data linearage can be established via the Metadata dictionary  The following two diagrams explain the situation in more detail

Public Information. Copyright 2012 – Instant Business Intelligence. 11 Different Approaches Reqts Model Design Data Mapping ETL Design/Build Apps Build Each phase must produce ‘cast in concrete’ outputs as they cannot be easily changed Changes we would like to make later in the project go into Release 2.0 or never get implemented Reqts Model Design Data Mapping ETL Design/Build Apps Build We retain the ability to change the database/ETL We do all modelling, mapping, ETL design/build and a lot of apps development at the same time with the ability to generate all ETL related objects. Convert SeETL to Tool Changes Traditional ETL approach Approach Using prototype tools - SeETL and BI4ALL Some vendors overlap activities and do iterations of smaller projects. No longer on critical path

Public Information. Copyright 2012 – Instant Business Intelligence. 12 A Perspective on BI Modeling Techniques And ETL Complexity   3NF Models   Complex to query   Little or no history   Cartesian Products   Lost Rows   No time variance in the model itself   Limited use   Breakthrough of their time Where we Started Data Model Sophistication and Functionality LessMore Thought Leadership now Industry ‘Leading Edge’ now Industry ‘Leading Edge’ now Industry ‘Best Practice’ now Trailing Edge   3NF Models   Time Variance + Stability Analysis   Complex to query   Lots of history   Cartesian Products   Lost Rows   Great archives   Really useful   Eg NCR Models   Leading companies doing dimensional models (Metaphor)   Combine 3NF + TV + SA and dimensional models   Rich history   Great archives   Great performance   No gaps   No Cartesian products   No lossy joins   Life is good   DWs are expensive   Eg IBM Models   No 3NF data anywhere   Archive using dimensional models   Functionally equivalent but no archive layer required   Suffers slow down of history in type 2 dimensions   What Ralph Kimball talks about   Has evolved since   No 3NF data anywhere   Archive using dimensional models   Functionally equivalent but no archive layer required   Significant reuse of tables   Field names have meaning   Data types have meaning   Eg Sybase Models   No 3NF data anywhere   Archive using dimensional models   Functionally equivalent but no archive layer required   Very high reuse of tables   Field names are meaningless   Data types are meaningless   Eg BI4ALL ETL Complexity Generally speaking LessMore

Public Information. Copyright 2012 – Instant Business Intelligence. 13 Summary  History of the Methodology  Overview of the Methodology  Discussion Points  Summary