Data Manager Best Practices Business Intelligence Solutions.

Slides:



Advertisements
Similar presentations
Little Used, but Powerful Features with GP Cathy Fregelette, CPA, PMP Practice Manager BroadPoint Technologies September 20, 2012.
Advertisements

CHAPTER OBJECTIVE: NORMALIZATION THE SNOWFLAKE SCHEMA.
ICIS-NPDES Plugin Design Preview Webinar ICIS-NPDES Full Batch OpenNode2 Plugin Project Presented by Bill Rensmith Windsor Solutions, Inc. 3/15/2012.
AXC01 DIXF: The Microsoft Dynamics AX Data Import and Export Framework
Introduction to ETL Using Microsoft Tools By Dr. Gabriel.
Data Manager Business Intelligence Solutions. Data Mart and Data Warehouse Data Warehouse Architecture Dimensional Data Structure Extract, transform and.
Technical BI Project Lifecycle
Achieving Competitive Advantage and ROI with MetaManager  Metadata Management  Content Enhancements  Standardization  Security and more…
DATA WAREHOUSE DATA MODELLING
Dimensional Modeling Business Intelligence Solutions.
Northwestern University Business Intelligence Solutions Build Cubes in Cognos 8.4.
An Introduction to Dimensional Data Warehouse Design Presented by Joseph J. Sarna Jr. JJS Systems, LLC.
Cognos 8.4 Upgrade Business Intelligence. Why Cognos 8.4 Increased Performance on Database due to optimized SQL and more filters passed in native SQL.
MCTS Guide to Microsoft Windows Server 2008 Network Infrastructure Configuration Chapter 11 Managing and Monitoring a Windows Server 2008 Network.
Data Warehouse Toolkit Introduction. Data Warehouse Bill Inmon's paradigm: Data warehouse is one part of the overall business intelligence system. An.
5 Copyright © 2009, Oracle. All rights reserved. Defining ETL Mappings for Staging Data.
Streams – DataStage Integration InfoSphere Streams Version 3.0
ETL Design and Development Michael A. Fudge, Jr.
ETL By Dr. Gabriel.
BUSINESS INTELLIGENCE/DATA INTEGRATION/ETL/INTEGRATION AN INTRODUCTION Presented by: Gautam Sinha.
Confidential The EB Group, LLC © The EB Group, LLC OBIEE Projects Best Practices Presentation.
Data Warehouse Tools and Technologies - ETL
FireRMS SQL Audit, Archiving & Purging Presented by Laura Small FireRMS Quality Assurance.
SQL Server Integration Services (SSIS) Presented by Tarek Ghazali IT Technical Specialist Microsoft SQL Server (MVP) Microsoft Certified Technology Specialist.
ISV Innovation Presented by ISV Innovation Presented by Business Intelligence Fundamentals: Data Loading Ola Ekdahl IT Mentors 9/12/08.
IMS 6217: Data Warehousing / Business Intelligence Part 3 1 Dr. Lawrence West, Management Dept., University of Central Florida Analysis.
1 The following presentation is from the Oracle Webcast “What’s New in P6 EPPM Release 8.1.” As a partner, you may not use the Oracle Power Point template,
The Business Intelligence Side of Blue Mountain RAM Bill Lucas, IT Systems Architect and Senior Software Engineer.
Session 4: The HANA Curriculum and Demos Dr. Bjarne Berg Associate professor Computer Science Lenoir-Rhyne University.
Access 2013 Microsoft Access 2013 is a database application that is ideal for gathering and understanding data that’s been collected on just about anything.
IT 456 Seminar 5 Dr Jeffrey A Robinson. Overview of Course Week 1 – Introduction Week 2 – Installation of SQL and management Tools Week 3 - Creating and.
ISQS 6339, Data Management and Business Intelligence Cubism – Bells and Whistles Zhangxi Lin Texas Tech University 1.
Data Management Console Synonym Editor
Soup-2-Nuts Alaska Department of Fish & Game Commercial Fisheries October, 2011.
Oracle Data Integrator Transformations: Adding More Complexity
Database Design and Management CPTG /23/2015Chapter 12 of 38 Functions of a Database Store data Store data School: student records, class schedules,
Carey Probst Technical Director Technology Business Unit - OLAP Oracle Corporation.
December 5, Repository Metadata: Tips and Tricks Peggy Rodriguez, Kathy Kimball.
Transportation: Loading Warehouse Data Chapter 12.
Copyright 2007, Paradigm Publishing Inc. ACCESS 2007 Chapter 3 BACKNEXTEND 3-1 LINKS TO OBJECTIVES Modify a Table – Add, Delete, Move Fields Modify a Table.
Data Staging Data Loading and Cleaning Marakas pg. 25 BCIS 4660 Spring 2012.
Oracle Data Integrator Data Quality (Integrity Control)
Metadata By N.Gopinath AP/CSE Metadata and it’s role in the lifecycle. The collection, maintenance, and deployment of metadata Metadata and tool integration.
7 Strategies for Extracting, Transforming, and Loading.
BI Practice March-2006 COGNOS 8BI TOOLS COGNOS 8 Framework Manager TATA CONSULTANCY SERVICES SEEPZ, Mumbai.
© 2012 Saturn Infotech. All Rights Reserved. Oracle Hyperion Data Relationship Management Presented by: Prasad Bhavsar Saturn Infotech, Inc.
Student Centered ODS ETL Processing. Insert Search for rows not previously in the database within a snapshot type for a specific subject and year Search.
20 Copyright © 2008, Oracle. All rights reserved. Cache Management.
1 Copyright © 2009, Oracle. All rights reserved. I Course Introduction.
Unit 8.2 Learning Objectives Data Warehouses – The Role of Data Warehouses The Role of Data Warehouses – Group Exercise Accessing Data in Views – Accessing.
Oracle Business Intelligence Foundation – Testing and Deploying OBI Repository.
SSMS SQL Server Management System. SQL Server Microsoft SQL Server is a Relational Database Management System (RDBMS) Relational Database Management System.
Day in the Life (DITL) Production Operations with Energy Builder Copyright © 2015 EDataViz LLC.
INCREMENTAL AGGREGATION After you create a session that includes an Aggregator transformation, you can enable the session option, Incremental Aggregation.
Physical Layer of a Repository. March 6, 2009 Agenda – What is a Repository? –What is meant by Physical Layer? –Data Source, Connection Pool, Tables and.
Copyright © 2006, Oracle. All rights reserved. Czinkóczki László oktató Using the Oracle Warehouse Builder.
Building the Corporate Data Warehouse Pindaro Demertzoglou Data Resource Management.
7 Copyright © 2006, Oracle. All rights reserved. Defining a Relational Dimensional Model.
DO YOU TRUST YOUR DATA? KNOW THE ANSWER WITH EIM! Jose Hernandez Director, Business Intelligence Dunn Solutions Group.
Building the Corporate Data Warehouse Pindaro Demertzoglou Lally School of Management Data Resource Management.
1 Copyright © 2008, Oracle. All rights reserved. Repository Basics.
Data Warehousing/Loading the DW—Topics
Creating Repositories from Multidimensional Data Sources
IBM COGNOS online Training at GoLogica Technologies
IBM DATASTAGE online Training at GoLogica
SSIS Demo Michael A. Fudge, Jr.
Open BIDS and create new SSAS Project Right Click on Data Source and click on New Data Source.
Analysis Services Analysis Services vs. the Data Warehouse vs. OLTP DB
Data Warehousing/Loading the DW—Topics
Presentation transcript:

Data Manager Best Practices Business Intelligence Solutions

ETL Catalog and Data Mart should be stored in different schemas A Data manager catalog provides a central repository for the information that defines how Data Manager extracts, transforms and delivers data. The catalog stores Data Manager Builds, connection specifications, Job Streams, user-defined functions and the dimensional framework. Development/Test/Production Catalog Tables Source Connection Source Connection Target Connection Target Connection

Development, Test, Production schemas, migration and version control strategy We recommend that you create a dedicated schema for each of your environments: development, test and production. 1.All builds should be created in the development catalog first. Backup the catalog every day before leave and check in to subversion/CVS. 2.When the development is done, the dev catalog can be backed up and restored to test catalog. Make sure to modify the target database connection in test catalog. QA/Testers validate the data in test schema and open ticket if any problem is found. Developers fix problems in Dev catalog and push to Test for the next release. This process may be repeated in several iterations. Check in the latest Test catalog in to subversion/CVS. 3.When the testing is done, backup test catalog and restore to production schema and modify the target database connection to production. Check in the latest production catalog in to subversion/CVS.

Automate the ETL, Deploy and Schedule the job -- A JobStream can multi-task events and allow commands to be executed in a parallel or serial manner. -- The developed JobStream can be published as Data Movement tasks into the IBM Cognos BI production environment, where they can be added to jobs and be scheduled for execution.

Create builds 1. Dimension Builds You can create complicated hierarchy during the “Create hierarchy” step SCDs can be easily defined at the “Create dimension build” step Lookups can be created after the reference dimension is completed. The lookup is used in fact build to load SKEYs from dimension tables based on the business key. Reference dimension is also used to handle unmatched numbers in fact build Insert level(s) for the hierarchy Create hierarchy for the reference dimension Insert reference dimension Create dimension build using the reference dimension Insert lookups In the reference dimension

2. Fact Builds Fact build can be easily created using the wizard. Lookups can be added in the Reference tab of the Transformation Model. It can replace the business key with the surrogate key in dimension for you automatically. Make sure to check the “Use surrogates when available” checkbox to enable this function. Late Arriving Facts can be handled in fact build. Three ways to handle unmatched members: -- accept unmatched number identifiers and save unmatched member details via reference structure. -- accept unmatched number identifiers. These identifiers will be stored in the catalog and will be loaded when your corresponding dimension build run next time. -- reject those unmatched number identifiers.

Customized Refresh strategies in the Fact build

Debugging Steps The following is an example on debugging ETL issue and solving the problem in data manager. JIRA issue: ETL is not pulling DIM_ALLOCATION.cfae_purpose_code correctly Description: According to OARD_source_to_target_maps.xls, DIM_ALLOCATION.cfae_purpose_code should be pulled from ALLOCATION.cfae_purpose_code. I found that the cfae_purpose_code in our target table is not match the ones in AIMS source table. 1. Check the mapping file to verify what exactly cfae_purpose_code is pulled from and find out if there are any transformation on this column 2. Run query ( or spot check) to verify the problem 3. If the problem is confirmed, check the query used in DM to pull this column. Run “retrieve 1 row” to verify if data in this column is correct. If it’s wrong, copy the query to Toad, debug this query and fix the problem in the query 4. If the data in the column retrieved by step 3 is correct, then the query used by DM in this build is correct. Check the DataStream to see if this Data Source is mapped correctly. If the mapping is incorrect, fix the mapping. If the fix in Data Stream affect Hierarchy, it’s level(s) and templates, modify them accordingly. 5. If the Data Stream is correctly mapped, then check the mapping in Hierarchy. Fix it here if any problem is found. 6. If you can’t find any problem from step 3-5, there is no problem in reference dimension. Go to check the dimension build. 7. In the dimension build, check the template to see if anything is defined properly. If any problem is found, fix it here. 8. If nothing wrong in step 7, check the mapping in the Dimension Table Properties. If any problem is found, fix it.

9. Hooray!!! You fixed the problem!!!

Useful Sources and References –Kimball, Ralph; et al. The Data Warehouse Lifecycle Toolkit. Wiley –Kimball, Ralph; Margy Ross. The Data Warehouse Toolkit: The Complete Guide to Dimensional Modeling. Wiley. –Kimball, Ralph; Joe Caserta. The Data Warehouse ETL Toolkit. Wiley – – – –

Question & Answer