Dr. Bjarne Berg February 16 th, 2006 ERP Data Warehousing - Architecture, Tools and trends An overview presentation University of North Carolina at Charlotte.

Slides:



Advertisements
Similar presentations
Chapter 1 Business Driven Technology
Advertisements

Supporting End-User Access
Data Warehouse Architecture Sakthi Angappamudali Data Architect, The Oregon State University, Corvallis 16 th May, 2005.
Integrated planning  Understand integrated planning process  Describe SAP integrated planning technology or tool (SAP-IP)
© 2005 Wellesley Information Services. All rights reserved. Build the ultimate enterprise data warehouse using SAP NetWeaver A look at JAD, RAD, XP, and.
0 © 2010 Wellesley Information Services. All rights reserved. Crystal Dashboard Builder, Web application designer, or SAP NetWeaver Visual Composer Dr.
Copyrights 2002 Introduction to SAP Enterprise Portals September SAP Enterprise Portal 101 Naeem Hashmi Chief Technology Officer Information Frameworks.
MS DB Proposal Scott Canaan B. Thomas Golisano College of Computing & Information Sciences.
Components and Architecture CS 543 – Data Warehousing.
Accelerated Access to BW Al Weedman Idea Integration.
WORKDAY TECHNOLOGY Stan Swete CTO - Workday 1.
ERP SYSTEMS 4:00-4:15. WHAT IS ERP?  Enterprise (E) Resource (R) Planning (P)  A set of integrated software modules for supporting all of an enterprises.
CS2032 DATA WAREHOUSING AND DATA MINING
Business Intelligence System September 2013 BI.
Introduction to Building a BI Solution 권오주 OLAPForum
Business Intelligence components Introduction. Microsoft® SQL Server™ 2005 is a complete business intelligence (BI) platform that provides the features,
Dr. Bjarne Berg Lenoir-Rhyne College
Knowledge Portals and Knowledge Management Tools
Introduction to Systems Analysis and Design
Center of Excellence for IT at Bellevue College. IT-enabled business decision making based on simple to complex data analysis processes  Database development.
MANAGEMENT SUPPORT SYSTEMS II 2. Enterprise Decision Support Systems 1.
Data Warehouse Toolkit Introduction. Data Warehouse Bill Inmon's paradigm: Data warehouse is one part of the overall business intelligence system. An.
Designing a Data Warehouse
Data Warehousing: Defined and Its Applications Pete Johnson April 2002.
Lecture-8/ T. Nouf Almujally
1 Components of A Successful Data Warehouse Chris Wheaton, Co-Founder, Client Advocate.
Enterprise Resource Planning BI: A Look at Capabilities and Value Propositions Wednesday, August 16, 2006, 4:30 PM - 5:30 PM Bjarne Berg, Director, Business.
LOGO Business Intelligence System Mr. Natapong Wongprommoon Solution Architect G-ABLE Company Limited
Introduction to Enterprise Systems
Basic Concepts of Datawarehousing An Overview Prasanth Gurram.
Understanding Data Warehousing
Data Warehousing at STC MSIS 2007 Geneva, May 8-10, 2007 Karen Doherty Director General Informatics Branch Statistics Canada.
Page 1 GADD Software & GADD Analytics 1.5 Public version, January 2015, gaddsoftware.com GADD Analytics.
Data Warehouse Overview September 28, 2012 presented by Terry Bilskie.
Bjarne Berg March 22 th, 2006 A billing analysis Data Warehouse University of North Carolina at Charlotte.
Enterprise Reporting Solution
Data warehousing and online analytical processing- Ref Chap 4) By Asst Prof. Muhammad Amir Alam.
MIS DATABASE SYSTEMS, DATA WAREHOUSES, AND DATA MARTS CHAPTER 3
Managing Knowledge in Business Intelligence Systems Dr. Jan Mrazek.
Bjarne Berg and Ben Perego March 20 th, 2006 Decision Support - What is new? An overview presentation University of North Carolina at Charlotte.
SAP Decision Support Environments in Higher Education
1 Reviewing Data Warehouse Basics. Lessons 1.Reviewing Data Warehouse Basics 2.Defining the Business and Logical Models 3.Creating the Dimensional Model.
Datawarehouse A sneak preview. 2 Data Warehouse Approach An old idea with a new interest: Cheap Computing Power Special Purpose Hardware New Data Structures.
By N.Gopinath AP/CSE. There are 5 categories of Decision support tools, They are; 1. Reporting 2. Managed Query 3. Executive Information Systems 4. OLAP.
Building Dashboards SharePoint and Business Intelligence.
Reporting & Analytics Stephen Chan Senior Solution Consultant.
Chapter 2 Introduction to Enterprise Systems Partial adoption from Magal and Word | Integrated Business Processes with ERP Systems | © 2011 Timothy L.
Rajesh Bhat Director, PLM Analytics Applications
1 Copyright © Oracle Corporation, All rights reserved. Business Intelligence and Data Warehousing.
Chapter 8: Data Warehousing. Data Warehouse Defined A physical repository where relational data are specially organized to provide enterprise- wide, cleansed.
© SAP SE ZR Reporting Abstract This unit focuses on different reporting techniques as supported by the SAP software and SAP Business Intelligence. It is.
SAP NetWeaver Business Intelligence SAP Netweaver Business Warehouse (SAP NetWeaver BW) the name of the Business Intelligence,
Data Mining and Data Warehousing: Concepts and Techniques What is a Data Warehouse? Data Warehouse vs. other systems, OLTP vs. OLAP Conceptual Modeling.
Business Intelligence Overview
Decision Support Systems
Defining Data Warehouse Concepts and Terminology
with the Microsoft BI Ecosystem
Data Warehouse.
Defining Data Warehouse Concepts and Terminology
Enterprise Resource Planning BI: A Look at Capabilities and Value Propositions Wednesday, August 16, 2006, 4:30 PM - 5:30 PM Bjarne Berg, Director,
Business Intelligence for Project Server/Online
Business Intelligence
Enterprise Resource Planning BI: A Look at Capabilities and Value Propositions Wednesday, August 16, 2006, 4:30 PM - 5:30 PM Bjarne Berg, Director,
Data Warehouse Overview September 28, 2012 presented by Terry Bilskie
Enterprise Resource Planning BI: A Look at Capabilities and Value Propositions Wednesday, August 16, 2006, 4:30 PM - 5:30 PM Bjarne Berg, Director,
Delivering an End-to-End Business Intelligence Solution
Supporting End-User Access
Introduction of Week 9 Return assignment 5-2
Data Warehousing Concepts
Presentation transcript:

Dr. Bjarne Berg February 16 th, 2006 ERP Data Warehousing - Architecture, Tools and trends An overview presentation University of North Carolina at Charlotte

2 What We will cover Enterprise Data Warehouse and BI Data Architecture ERP BI Example – SAP Business Warehouse ERP Data Warehouse Content and Methodologies Data Warehouse Presentation tools Post-implementation Reviews

3 Source: SAP What Do Users Really Want?

Logical Enterprise DW Architecture Metadata Data Extraction Integration and Cleansing Processes Custom Developed Applications Data Mining Statistical Programs Query Access Tools Data Resource Management and Quality Assurance Summarized Data Segmented Data Subsets Functional Area Summation Marketing and Sales Purchasing Corporate Information Product Line Location Purchasing Systems Invoicing Systems General Ledger External Data Sources Other Internal Systems Translate Attribute Calculate Derive Summarize Synchronize Source DataExtract Operational Data Store Transform Data Warehouse Applications Source: Bjarne Berg, “Introduction to Data Warehousing”, Price Waterhouse Global System solution Center, 1997

5 William H. Inmon’s Corporate Information Factory (CIF)

specific application or workgroup focus narrow scope customized or stand alone analysis interactive query highly summarized single subject and department oriented uses dimensional data modeling Functional Systems: Characteristics Operational Data StoreData WarehouseData Mart acts as source to populate DW and marts Often used for operational reporting detailed, atomic data huge data volumes integrated, clean data cross-functional and cross- departmental supports data mining may use third normal form modeling (NOT dimensional) provides mgmt reporting summarized data tuned to optimize query performance multiple departments or processes staging area for data marts uses dimensional data modeling

7 Level of Pre-delivered Content Toolsets & accelerators Analytical applications for specific industries Level of Embedded Analytics Complex (score cards, budgeting, planning, KPI) Interactive Mgmt. reporting (OLAP, MQE) Evolution of ERP Data Warehousing Emerging (1st generation) Vertical approach (2nd generation) Horizontal approach (2nd generation) Integrated analytical (3rd generation) Source: Mike Schroeck, David Zinn and Bjarne Berg, “Integrated Analytics – Getting Increased Value from Enterprise Resource Planning Systems”, Data Management Review, May, 2002; Adapted: Bjarne Berg “How to Manage a BW Project”, BW & Portals Conference, 2004, Orlando

8 What We will cover Enterprise Data Warehouse and BI Data Architecture ERP BI Example – SAP Business Warehouse ERP Data Warehouse Content and Methodologies Data Warehouse Presentation tools Post-implementation Reviews

9 All major ERP vendors have packaged DSS solutions SAP, Baan, PeopleSoft, Oracle, JD Edwards and others have all packaged Data warehouse solutions. This is commonly known as Business Intelligence (BI) products. There are many types of BI products: 1.Data Warehouses (DW) 2.Strategic Enterprise management (SEM) 1.Corporate Performance mgmt (CPM) 2.Business planning & simulation (BPS) 3.Business Consolidation services (BCS) 4.Stakeholder relationship mgmt (SRM) 3.Data Mining 4.Advanced Planning and Optimization (APO) 5.Supply Chain Event Manager (SCEM) 6.Customer Relationship Manager (CRM) The DW is the source of the data for all these integrated analytical applications (iAnalytics)

10 ERP Data warehousing – Example: SAP Business Warehouse architecture Source: SAP AG, 2005

11 A Detailed view of management tools within BW Source: SAP AG 2004

12 Leveraging Standard Content As a guiding principle we map requirements to standard content before we start customizing. However, we may also have external data sources that require custom ODSs and InfoCubes. An example from a large manufacturing company Source: SAP, Sdn BW 2005

13 What We will cover Enterprise Data Warehouse and BI Data Architecture ERP BI Example – SAP Business Warehouse ERP Data Warehouse Content and Methodologies Data Warehouse Presentation tools Post-implementation Reviews

14 Vendor Methodologies Each vendor has their own methodology built on the System Development Life- Cycle (SDLC) approaches

15 Framework for picking your "poison" Source: Bjarne Berg, “How to pick JAD, RAD, XP or a SDLC Methodology for your project“, Nov. 2005,

16 Methodology to build a data warehouse Source: Bjarne Berg, “Managing a BW project “, Project Management Conference ,

17 Standard content The development phase: Model & leverage standard models Storage Requirements Storage Objects Map functional requirements to the standard content before you make enhancements + 1. Create a model based on pre-delivered content 2. Map your data requirements to the delivered content and identify gaps. 3. Identify where the data gaps are going to be sourced from. Source: Bjarne Berg, “25 tips & tricks for managing BW projects“, BW & Portals Conference ,

18 The ETL Processing of a data warehouse SAP NetWeaver has a GUI and formula editors to write complex transfer rules, update rules and data mapping. On a typical DW project, 40-60% of project effort will be spent on data integration, transformation and loads (used to be higher when tools were more immature). Source: Bjarne Berg, “An A to Z Guide in Managing SAP NetWeaver Projects” - SAP NetWeaver and Portals Conference, March. 2006, Las Vegas

19 Performance differences of architectures Source: Alexander Peter, SAP AG, 2006

20 What We will cover Enterprise Data Warehouse and BI Data Architecture ERP BI Example – SAP Business Warehouse ERP Data Warehouse Content and Methodologies Data Warehouse Presentation tools Post-implementation Reviews

21 Deciding Which Front-end To Use For A Global BW System A major decision for the global BW system is the selection of which delivery mechanism to support and who get access to which tool. Most companies start with BW OLAP for web and add other types of interfaces later..

22 The BW Toolset BW is an enterprise- wide comprehensive decision support (DSS) system for analytical needs of an enterprise. BW can also support some degree of operational reporting Therefore it has many presentation tools and options for companies to leverage. This part focuses on the presentation tools

23 BEx what is it? BEx is the 'umbrella name' for many of the presentation tools in BW. Therefore you will notice the tool names are often pre-fixed with names such as BEx Broadcaster, BEx Query Designer and BEx workbooks etc. Source: Bjarne Berg, “Access the easy, quality and performance of your SAP BW based reporting activities” - SAP Professional Journal, Las Vegas, Chicago, Philadelphia, Orlando - Oct/Nov/Dec 2005

24 The BEx Query Designer The BEx query designer is a simple tool that you use to develop queries for users. While the Query Designer defined the query, you have many options on how you deploy the queries You can deploy the queries through web templates, broadcasted s, portals, Excel and more.

25 The BEx Ad-Hoc Query Designer A simplified version of the BEx query designer is the "ad-hoc query designer". This is a web tool that can be used by power users to define their own queries. It can be made available through a BW web template as well as directly into a web portal.

26 Using standard protocols to render in Microsoft tools 1. BEx Workbooks is Excel 'on steroids'. 2. BW queries can be opened in Excel 3. Queries can be refreshed and standard Excel functions can be used This is a great tool for financial analysts and power users. How does it work? Think ODBO……

27 BEx web reports The most common way of deploying BW queries are through a standard, or customized web template with built-in navigational features. Global web templates can be enhanced with logos, colors, fonts and additional features at a low cost of development and ownership.

28 Building iAnalytics using a Web Application Designer (WAD) OLAP Processor BEx Web Application Designer BEx Analyzer/ Query Designer Web Service Charting Engine Web Browser HTLM Templates InfoProvider Source: SAP AG WAD is a tool to build powerful iAnalytics applications and to interact with portals and custom templates. BW queries can be manipulated in WAD for better display

29 Analysis Process Designer Using the BW analytics toolkit, it is possible to build association data sets that can be fed back into the OLTP engine. The way to approach this by using APD (analysis process designer). This is a visual data processing composition engine that enables organization of data flows and updates into the OLTP database (transaction system). Images: Bhanu Gupta, ASUG 2005 Illinois chapter

30 Batch reporting Batch reporting refers to the mass creation of reports such as account statements, that otherwise would be impossible to create on-demand in a data warehouse with billions of records. Crystal is the leader of batch reporting of "pixel controlled' formatted reports.

31 Different Needs and Broadcaster Tools “Casual” users, non-DW users, traveling users, and executives are all prime candidates for broadcasted reports! Flat Reporting Formatted Formatted Print Print Form based Form based Static Static Predictable access Predictable access OLAP Reporting Drill Down Drill Down Slice and Dice Slice and Dice Analyse Analyse Data Mining Data Mining Search and discover Search and discover KPI & Scorecard Formatted Simple Simple Easy to view Easy to view Limited nav Limited nav Aggregates Aggregates Reporting needs Reporting tools

32 Scheduling Broadcasts and exception reports We can schedule the job to run every time the infocube is changed (the data load process chain has run) We can limit the users to when they can schedule jobs to avoid stress on the system. We can provide multiple pre-defined times when they can schedule the reports. We can also allow the users to create their own scheduling times (Only authorized users see this) Source: Bjarne Berg, “SAP BeX Broadcaster - How to make it work in practice” - SAP Professional Journal, Philadelphia, Dec 2005

33 Data Visualization from the Data Warehouse Visual Composer tools are used to visualize high volumes of data. It is a GUI modeling and design tool for rapid application development and prototyping in a code-free development environment. Most Visual Composer tools supports both JSP and HTMLB; and some also support Web Dynpro metadata as well. SAP NetWeaver Visual Composer

34 Example: Data Visualization from the Data Warehouse  Offers the integration with BI InfoProviders (e.g. InfoCubes, BEx Queries and Query Views, BEx Web Applications) via WEB API as well as with heterogeneous data sources (including OLAP and relational data sources) via Java Connectors

Data Mining Technologies Multi-Strategy - IBM Intelligent Miner - SAS - Thinking Machines Darwin Neural Network - HNC Data Mining Workstation - NeuralWare Predict Other Algorithms - DataMind - Information Discovery IDIS Tree-Based -Agnoss Knowledge Seeker -Pilot Discovery Server -SPSS -Silicon Graphics MineSet Visualization -Advanced Visual Systems: AVS Express -NeoVision Hypersystems: Heatmaps -Visible Decisions: Discovery -Visual Neumerics: PV Wave

36 The Data Warehouse and BI Application Testing Since the data is used for decision support and can result in poor decisions being made, testing is a big part for a data warehouse development or BI application project. Typical testing include:  Unit test  Integrated unit test  System test  Integration test  Performance test

37 What We will cover Enterprise Data Warehouse and BI Data Architecture ERP BI Example – SAP Business Warehouse ERP Data Warehouse Content and Methodologies Data Warehouse Presentation tools Post-implementation Reviews

38 After go-live: Track data warehouse load performance During the first 6 weeks after each go-live, organizations should formally track the load performance by process chain to detect any systematic issues This is a real example from a Fortune-50 manufacturing company that had 77 nightly load jobs (process chains).

39 A stabilization period after each go-live is normal until the new process chains has been tuned in the production box. This is a time when active monitoring of process chains should occur After go-live: Track Data Warehouse load performance

Metadata – What to track Organization ­ reporting structure ­ IT initiatives and contacts Technology ­ hardware and software ­ application systems ­ query access tools ­ databases and networks Administrative ­ security profiles ­ version control ­ back-up and recovery ­ quality assurance processes Data ­ data sources ­ data models ­ physical data structures ­ extraction, integration, and transformation rules ­ data currency ­ data definitions ­ data ownership Business Processes ­ functions ­ algorithms and business rules

41 Resources Five Core Metrics: The Intelligence Behind Successful Software Management – by Lawrence H. Putnam & Ware Myers Enterprise Architecture Planning : Developing a Blueprint for Data, Applications, and Technology; by Steven H. Spewak ISBN: Publisher: Wiley Rapid Development by Steve McConnell Paperback: 680 pages ; Publisher: Microsoft Press; ISBN: Start to Finish Guide to IT Project Management by Jeremy Kadlec, Digital: 109 pages. Publisher: NetImpress; ISBN: B0000W86H2

42 More Resources Beck, Kent (1997) “Chrysler Comprehensive Compensation with XP”., published Feb. 2001: The Agile Software Development Alliance Beck, Kent (2000)., Extreme Programming Explained: embrace change., Addison Wesley Longman, Inc., 2000 Berg, Bjarne (1997) “Introduction to Data Warehousing”., module 2., pp New York., NY., Price Waterhouse LLP. October Berg, Bjarne (2004) “Managing BW projects – part-2”., SAP project management conference, Las Vegas, NV, WIS publishing, November. Boehm, Barry (1986) "A Spiral Model of Software Development and Enhancement", ACM SIGSOFT Software Engineering Notes, August. Boehm, Barry (1988) "A Spiral Model of Software Development and Enhancement" IEEE Computer, vol.21, #5, May, pp

43 More Resources Botkin, John (1998)., "Customer Involved Participation as Part of the Application Development Process." University of Maine. Caristi, James (2002) “Extreme Programming: Theory & Practices”., Valparaiso University., SIGSEE 2002 conference tutorial. Damian, Adrian., Hong Danfeng., Li, Holly., Pan, Dong (1999) "Joint Application Development and Participatory Design". University of Calgary, Dept. of Computer Science. Dennis, Alan R., Hayes, Glenda S., Daniels, Robert M. Jr. (1990) "Business process modeling with group support systems". Journal of Management Information Systems Spring. Gilb, Tom (1989)., “Principles of Software Engineering Management”., Addison-Wesley Longman. Jennerich, Bill (1990)., "Joint Application Design -- Business Requirements Analysis for Successful Re- Engineering." UniSphere Ltd., November. Journal of Systems Management (1995)., "JAD basics"., Sept./Oct. ed. Keefer, Gerold (2003)., “Extreme Programming Considered Harmful for Reliable Software Development”., AVOCA GmbH., September ed.

44 More Resources Kettemborough, Clifford (1999)., “The Prototyping Methodology“., Whitehead College, University of Redlands Larman, CraigVictor R. Basili (2003)., Iterative and Incremental Development: A Brief History”., IEEE Computer, pp Martin, James (1990)., “Information Engineering Planning & Analysis, Book II”., Prentice-Hall, Inc. Nobuhiro, Kataoka., Hisao, Koizumi., Kinya, Takasaki., & Norio Shiratori (1998). "Remote Joint Application Design Process Using Package Software". 13th International Conference on Information Networking (ICOIN '98)., January., pp Soltys, Roman., Crawford, Anthony (1998) "JAD for business plans and designs"., The Process Improvement Institute., October Univerity of California (1996)., “Application Development Methodology”., UCD., on-line at

45 More Resources The University of Texas at Austin (2004) "Joint Application Development (JAD) What do you really want?" Accessed on October 24. Yatco, Mei (1999) “Joint Application Design/Development”., School of Business, University of Missouri-St. Louis. Wood, J. and D. Silver (1995),. “ Joint Application Development”., 2nd ed., New York : Wiley. Wetherbe, James C. (1991)., "Executive Information Requirements: Getting It Right", MIS Quarterly, March., p. 51.

46 Your Turn!!! How to contact me: Web page: