1. ISQS 6339, Business Intelligence Data Warehousing Zhangxi Lin Texas Tech University 1 1

2. Outlines So far students have learned ◦ Basic concepts of business intelligence ◦ The definition and importance of data warehouse In this lecture, the following topics will be covered ◦ SQL Server 2008 data mart case study  How to access data in a network directory  How to access SQL Server 2008 on the Citrix Server  How to load data from an Excel file to a database ◦ Data warehouse overview ◦ Data warehouse architecture ◦ Data integration ISQS 6347, Data & Text Mining2

3. Data Warehousing Definitions and Concepts Data warehouse ◦ Video – Overview of data warehouse 2’38” Video – Overview of data warehouse A physical repository where relational data are specially organized to provide enterprise-wide, cleansed data in a standardized format Benefits of data warehouse 3’18” Benefits of data warehouse 3

4. Data mart Definition A localized data warehouse that stores only relevant data to a department or even an individual ◦ Dependent data mart A subset that is created directly from a data warehouse ◦ Independent data mart A small data warehouse designed for a strategic business unit or a department 4

5. Data Mart - The IMW Case IMW, standing for Internet Media Works!, is an ASP in real estate information services. It is headquartered in Austin, Texas. CEO is Gary Anderson. Web page: http://www.inetworks.comhttp://www.inetworks.com 5

6. About IMW ISQS 6347, Data & Text Mining 6 Based in Austin, Texas, IMW (Internet Media Works!) is an ASP, specialized mainly in web-based application development, database integration, and web development and hosting for all kinds of businesses.Internet Media Works IMW has been more successful in selling its e-business services for commercial real estate. Its services include lead generation, real estate transaction management, property listing, realtor membership management, real estate indices, real estate auctions, etc., with COMMREX as a complete e- business solution.COMMREX IMW used to have up to 6 full-time employees and a few part-time employees.

7. ISQS 6347, Data & Text Mining 7 Website Hosting Services Core Membership Database Services Core Property Listing Database Services Optional Website Hosting Services Optional Membership Database Services Optional Property Listing Database Services Public User Application Services Networking and System Operation ServicesPublic User Support Internet Service Provider’s Services IMW’s Web-Based Application Services IMW’s Services

8. Why need Data Mart? Data mart complements the centralized data warehousing based on UDM model, for the situations where UDM cannot be used ◦ Legacy databases ◦ Data are from nondatabase sources ◦ No physical connection the centralized data warehouse ◦ Data are not clean 8

9. Data Mart Structures Fact tables ◦ Measures Dimension tables ◦ Dimensions and Hierarchies ◦ Attributes (or columns) Dimensional modeling – Stars and Snowflakes 9

10. Measures A numeric quantity expressing some of the organization’s performance. The information represented by this quantity is used to support or evaluate the decision making and performance of the organization. A measure is also called a fact The table holding measure information is called as a fact table Dimensions vs. Measures 2’38” Dimensions vs. Measures

11. 11 Commrex Real Estate Operational Database Users: property listors, webmaster, marketing manager of IMW Objective: Encourage realtors to use the online ASP services with the best information services to increase IMW’s revenue. Value Chain ◦ Listors create their account ◦ Listors post their real estate properties to the web-based database services and pay listing fees ◦ Property buyers search the website-based database and buy properties from listors. This is the incentive for listors to use the ASP services Business Processes ◦ Listor sign up ◦ Listor account management ◦ Property data posting ◦ Property search ◦ Property database maintenance 11

12. ISQS 6339, Data Mgmt & BI, Zhangxi Lin12 Property ID Listor ID Address Property Type City                     Company ID Chapter Functions Specializations                    Comp Name Address      Telephone # Listor Name UpdateDate Feature Property Type Subtype 1      Type Name      Subtype 2 Subtype n M:1 M:M Primary Key Secondary Key Link to a table Legends Property Listing Database Membership Database IMW’s Database ERD Model Company ID TransactionID PropID UserID M:1

13. Commrex Data Warehousing Users: CEO of IMW, IMW business analyst, IMW marketing manager Analytic themes ◦ Fast retrieval of business key performance indicators (KPIs) ◦ Decision making on business promotions Applications ◦ Geographic distribution of property listings ◦ Scorecard for main performance indicators ◦ Dashboard Questions ◦ How to model data warehouse? ◦ What are required in data transformation and preprocessing? ◦ Any missing dimension for data ware housing? ◦ How to perform routine data warehouse updates – frequency, timing, etc.

14. ISQS 6339, Data Mgmt & BI, Zhangxi Lin14 Property ID Listor ID Address PropType City                     Company ID Chapter Functions Specializations                    Company ID Address      Telephone # Listor Name UpdateDate Features PropType … SubName Primary Key Secondary Key Link to a table Legends Property Listing Fact Membership Dimension IMW’s Data Warehouse Dimensional Model Company Dimension Property Type Dimension Comp Name      Year Month Date Quarter

15. Data Warehouse Overview 15

16. Data Warehousing Characteristics Basic characteristics of data warehousing ◦ Subject oriented ◦ Integrated ◦ Time variant (time series) ◦ Nonvolatile (not allow to change) Others ◦ Web based ◦ Relational/multidimensional ◦ Client/server ◦ Real-time ◦ Include metadata 16

17. Data Warehousing Process Overview Data in DW are constantly accumulated. ◦ Organizations continuously collect data, information, and knowledge at an increasingly accelerated rate and store them in computerized systems The number of users is constantly increasing. ◦ The number of users needing to access the information continues to increase as a result of improved reliability and availability of network access, especially the Internet The organization using data warehouse relied on DW more and more 17

18. Data Warehousing More Concepts Operational data stores (ODS) A type of database often used as an interim area for a data warehouse, especially for customer information files Enterprise data warehouse (EDW) A large-scale data warehouse used across the enterprise for decision support. It integrates different sources of information into a consolidated information system. Metadata (Video 1’41”)Video Data about data. In a data warehouse, metadata describe the contents of a data warehouse and the manner of its use ◦ Syntactic metadata, structural metadata, and semantic metadata 18

19. Data Warehousing Process Overview 19

20. Data Warehousing Process Overview The major components of a data warehousing process ◦ Data sources ◦ Data extraction ◦ Data loading ◦ Comprehensive database ◦ Metadata ◦ Middleware tools 20

21. Data Warehouse Architectures 21

22. Three Parts of Data Warehouse The data warehouse that contains the data and associated software Data acquisition (back-end) software that extracts data from legacy systems and external sources, consolidates and summarizes them, and loads them into the data warehouse Client (front-end) software that allows users to access and analyze data from the warehouse 22

23. Three-Tier Data Warehouse 23

24. Alternative Data Warehouse Architectures (1) 24

25. Alternative Data Warehouse Architectures (2) 25

26. Alternative Data Warehouse Architectures (3) 26

27. Alternative Data Warehouse Architectures (4) 27

28. Alternative Data Warehouse Architectures (5) 28

29. 29 Architectures Comparison

30. Teradata’s EDW 30

31. Hadoop – for BI in the Cloudera Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a distributed computing environment. Hadoop makes it possible to run applications on systems with thousands of nodes involving thousands of terabytes.terabyte Hadoop was inspired by Google's MapReduce, a software framework in which anapplication is broken down into numerous small parts. Doug Cutting, Hadoop's creator, named the framework after his child's stuffed toy elephant.GoogleMapReduceapplication 31

32. Apache Hadoop The Apache Hadoop framework is composed of the following modules : ◦ Hadoop Common - contains libraries and utilities needed by other Hadoop modules ◦ Hadoop Distributed File System (HDFS). ◦ Hadoop YARN - a resource-management platform responsible for managing compute resources in clusters and using them for scheduling of users' applications. ◦ Hadoop MapReduce - a programming model for large scale data processing. ISQS 6339, Data Mgmt & BI32

33. MapReduce 33 MapReduce is a framework for processing parallelizable problems across huge datasets using a large number of computers (nodes), collectively referred to as a cluster or a grid.

34. How Hadoop Operates ISQS 6339, Data Mgmt & BI34

35. Cloudera’s Hadoop System 35

36. Hadoop 2: Big data's big leap forward The new Hadoop is the Apache Foundation's attempt to create a whole new general framework for the way big data can be stored, mined, and processed. The biggest constraint on scale has been Hadoop’s job handling. All jobs in Hadoop are run as batch processes through a single daemon called JobTracker, which creates a scalability and processing-speed bottleneck. Hadoop 2 uses an entirely new job-processing framework built using two daemons: ResourceManager, which governs all jobs in the system, and NodeManager, which runs on each Hadoop node and keeps the ResourceManager informed about what's happening on that node. ISQS 6339, Data Mgmt & BI36

37. MapReduce 2.0 – YARN (Yet Another Resource Negotiator) ISQS 6339, Data Mgmt & BI37

38. Teradata Big Data Platform 2013-12-02 林漳希 @ 清华大学 38

39. Dell representation of the Hadoop ecosystem 39

40. Nokia’s Big Data Architechture 2013-12-02 林漳希 @ 清华大学 40

41. Comparison between big data platform and traditional BI platform 2013-12-02 林漳希 @ 清华大学 41

42. Resolving legacy problem – Dual platform 2013-12-02 林漳希 @ 清华大学 42

43. Ten factors that potentially affect the architecture selection decision 1. Information interdependence between organizational units 2. Upper management’s information needs 3. Urgency of need for a data warehouse 4. Nature of end-user tasks 5. Constraints on resources 6. Strategic view of the data warehouse prior to implementation 7. Compatibility with existing systems 8. Perceived ability of the in-house IT staff 9. Technical issues 10. Social/political factors 43

44. Data Integration 44

45. Data Integration Integration that comprises three major processes: ◦ data access, ◦ data federation, and ◦ change capture. When these three processes are correctly implemented, data can be accessed and made accessible to an array of ETL and analysis tools and data warehousing environments ETL Tools 4’56” ETL Tools 45

46. Data Integration Enterprise application integration (EAI) A technology that provides a vehicle for pushing data from source systems into a data warehouse, including application functionality integration. Recently service- oriented architecture (SOA) is applied Enterprise information integration (EII) An evolving tool space that promises real-time data integration from a variety of sources, such as relational databases, Web services, and multidimensional databases Extraction, transformation, and load (ETL) A data warehousing process that consists of extraction (i.e., reading data from a database), transformation (i.e., converting the extracted data from its previous form into the form in which it needs to be so that it can be placed into a data warehouse or simply another database), and load (i.e., putting the data into the data warehouse) 46

47. Transformation Tools: To purchase or to Build in-House Issues affect whether an organization will purchase data transformation tools or build the transformation process itself ◦ Data transformation tools are expensive ◦ Data transformation tools may have a long learning curve ◦ It is difficult to measure how the IT organization is doing until it has learned to use the data transformation tools Important criteria in selecting an ETL tool ◦ Ability to read from and write to an unlimited number of data source architectures ◦ Automatic capturing and delivery of metadata ◦ A history of conforming to open standards ◦ An easy-to-use interface for the developer and the functional user 47

48. Open Source Software for Big Data Oracle VM VirtualBox Cloudera Hadoop - Get Started With Enterprise Hadoop ‎ Cloudera Hadoop - Get Started With Enterprise Hadoop ‎ Hortonworks Data Platform - Hortonworks.com ‎ Hortonworks Data Platform - Hortonworks.com ‎ Google Hadoop Solutions - google.com ‎ Google Hadoop Solutions - google.com ‎ Hadoop on Google Cloud Platform Hadoop & NoSQL - MarkLogic.com ‎ Hadoop & NoSQL - MarkLogic.com ‎ 48

49. Structure and Components of Business Intelligence 49 SSMS SSIS SSAS SSRS SAS EM SAS EM SAS EG SAS EG MS SQL Server 2008 BIDS

50. Exercise 1 – Walk through data warehousing process Learning Objectives ◦ To gain a general impression how to use SQL Server 2008 to implement a data mart Tasks ◦ Create your database with SSMS, named as ISQS6339_lastname ◦ Import data from Commrex_2011.xls ◦ Use SSMS to create a ERD diagram ◦ Create a SSAS project using BIDS ◦ Define data source, data source view, and cube Deliverable: ◦ One-page printout of the screenshot of the cube diagram 50