1. Zhangxi Lin Texas Tech University
ISQS 6339, Business Intelligence Supplemental Notes on the Term Project (Spring 2017)
Zhangxi Lin
Texas Tech University

2. Projects Two data warehousing projects (70%)
SQL Server based
Hadoop based
Big data collaborative studies (30%).
One presentation – minutes, and another 10 minutes for discussion
Report & references
Videos and demonstrations
Total 60 points

3. Term project
3-5 students form a team to fulfill a data mart development project.
Stage 1 (10%): One-page project proposal. April 11
Stage 2 (20%): Data mart Implementation. April 20
Stage 3 (30%): Collaborative study. Due April 27
Stage 4 (20%): Hadoop Project completed. Due May 4
Stage 5 (20%): Final report. Due May 12
Detailed instructions:

4. Merits of data warehousing projects
Carefully developed project proposal demonstrating the understanding of the business requirements, attractive analytics themes, and clearly defined project goal and objectives
Comprehensive data mart design, such as multiple fact tables, with supporting analytic themes
Applications of advanced ETL model or techniques, such as slowly changing dimensions, the use of containers, etc.
Advanced OLAP cube design, and/or optional MDX scripting by self-taught
Rich data analysis outcomes
Well-presented final report
Demonstrating the creative ideas and skillful data warehousing ability

5. Hadoop projects

6. Components Load Balancer Oozie Solr, SolrCloud, SolrJ, HA NewSQL
Kafka, Storm, Impala
REST ZK
MySQL
Nginx/HA-Proxy
Flume
Sqoop
Ganglia
Technology stack
Tomcat, Jetty 
Avro

7. Big Data Presentation Topics
No:
Topic
Components
Team#
Presentation 1
Data warehousing
Focus: Hadoop Data warehouse design
HDFS, HBase, HIVE, NoSQL/NewSQL, Solr
DW1 2
Publicly available big data services
Focus: tools and free resources
Hortonworks, CloudEra, HaaS, EC2
DW2 3
MapReduce & Data mining
Focus: Efficiency of distributed data/text mining
Mahout, H2O, R, Python
DW3 4
Big data ETL
Focus: Heterogeneous data processing across platforms
Kettle, Flume, Sqoop, Impala
DW4 5
System management:
Focus: Load balancing and system efficiency
Oozie, ZooKeeper, Ambari, Loom, Ganglia
DW5 6
Application development platform
Focus: Algorithms and innovative development environments
Tomcat, Neo4J, Pig, Hue
DW6 7
Tools & Visualizations
Focus: Features for big data visualization and data utilization.
Pentaho, Tableau
Saiku, Mondrian, Gephi,
DW7 8
Streaming data processing
Focus: Efficiency and effectiveness of real-time data processing
Spark, Storm, Kafka, Avro

8. Data Warehousing Methodology
- Implementing data warehouse systematically 8

9. Dimensional Modeling Process
Preparation
Identify roles and participants
Understanding the data architecture strategy
Setting up the modeling environment
Establishing naming conventions
Data profiling and research
Data profiling and source system exploration
Interacting with source system experts
Identifying core business users
Studying existing reporting systems
Building Dimensional models
High-level dimensional model design
Identifying dimension and fact attributes
Developing the detailed dimensional model
Testing the model
Reviewing and validating the model

10. Business Dimensional Lifecycle
Req’ts
definition
Technical
Arch.
Design
Product
Selection &
Installation
Growth
Dimensional
Modeling
Physical
Design
ETL design
&
Development
Deployment
Project
Planning
BI Appl.
Specification BI
Application
Development
Maintenance
Project Management 10

11. Data Profiling
Data profiling is a methodology for learning about he characteristics of the data
It is a hierarchical process that attempt to build an assessment of the metadata associated with a collection of data sets.
Three levels
Bottom – characterizing the values associated with individual attributes
Middle – the assessment looking at relationships between multiple columns within a single table.
Highest level – the profile describing relationships that exist between data attributes across different tables.
Can run a program against the sandbox source system to obtain the needed information. 11

12. ETL Methodology Develop a high-level map
Build a sandbox source system (optional)
Detailed data profiling
Make decisions
The source-to-target mapping
How often loading tables
The strategy for partitioning the relational and Analysis Services fact table
The strategy for extracting data from each source system
De-duplicate key data from each source system (optional)
Develop a strategy for distributing dimension tables across multiple database servers (optional) 12

13. Sandbox Source System
Sandbox
A protected, limited environment where applications are allowed to "play" without risking damage to the rest of the system.
A term for the R&D department at many software and computer companies. The term is half-derisive, but reflects the truth that research is a form of creative play.
In the DW/BI context, sandbox source system is a subset of source database for analytic exploration tasks
How to create
Set up a static snapshot of the database
By sampling 13

14. Decision Issues in ETL System Design
Source-to-target mapping
Load frequency
How much history is needed 14

15. Strategies for Extracting Data
Extracting data from packaged source systems –self-contained data sources
May not be good to use their APIs
May not be good to use their add-on analytic system
Extracting directly from the source databases
Strategies vary depending on the nature of the source database
Extracting data from incremental loads
How the source database records the changes of the rows
Extracting historical data 15