1. Introduction to Big Data James Miller.

2. Some material from Big Data for Dummies and

3. . .

4. What is Big Data? Large Volumes High Velocity Wide Variety
· 8 Bits = 1 Byte
· 1000 Bytes = 1 Kilobyte
· 1000 Kilobytes = 1 Megabyte
· 1000 Megabytes = 1 Gigabyte
· 1000 Gigabytes = 1 Terabyte
· 1000 Terabytes = 1 Petabyte
· 1000 Petabytes = 1 Exabyte
· 1000 Exabytes = 1 Zettabyte
· 1000 Zettabytes = 1 Yottabyte
· 1000 Yottabytes = 1 Brontobyte
· 1000 Brontobytes = 1 Geopbyte

7. Waves of Data Management
Relational Databases and Data Warehouses
Web and Content Management - unstructured including audio and video
Big Data

8. Where is data available?

10. Other major data sources
Social Networks, Twitter
Internet of Things (airplanes, cars, electric meters, machine logs)
and other text
Satellite Images

11. Why study Big Data / Data Analytics?

12. Why Big Data Now?
In the late 1990s, search engine and Internet companies like Google, Yahoo!, and Amazon.com were able to expand their business models, leveraging inexpensive hardware for computing and storage.
Next, these companies needed a new generation of software technologies that would allow them to monetize the huge amounts of data they were capturing from customers. These companies could not wait for results of analytic processing. They needed the capability to process and analyze this data in near real time.

13. Distributed Computing on commodity hardware

14. Why Virtual Machines?

15. Have someone else build and run your Big Data operation
Amazon Elastic Public Compute Cloud
Google Big Data Services
Microsoft Azure

16. What do you find in Big Data Centers
A room full of racks
Racks full of cases of standard sizes
Cases full of blades (a blade is an individual server on a circuit card that fits in a slot)
Thousands of servers
Organized wiring
Specialized data management software
Virtualized resources (machines, storage, networks)
See optional links to Data Center Videos in Canvas

17. How can we use all this hardware?
How will all the software fit together?
What do we need besides relational databases?
How can we put many servers to work on our big data job(s)?

18. One view of Big Data Architecture

19. Big Data Management

20. This looks like trouble

21. Hadoop – What is it and why is it important?

22. Map Reduce

23. Hadoop and Map Reduce

24. Hadoop Ecosystem
HBASE - billions of rows, uses Hadoop Distributed File System
Hive – batch oriented data warehousing layer
Pig and Pig Latin –language for loading and retrieving data stored in HDFS]
Sqoop – bulk import from relational, bulk export, generates map and reduce jobs.
Zookeeper – manages parallel processing and big jobs across racks

25. Analytics and Big Data

26. Analytics Examples See link to optional reading in Canvas
Fraud Detection – insurance claims, credit card transactions
Monetize your data
Google ads
License your data to others
Predict which customers are good targets of upselling when the call in to the call center

27. Data Mining
Analyze large amounts of data to find patterns in that data
Techniques
Classification Trees
Logistic Regression
Neural Networks
Clustering Techniques

28. Here’s a classification tree example
Here’s a classification tree example. Consider the situation where a telephone company wants to determine which residential customers are likely to disconnect their service. The telephone company has information consisting of the following attributes: how long the person has had the service, how much he spends on the service, whether he has had problems with the service, whether he has the best calling plan for his needs, where he lives, how old he is, whether he has other services bundled together with his calling plan, competitive information concerning other carriers plans, and whether he still has the service or has disconnected the service. Of course, you can find many more attributes than this. The last attribute is the outcome variable; this is what the software will use to classify the customers into one of the two groups — perhaps called stayers and flight risks.

29. The data set is broken into training data and a test data set
The data set is broken into training data and a test data set. The training data consists of observations (called attributes) and an outcome variable (binary in the case of a classification model) — in this case, the stayers or the flight risks. The algorithm is run over the training data and comes up with a tree that can be read like a series of rules. For example, if the customers have been with the company for more than ten years and they are over 55 years old, they are likely to remain as loyal customers.
These rules are then run over the test data set to determine how good this model is on “new data.” Accuracy measures are provided for the model. For example, a popular technique is the confusion matrix. This matrix is a table that provides information about how many cases were correctly versus incorrectly classified. If the model looks good, it can be deployed on other data, as it is available (that is, using it to predict new cases of flight risk). Based on the model, the company might decide, for example, to send out special offers to those customers whom it thinks are flight risks.

30. Text Analytics
Tweets, Logs, s

31. Text Analytics

32. Customized Approaches

33. Customized Approaches
R Environment
Effective data-handling and manipulation components.
Operators for calculations on arrays and other types of ordered data.
Tools specific to a wide variety of data analyses.
Advanced visualization capabilities.
S programming language designed by programmers, for programmers
with familiar concepts such as loops, conditionals, user defined recursive functions
and many options for input/output. .

34. Customized Approaches
Avoid 100% custom coding
TA-Lib for stock market and individual stock analysis
GeoTools - An open source geospatial toolkit for manipulating GIS data in many forms

35. Big Data Analysis Framework
Support for multiple data types including unstructured
Handle batch processing and/or real time data streams
Use what already exists in your environment
Support NoSQL and other newer forms of accessing data
Overcome latency
Provide cheap storage
Integrate with cloud deployments

36. What data do you need?
Exploratory Stage – look for patterns in large amounts of data.
Hadoop and MapReduce – experiments
FlumeNG – stream data directly into Hadoop
May eliminate “uninteresting data” where no patterns are found
Codify (attempt to implement repeatable process)
Example – retailer discovers chatter upcoming college football event in a particular geographic area near one of its stores
Integration (look for specific customers that should receive offers, increase inventory ….)
Customer data, inventory data.

37. Data Transformation Big Data may be “dirty” Extract Transform Load
Look for patterns first without cleaning data
When patterns useful to the business are found, then apply quality standards
Extract Transform Load
May be better to ELT (Extract Load Transform)
Can process faster and in parallel in Hadoop
Can use Hadoop environment tools – HiveQL, Pig Latin

38. Streaming and Complex Event Processing
Recognition of a pattern in the stream of data triggers CEP
Example: System detects that a customer has used the loyalty program at checkout and triggers and event to determine if this is an important customer and if so, provides a special offer or a gift at checkout.

39. Getting started with Big Data

40. Do and Don’t Do involve all business units in your big data strategy
Do evaluate all delivery models for big data (cloud)
Do think about your traditional data sources as part of your strategy
Do plan for consistent metadata
Do distribute your data
Don’t rely on a single approach for big data analytics.
Don’t go big before you are ready
Don’t overlook the need to integrate data
Don’t forget to manage data securely
Don’t overlook the need to manage the performance of your data