CSCI5570 Large Scale Data Processing Systems A General Introduction to Big Data Applications and Infrastructures James Cheng CSE, CUHK

Big Data Applications Big data applications in science Big data applications in industry Big data applications for social good

Big Data Applications Big data applications in science – Genomic studies (Big Data Institute at CUHK-SZ, working with hospitals and companies) – Astronomical data analysis – Complex physics simulations – Biology and environmental research – …

Big Data Applications Big data applications in industry – Sales conversion optimization – Consumer behavior analysis – Customer segmentation – Security threat prediction – Predictive support – Market basket analysis – Pricing optimization – Other industry-specific applications Source: Big data use cases by Dell net/Dell/big-data-use- cases

Sales Conversion Optimization Collect data from the process how consumers go through Internet advertising or search, to conversion into sales Analyze data for the entire sales conversion process: from a click on an ad to the final transaction Uncover insights on how the conversion process can be improved

Sales Conversion Optimization Example 1: – Industry: communication – Companies: T-Mobile, Celcom Axiata Berhad – Usage: customer retention, product promotion, market share acquiring … Example 2: – Industry: finance – Companies: Credem – Usage: financial product/service prediction, consumer targeting …

Consumer Behavioral Analytics Analyze customers’ purchasing habits, as well as data about customers from multiple sources Understand why customers like certain products, and craft personalized marketing campaigns to boost profits

Consumer Behavior Analysis Example 1: – Industry: food & beverage – Companies: Starbucks, McDonald’s, Nestle – Usage: customer loyalty building, customer experience enhancing, customer sentiment monitoring, brand image maintaining, crisis control … Example 2: – Industry: finance – Companies: Mastercard – Usage: customers’ spending patterns, shoppers’ interests, consumer benchmarking …

Customer Segmentation Analyze data about consumers from multiple sources, e.g., social media data and transaction history Classify customers into different groups, and target each group with personalized offers

Customer Segmentation Example 1: – Industry: retail – Companies: Walmart, Nordstrom – Usage: customized marketing, customized shopping experience, targeted product promotion … Example 2: – Industry: hotel/travel – Companies: IHG (International Hotel Group) – Usage: personalized web experience, marketing mix adjusting, sales boosting …

Security Threat Prediction Track trends in IT security breaches Analyze anomalies that indicate a potential security breach Proactively go after threats before they strike

Security Threat Prediction Example 1: – Industry: banking – Companies: Rabobank, Zion’s Bank – Usage: fraud detection, financial criminal activity prediction … Example 2: – Industry: e-commerce – Companies: Amazon – Usage: warehouse security, …

Predictive Support Analyze sensor data and other machine- generated data Predict potential equipment malfunctions Reduce lost profits due to downtime Improve safety for employees and customers Examples: – Travel industry: Southwest Airlines – Transportation industry: Union Pacific Railroad – Cloud storage industry: Engine Yard

Market Basket Analysis and Pricing Optimization Analyze market basket data, pricing data, and data from multiple sources Optimize product selection and pricing, and decide where to target ads. Examples: – Household retail industry: P&G – Beverage industry: Coca-Cola – Travel industry: Etihad – Car manufacturing industry: Ford

Industry-Specific Applications Insurance (e.g., Discovery Health): identify fraudulent claims Healthcare (e.g., Aurora Health Care): control healthcare quality, find trends in diseases Travel (e.g., Kayak): flight price prediction HR/Recruiting (e.g., Catalyst IT): screen job candidates Farming (e.g., John Deere): plan farming, boost efficiency and yields

Summary of Industrial Applications 6 big categories of big data use cases (plus 5 more industry-specific use cases) A wide range of industries: communication, finance, food & beverage, retail, hotel, travel, banking, e- commerce, transportation, Cloud storage, car manufacturing, insurance, healthcare, HR/recruiting, farming Proven working in many companies, with thousands to 2 million employees If the nature of a company falls into any of the industries, or is similar to the business of any of the companies just presented, then the large scale data processing systems taught in this course will be very helpful!

Big Data Applications Big data applications for social good – Physical education in junior schools (a project involving hundreds of schools in Hong Kong, each school with students) – Healthy ageing – Health care monitoring – Air pollution control – …

Big Data Solutions Deep learning – The universal big data solution? – The best big data solution? Source: 马小平 THU

Big Data Solutions A big data application often requires a combination of multiple types of systems to develop a good solution What types of systems are generally available today for big data solutions?

Systems for Big Data Solutions General-purpose big data platforms: Hadoop, Spark, Flink, Dato, Naiad, Husky … NoSQL: MongoDB, Cassandra, CouchDB … Key-value stores: Redis, Memcached … Search engines: ElasticSearch, Solr … Machine learning systems: Petuum, GraphLab, TensorFlow, mxnet, Angel, DMTK … Graph computing systems: Pregel, Giraph, GraphLab, …

Systems for Big Data Solutions General- purpose platforms NoSQL Key-value stores Search engines Graph systems Machine learning systems Great! So many big data tools available!

A Typical Big Data Solution Graph Analytics Machine Learning Map Reduce Stream Processing Stream Processing SQL OLAP SQL OLAP Powerful Computing Engine (e.g., Husky) APIs Search Engine Messaging System Key-value Stores NoSQL Hadoop Ecosystem Data Storage Data Collection Data Processing Graph Analytics Machine Learning Map Reduce Stream Processing Stream Processing SQL OLAP SQL OLAP User-Friendly Application Interface With such a platform (e.g., Husky), you can easily build high- performance end-to-end big data business solutions! Smart city Finance Marketing Scientific research Anything about big data