BI 202 Data in the Cloud Creating SharePoint 2013 BI Solutions using Azure 6/20/2014 SharePoint Fest NYC

About Me Coskun Cavusoglu Enterprise Architect PwC, Tax Technology COSKUN pronounced

Agenda  Different Flavors of Azure and Windows Azure SQL Services  Creating Reports Using Data in the Cloud  Leveraging Big Data in your BI Solutions  Using Windows Azure Data Market Feed in your Reports

Windows Azure Overview of different flavors of Azure and Windows Azure SQL Services

What is Windows Azure? Windows Azure is an open and flexible cloud platform that enables you to quickly build, deploy and manage applications across a global network of Microsoft-managed datacenters

Different flavors of Windows Azure

Windows Azure Architecture

Windows Azure Data Management Overview of different flavors of Azure and Windows Azure SQL Services Volume,velocity,variosity

Windows Azure Data Management Options  SQL Database, a managed service for relational data (PaaS)  SQL Server in a VM, with SQL Server running in a Windows Azure Virtual Machines (IaaS)  Blob Storage, which stores collections of unstructured bytes (PaaS)  Table Storage, providing a NoSQL key/value store (PaaS)

Windows Azure SQL Database Microsoft Windows Azure SQL Database is a cloud-based relational database service that is built on SQL Server technologies and runs in Microsoft data centers on hardware that is owned, hosted, and maintained by Microsoft.  Business-class relational database management engine for transactional integrity  Built-in datacenter replicas, 1 primary, 2 replicas  Support for dynamic scale out of thousands of distributed databases

Windows Azure Table Storage Windows Azure Table Storage is a NoSQL approach. Despite its name, Table Storage doesn’t support standard relational tables. Instead, it provides what’s known as a key/value store, associating a set of data with a particular key, then letting an application access that data by providing the key.

Different Windows Azure data management options

Windows Azure Data Services

Windows Azure SQL Data Sync While SQL Database does maintain three copies of each database within a single Windows Azure datacenter, it doesn’t automatically replicate data between Windows Azure datacenters. Instead, it provides SQL Data Sync Figure from -

Windows Azure SQL Federations Windows Azure SQL Federations enable the database tier to provide built- in support for horizontal partitioning or ‘sharding’ of data. Download the SQL Federation specification -

DEMO Volume,velocity,variosity Windows Azure Data Management

DEMO Creating Reports Using Data in the Cloud

Big Data An overview of Big Data concepts and how to use big data without crashing your budget and servers. The Solution to the Three V’s problem: Variety, Volume, Velocity

What is Big Data? Big Data is about much more than data. It represents a new way of doing business – one that is driven by data- based decision-making. A new way to think about data – and a new way of doing business…

What can I do with Big Data? Leveraging Big Data capabilities, large volumes of varied sources of data – both internal and from third-parties – can deliver “intelligence at the moment” - insight and intelligence derived from fast moving data sets can  Help inform split second strategy decisions  Spur innovation,  Inspire new products,  Enhance customer relationships,  Uncover fraud,  Bolster operations  Build competitive advantage.

When do I need a Big Data Solution?  Variety  85 % of data does not match existing data schemas  All sorts of data semi-structured and unstructured data  Volume:  Databases a growing faster than ever – 10 x every 5 years  Velocity  Growing # of applications, devices and users generating and requesting data

Executing a query using a relational database system Figure from Developing Big Data Solutions on Windows Azure Big Data Solutions on Windows Azure

Executing a query using a Big Data Solution Figure from Developing Big Data Solutions on Windows Azure Big Data Solutions on Windows Azure

Major Differences between a Big Data solution and existing relational database systems

Big Data platforms  Although there are different ways you can implement a big data solution the industry has been mostly using a technology called Hadoop.  Cloudera's Impala, the Apache Drill effort led by MapR, IBM BigSQL, Hortonworks' Stinger project, and EMC's Pivotal Distribution are all high-profile SQL-on-Hadoop options

So, what is Hadoop? The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing. The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.

Hadoop Modules  Hadoop Common: The common utilities that support the other Hadoop modules.  Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.  Hadoop YARN: A framework for job scheduling and cluster resource management.  Hadoop MapReduce: A YARN-based system for parallel processing of large data sets.

Other Hadoop related projects  Ambari™: A web-based tool for provisioning, managing, and monitoring Apache Hadoop clusters which includes support for Hadoop HDFS, Hadoop MapReduce, Hive, HCatalog, HBase, ZooKeeper, Oozie, Pig and Sqoop. Ambari also provides a dashboard for viewing cluster health such as heatmaps and ability to view MapReduce, Pig and Hive applications visually alongwith features to diagnose their performance characteristics in a user-friendly manner. Ambari™  Avro™: A data serialization system. Avro™  Cassandra™: A scalable multi-master database with no single points of failure. Cassandra™  Chukwa™: A data collection system for managing large distributed systems. Chukwa™  HBase™: A scalable, distributed database that supports structured data storage for large tables. HBase™  Hive™: A data warehouse infrastructure that provides data summarization and ad hoc querying. Hive™  Mahout™: A Scalable machine learning and data mining library. Mahout™  Pig™: A high-level data-flow language and execution framework for parallel computation. Pig™  ZooKeeper™: A high-performance coordination service for distributed applications. ZooKeeper™

Using Windows Azure for Big Data Solutions Windows Azure HDInsight gives you the ability to gain the full value of Big Data with a modern, cloud-based data platform that manages data of any type, whether structured or unstructured, and of any size. Windows Azure HDInsight is a Big Data solution powered by Apache Hadoop HDInsight

Where does HDInsight fall in your Data Platform?

DEMO Windows Azure Marketplace