1. Server & Tools Business
7/14/2019
Microsoft Big Data Essentials Module 4 – Developing Big Data Applications with .NET
Saptak Sen, Microsoft
Bill Ramos, Advaiya
Welcome to the Day 2, Module 3 session of Big Data Boot Camp, “Developing Big Data Applications for .NET Developers.”
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Server & Tools Business
7/14/2019
Agenda
LINQ to HIVE
Reactive Extensions (Rx)
Demos
Talk Track: In this session, you will learn about features such as LINQ to Hive and Reactive Extensions (Rx), which help .NET developers use tools they already know to create applications that interact with HDInsight clusters more effectively. You’ll learn how to use the LINQ to Hive extensions that are part of the Microsoft .NET SDK for Hadoop to create and compile LINQ queries to use against Hive data. You’ll also learn how to use Reactive Extensions (Rx) with LINQ to handle streams of data coming into your data environment.
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. LINQ to Hive client libraries
Creates and compiles LINQ queries to use against Hive data
Translates C# or F# LINQ queries into HiveQL queries and executes them on the Hadoop cluster
C#, F#, and .NET
LINQ to Hive client libraries
HiveQL query
Talk Track: Lets first start with LINQ to HIVE. LINQ to Hive lets you author Hive queries using LINQ. The LINQ is compiled to Hive and then executed on your Hadoop cluster. This job is submitted using the LINQ to HIVE client libraries.
In short, the LINQ to Hive client library translates C# or F# LINQ queries into HiveQL queries and executes them on the Hadoop cluster.
Key Points:
LINQ to HIVE
References: content/blob/master/ITPro/Services/hdinsight/using-hdinsight-sdk.md
Hadoop cluster

4. Demo 1: Working with LINQ Queries
Server & Tools Business
7/14/2019
Demo 1: Working with LINQ Queries
Batch Layer
Speed Layer
Serving Layer
HDInsight
Hive
LINQ to Hive
Talk Track: In this demo, I’ll show you how .NET developers can use LINQ to Hive to create applications that interact with the HDInsight cluster—all using tools you already know. You’ll also learn how to use the LINQ to Hive extensions that are part of the Microsoft .NET SDK for Hadoop to create and compile LINQ queries to use against Hive data.
|----Click----|
A Hive table on Microsoft HDInsight cluster.
As soon as a LINQ to HIVE query is fired from the Hadoop command prompt via a C# Visual Studio code, it sends the request to the HIVE table.
As the HIVE table gets a request from the LINQ to HIVE query, it sends the output back to the Hadoop command prompt.
LINQ to HIVE query
LINQ to Hive query
Hive table
Output
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Server & Tools Business
7/14/2019
Introducing Reactive Extensions (Rx)
Let’s now take a look at Reactive Extensions for .NET
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Reactive Extensions (Rx): Pulling vs. Pushing Data
Server & Tools Business
7/14/2019
Reactive Extensions (Rx): Pulling vs. Pushing Data
Application
In interactive programming, pulling data from a sequence that represents the source (IEnumerator)
In reactive programming, subscribing to a data stream (called observable sequence in Rx), with updates handed to it from the source
On Next
Move Next
PUSH
Got next?
Pull
Have next!
Interactive
Reactive
Talk Track: Now that you have got an idea about what is LINQ to HIVE, let’s take a look at interactive programming. Here, the application actively polls a data source for more information by pulling data from a sequence that represents the source. Such behavior is represented by the iterator pattern of IEnumerable<T>/IEnumerator<T>. The IEnumerable<T> interface exposes a single method GetEnumerator(), which returns an IEnumerator<T> to iterate through this collection. The IEnumerator<T> allows you to get the current item by returning the Current property, and to determine whether there are more items to iterate by calling the MoveNext method. This enumeration pattern is synchronous, which means that the application might be blocked while polling the data source. Such a pulling pattern is similar to visiting your library and checking out a book. After you are finished with the book, you make another visit to check out another book.
On the other hand, in reactive programming, the application is offered more information by subscribing to a data stream (called an observable sequence in Rx), and any update is handed to it from the source. The application is passive in the data retrieval process: Apart from subscribing to the observable source, it does not actively poll the source, but merely react to the data being pushed to it. When the stream has no more data to offer or when it errs, the source will send a notice to the subscriber. In this way, the application will not be blocked by waiting for the source to update. This pushing pattern is employed by Reactive Extensions. It is similar to joining a book club, where you register your interest in a particular genre, and books that match are automatically sent to you as they are published. You do not need to stand in line to acquire something that you want. Employing a pushing pattern is helpful in many scenarios, especially in a UI-heavy environment where the UI thread cannot be blocked while the application waits for some events. This is also essential in programming environments such as Microsoft Silverlight, which has its own set of asynchronous requirements.
In short, by using Reactive Extensions, you can make your applications more responsive. The pushing pattern implemented by Rx is represented by the observable pattern of IObservable<T>/IObserver<T>. The IObservable<T> interface is a dual of the familiar IEnumerable<T> interface. It abstracts a sequence of data and keeps a list of IObserver<T> implementations that are interested in the data sequence. The IObservable will notify all observers automatically of any state changes. To register an interest through a subscription, you use the Subscribe method of IObservable, which takes on an IObserver and returns an IDisposable. This gives you the ability to track and dispose of the subscription.
Key Points:
Reactive Extensions (Rx): Pulling vs. Pushing Data
Reference: asynchronous-programming-blues and us/library/hh242985(v=vs.103).aspx
IEnumerable<T>
IEnumerator<T>
IObservable<T>
IObserver<T>
Environment
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Demo 2: Reactive Extensions (Rx)
Server & Tools Business
7/14/2019
Demo 2: Reactive Extensions (Rx)
Batch Layer
Speed Layer
Serving Layer
Windows Azure HDInsight | Blob Storage
Reactive Extensions (Rx)
.NET Application
Store data and raise alerts
with defined Observables
.NET application to interact with
HDInsight cluster
Define Observables and LINQ query
Talk Track: In this last demo, I’ll show you features of Reactive Extensions that allow .NET developers to use tools they already know to create applications that interact with the HDInsight cluster. You’ll also learn how to use Reactive Extensions with LINQ to handle streams of data coming into the data environment.
|----Click----|
Continuously streaming data is available under the speed layer.
Reactive Extensions exist with the LINQ query at the same layer to handle that streaming data. Residing under the serving layer, a .NET application in C# defines the observables, and the LINQ query to interact with the HDInsight cluster.
The .NET code results in storing data to the Windows Azure HDInsight cluster and raising alerts with defined observables.
Windows Azure
HDInsight
Reactive Extensions
with LINQ query to handle the streaming data
Streams
of data loading
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. Server & Tools Business
7/14/2019
Learn more
LINQ to Hive
wikipage?title=LINQ%20to%20Hive&referringTitle=Home
Using the Hadoop .NET SDK with the HDInsight Service
hdinsight/howto-net-libraries/
The Reactive Extensions (Rx) Developer Resources
Reactive Extensions (Rx)
hh242985(v=vs.103).aspx
That’s it for this session. To learn more about what I just showed you, check out these resource links for LINQ to Hive, the Hadoop .NET SDK, and Reactive Extensions.
Thank you!
END OF PRESENTATION
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. Questions?