9/18/2018 Big Data Analytics with HDInsight Module 6 – Storm Essentials Asad Khan Nishant Thacker Principal PM Manager Technical Product Manager Microsoft Microsoft © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Storm Essentials What is Storm Storm Use Cases Introduction to Storm Storm Topologies Storm on HDInsight Storm Essentials Tools

Storm Essentials What is Storm Storm Use Cases Introduction to Storm Storm Topologies Storm on HDInsight Storm Essentials Tools

What is Apache Storm? Streaming data analysis Easy to program Fast A distributed real time processing platform Fast Clocked at 1M+ messages per second per node Scalable Thousands of workers per cluster Fault Tolerant Failure is expected, and embraced Reliable Guaranteed message delivery Exactly-once semantics

Storm Essentials What is Storm Storm Use Cases Introduction to Storm Storm Topologies Storm on HDInsight Storm Essentials Tools

Store in query able store Storm use case on HDInsight— Connected cars Queuing service Get data Store in blob Get reference data Do machine learning Store in query able store Live dashboard Cloud gateways Queuing Service

Store in query able store Storm use case on HDInsight— Connected cars Azure Blob HBase Azure ML DocumentDB Queuing Service Event Hubs PowerBI Queuing service Get data Store in blob Get reference data Do machine learning Store in query able store Live dashboard Cloud gateways Event Hubs

Storm customer use cases 9/18/2018 Storm customer use cases Input Operators (Examples) Side Lookup Output Programming Language Connected Cars Event hubs Window based aggregation, Join stream/split stream HBase, ML DocumentDB C# hybrid, Java ETL Event Hubs Partitioning/ organize N/A WASB Java IoT Window based aggregation Hbase, ML DocumentDB, HBase Fraud detection ServiceBus Queue Filter ML Hbase C# hybrid Social analytics Twitter Groupby/trending topics Realtime dashboard (BI) Trident Network monitoring Kafka Split (on success/ failure) SQL Log Search Storage Queue/ Event Hub Parsing & index Elastic Search Mobile engagement Eventhub Count HBase SignalR © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Apache Storm and Event Hubs Tech Ready 15 9/18/2018 Apache Storm and Event Hubs Event producers Collection Event Queuing System Transformation Long-term storage Presentation and action Azure ML Storage adapters Stream processing Applications Web and social Devices Sensors Live Dashboards Search and query Data analytics (Excel) Web/thick client dashboards Devices to take action Event hub Apache Storm on HDInsight Apache HBase on HDInsight DocumentDB Solr Azure Search MongoDB SQL Cloud gateways (web APIs) Field gateways Kafka/RabbitMQ/ ActiveMQ Event hubs © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Storm Essentials What is Storm Storm Use Cases Introduction to Storm Storm Topologies Storm on HDInsight Storm Essentials Tools

Storm introduction {…} Tuple Stream Spout Core unit of data Immutable set of key/value pair Unbounded sequence of Tuples Source of streams Wraps a streaming data source and emits Tuples

Spout API Lifecycle API public interface Ispout extends Serializable { void open(Map conf, TopologyContext context, SpoutOutputCollector collector); void close(); void activate(); void deactivate(); void nextTuple(); void ack(Object id); void fail(Object id); } Lifecycle API

Spout API Core API public interface Ispout extends Serializable { void open(Map conf, TopologyContext context, SpoutOutputCollector collector); void close(); void activate(); void deactivate(); void nextTuple(); void ack(Object id); void fail(Object id); } Core API

Spout API Reliability API public interface Ispout extends Serializable { void open(Map conf, TopologyContext context, SpoutOutputCollector collector); void close(); void activate(); void deactivate(); void nextTuple(); void ack(Object id); void fail(Object id); } Reliability API

Using Bolts Write to a data store Read from a data store Perform arbitrary computation (Optionally) Emit additional streams {…} Compute Core functions of a streaming computation | Receive tuples and do stuff

Bolt API Lifecycle API public interface IBolt extends Serializable { void prepare(Map sormConf, TopologyContext context, OutputCollector collector); void cleanup(); void execute(Tuple input); } Lifecycle API

Bolt API Core API public interface IBolt extends Serializable { void prepare(Map sormConf, TopologyContext context, OutputCollector collector); void cleanup(); void execute(Tuple input); } Core API

Storm Essentials What is Storm Storm Use Cases Introduction to Storm Storm Topologies Storm on HDInsight Storm Essentials Tools

Topologies

Stream groupings {…} Stream Grouping Description Shuffle 9/18/2018 Stream groupings Stream Grouping Description Shuffle Sends tuples to bolts in random, round robin sequence Fields Sends tuples to a bolt based on one or more fields in the tuple All Sends a single copy of each tuple to all instances of a receiving bolt Custom Customized processing sequence. Use to get maximum flexibility of topology processing based on factors such as data types, load, and seasonality. Global Sends tuples generated by all instances of a source to a single target instance Stream groupings determine how Storm routes Tuples between tasks in a topology ? {…} © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Topology deployment Supervisor Zookeeper

What is Trident? Fluent Merges and joins High level abstraction Stream-oriented API Merges and joins High level abstraction Built on Storm’s core primitives Built for Aggregation, groupings, functions, & filters

Trident API—fluent and stream oriented TridentTopology topology = new TridentTopology(); FixedBatchSpout spout = new FixedBatchSpout(…); Stream stream = topology.newStream(“words”, spout); stream.each(…, new Myfunction()) .groupBy() .each(…, new MyFilter()) .persistentAggregate(…);

Storm Essentials What is Storm Storm Use Cases Introduction to Storm Storm Topologies Storm on HDInsight Storm Essentials Tools

Apache Storm on HDInsight 9/18/2018 Apache Storm on HDInsight Managed services Event Hub Open source platform Visual Studio Analyse millions of messages per second Azure Scale-up and scale-down HBase, SQL Database, DocumentDB Speed © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Topology 2008-10-21 22:56:59 Here's to the crazy ones, the misfits, the rebels, the troublemakers, the round pegs in the square holes... @stevejobs [Lang] en… 2008-10-21 22:56:59 Here's to the crazy ones, the misfits, the rebels, the troublemakers, the round pegs in the square holes... es mobile for ja iphone es mobile ja iphone mobile 4 1 iphone 5 @stevejobs [Lang] en… Twitter tweets Ignore word Bolt Word count Bolt Splitter Bolt

Get started today! For more information visit: http://azure.microsoft.com/en-us/services/hdinsight/

© 2014 Microsoft Corporation. All rights reserved © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.