Presentation is loading. Please wait.

Presentation is loading. Please wait.

Janet works on the Azure Stream Analytics team, focusing on the service and portal UX. She has been in the data space at Microsoft for 5 years, previously.

Similar presentations


Presentation on theme: "Janet works on the Azure Stream Analytics team, focusing on the service and portal UX. She has been in the data space at Microsoft for 5 years, previously."— Presentation transcript:

1

2

3 Janet works on the Azure Stream Analytics team, focusing on the service and portal UX. She has been in the data space at Microsoft for 5 years, previously working on Azure SQL Database and SQL Server Data Tools. Janet has a computer science degree from the University of Washington. Konstantin Zoryn Senior Software Engineer Janet Yeilding Program Manager II Konstantin is a developer on the Azure Stream Analytics team working on Query Language and Query Processing. In the past he worked in the Data Platform Group on structured data search and data classification. These technologies shipped in several products including PowerPivot and PowerBI.

4

5

6

7 Analyze large Volumes of high Velocity streaming data coming from a Variety of sources Alerts Dashboards Command & Control Action Downstream Workflow & Apps Store in Hadoop, SQL, Blob, etc. for later historical analysis Machine Learning Fully managed, highly available and resilient Rapid development with SQL-like language. Low cost to develop and manage

8 TimeHmdtTempPrssLghtSensorId 2014-12-11T02:24:56.8850110Z42.772.698187.75.038R-PI Olivier's Office 2014-12-11T02:24:51.2785190Z45.274.498103.250.05R-Pi Elisa

9

10 Access event timestamps in your query with System.Timestamp SELECT System.Timestamp AS Time FROM SensorReadings The system assigns timestamps automatically based on arrival time to input source SELECT * FROM SensorReadings Or you can assign it from the event payload SELECT * FROM SensorReadings TIMESTAMP BY time Every event that flows through the system has a timestamp

11 Windows can be – Tumbling, Hopping, Sliding Windows are fixed length An output event’s timestamp corresponds to the end of the window Must be used in a GROUP BY clause 154286 4 0520251015 Time Window 1Window 2 Window 3 1214 Output Events (SUM aggregate)

12 SELECT sensorId, COUNT(*) AS Count FROM SensorReadings TIMESTAMP BY time GROUP BY sensorId, TumblingWindow(second, 10) Tell me the count of sensor readings per device every 10 seconds 1542686 5 05201015 Time (secs) 15426 86 25 A 10-second Tumbling Window 30 361 5361 5 2 4

13 SELECT sensorId, COUNT(*) AS Count, AVG(temp) FROM SensorReadings TIMESTAMP BY time GROUP BY sensorId, HoppingWindow(second, 10, 5) Every 5 seconds give me the count of sensor readings and the average temperature over the last 10 seconds 1542687 0520101525 A 10-second Hopping Window with a 5-second “Hop” 30 426 87 5361 15 426 8753 61 53 5 3 2 4 4

14 SELECT sensorId, MIN(temp) as temp FROM SensorReadings TIMESTAMP BY time GROUP BY sensorId, SlidingWindow(second, 5) HAVING MIN(temp) > 75 Tell me if the temperature is above a threshold of 75 for a total of 5 seconds 1 5 05201015 Time (secs) A 10-second Sliding Window 51 5 1 8 5 8

15 {device1, 46.45} {device2, 45.22} {device2, 45.27} {device1, 46.55} Event Stream: Find cases where the humidity reading changed by at least.1 within 5 seconds {device1, 46.45}{device2, 45.22} {device1, 46.55} {device2, 45.27} {device1, 46.48}

16 Reference Data Correlating event streams with reference data Static or slowly-changing data Same programming experience as standard ASA JOIN but without temporal bound SELECT SensorReadings.sensorID, SensorReadings.temp FROM SensorReadings JOIN thresholdRefData ON SensorReadings.sensorID = thresholdRefData.sensorID WHERE SensorReadings.temp > thresholdRefData.value

17 Multiple Outputs Explicitly specify an output stream using INTO

18 WITH Define a temporary named result set to develop complex queries more elegantly WITH step1 AS ( SELECT * FROM input ), step2 AS ( SELECT * FROM step1 ) SELECT * FROM step2

19 Machine Learning Azure ML and Stream Analytics are now integrated in private preview Azure ML can publish web endpoints for operationalized models Azure Stream Analytics can bind custom function names to such web endpoints Example: apply anomaly detection function event-by-event SELECT temp, myfunction(temp) FROM SensorReadings Coming Soon!

20

21

22

23 DML SELECT FROM WHERE GROUP BY HAVING CASE WHEN THEN ELSE INNER/LEFT OUTER JOIN UNION CROSS/OUTER APPLY CAST INTO ORDER BY ASC, DSC Appendix: Stream Analytics Query Language Scaling Extensions WITH PARTITION BY OVER Date and Time Functions DateName DatePart Day Month Year DateTimeFromParts DateDiff DateAdd Windowing Extensions TumblingWindow HoppingWindow SlidingWindow Aggregate Functions Sum Count Avg Min Max StDev StDevP Var VarP String Functions Len Concat CharIndex Substring PatIndex Temporal Functions Lag, IsFirst CollectTop

24


Download ppt "Janet works on the Azure Stream Analytics team, focusing on the service and portal UX. She has been in the data space at Microsoft for 5 years, previously."

Similar presentations


Ads by Google