1. Real-Time streaming in Power BI
Chris Koester
Lead Data Analytics Consultant / Managing Consultant

2. Agenda Benefits of real-time data Working with real-time data
Real-time streaming in Power BI
Demo

3. Some data Most valuable in real-time
Or value can decline over time

4. Awareness of current conditions
Simulated high CPU usage event

5. How real-time is real-time?

6. Push to streaming & storage simultaneously
Requires code change on data generation side

7. Pull from storage then push1 2
Could add to existing solution without changing application code.
Trigger that handles data on insert
Query at an interval using CDC to get only new rows
There are also 3rd party applications that offer real-time replication such as Attunity.

8. Streaming in big Data Architecture
Advantages of decoupling ingestion from stream processing not limited to big data.
Diagram source -

9. Streaming in Azure Event Hub alternatives -Apache Kafka -AWS Kinesis
Stream processing options
Diagram sources -

10. Event hub Benefit #1 Data retained 1-7 days Stream job can be replayed
Diagram sources -

11. Event hub Benefit #2 Multiple stream jobs on same data
Jobs can perform different calculations (Count, Max, avg, etc.) over different time windows (30 seconds, 1 hour, etc.)
Diagram sources -

12. Multiple outputs from stream analytics
Event Hub to Stream Analytics, then multiple outputs from Stream Analytics

13. Multiple outputs from stream analytics
Make
Time
Honda
T00:00:01
Make
Time
Toyota
T00:00:01

14. real-time datasets in power bi
Streaming
Push
PubNub

15. Streaming datasets Data is not permanently stored
Maximum time window of one hour
Reports not available
Must use custom dashboard tile
Smooth animations
See historic data analysis section -

16. Push datasets Data stored for historical reporting* Reports available
Dashboard visuals update in real-time
*REST API limitations
- 75 max tables
- 200,000 max rows stored per table in FIFO dataset
- 5,000,000 max rows stored per table in ‘none retention policy’ dataset

17. Hybrid options Push dataset and streaming dataset simultaneously
1 - Use pushStreaming mode flag in API
2 – Enable “historic data analysis” in streaming dataset UI
All requests to REST APIs (Such as option 1) are secured using Azure AD Oauth

18. Hybrid option 2 -Enable “historic data analysis”

19. Pubnub
Similar to Streaming Dataset
Uses existing stream from PubNub

20. azure stream Analytics
Streams from Azure Stream Analytics are configured on the Azure Stream Analytics side

21. gotchas UTC for Streaming Datasets
Desired time zone in Power BI Reports