1. 9/9/2015 13:17 1 Can Real-Time Systems be built with Off the Shelf Components? Krithi Ramamritham Real-Time Systems Laboratory University of Massachusetts, Amherst & Indian Institute of Technology, Bombay

2. 9/9/2015 13:17 2 Talk Outline  Using off-the-shelf components for Real-Time applications?  Future application characteristics  Problems and challenges  Characteristics and limitations of OTS OSs  OTS component based solutions for distributed Real-Time applications  User-level scheduling of communicating RT tasks  Conclusions, recommendations, future

3. 9/9/2015 13:17 3 EWS Network Field Devices Field Network PLC Field Devices Field Network PLC Operator Commands Video/audio Device monitoring Background Server M.Services LynxOS M. Services VxWorks M. Services Operator Stations M. Services NT Industrial Control Environment

4. 9/9/2015 13:17 4 Future Application Characteristics  Interactions and communication  Human/device to human/device  Co-existence of multiple types of media  Small control and signal data  Periodic updates  Bursty file/page/image access and transfer  Continuous media  Computer and communication technology advances

5. 9/9/2015 13:17 5 What Are the New Challenges?  Communication centered  Operating System is no-longer standalone  Support for real-time and non-real-time co-existence  Integrated, end-to-end solutions required  Application-level control of system resources  Need to support new applications and technologies

6. 9/9/2015 13:17 6 System Research to Meet the Challenges Network Architecture Network Interface Design Resource Manager Scheduling Algorithms Scheduler Middleware Services Traffic Management Rate Control

7. 9/9/2015 13:17 7 Real-Time Spectrum Hard Soft

8. 9/9/2015 13:17 8 Real-Time OS Spectrum Hard Soft Real-Time Operating System General-Purpose Operating System VxWorks, Lynx, QNX,... Intime, HyperKernel, RTLinux (Windows NT, Linux)

9. 9/9/2015 13:17 9 Using General Purpose Operating Systems  GPOS offer some capabilities useful for real-time system builders  RT applications can obtain leverage from existing development tools and applications  Some GPOSs accepted as de-facto standards for industrial applications

10. 9/9/2015 13:17 10 Windows NT -- for RT applications?  Scheduling and priorities  Preemptive, priority-based scheduling non-degradable priorities priority adjustment  No priority inheritance  No priority tracking  Limited number of priorities  No explicit support for guaranteeing timing constraints

11. 9/9/2015 13:17 11 Windows NT -- for RT applications? (contd.)  Quick recognition of external events  Priority inversion due to Deferred Procedure Calls (DPC)  I/O management  Timers granularity and accuracy  High resolution counter with resolution of 0.8  sec.  Periodic and one shot timers with resolution of 1 msec.  Rich set of synchronization objects and communication mechanisms.  Object queues are FIFO

12. 9/9/2015 13:17 12 Talk Outline  Using off-the-shelf components for Real-Time applications?  Future application characteristics  Problems and challenges  Characteristics and limitations of OTS OSs  OTS component based solutions for distributed Real-Time applications  User-level scheduling of communicating RT tasks  Conclusions, recommendations, future

13. 9/9/2015 13:17 13 Goals - I  Evaluate the real-time capabilities of NT.  Identify areas where NT is (not) suitable for real- time applications.  Determine to what extent the unpredictable parts of NT can be “masked”.  Offer recommendations to designers using NT.

14. 9/9/2015 13:17 14 Priority Model Real-time class 26 25 24 23 22 16 Idle Above Normal Normal Below Normal Lowest Highest 31 Time-critical Dynamic classes 15 Time-critical 14 13 12 11 15 High class 1 Idle 9 8 7 11 Normal class 10 5 4 3 2 6 Idle class Thread Level

15. 9/9/2015 13:17 15 Thread Priority = Process class + level Real-time class 26 25 24 23 22 16 Idle Above Normal Normal Below Normal Lowest Highest 31 Time-critical Dynamic classes 15 Time-critical 14 13 12 11 15 High class 1 Idle 9 8 7 11 Normal class 10 5 4 3 2 6 Idle class Thread Level

16. 9/9/2015 13:17 16 Scheduling  Threads scheduled by executive.  Priority based preemptive scheduling. Interrupts Deferred Procedure Calls (DPC) System and user-level threads

17. 9/9/2015 13:17 17 Servicing an interrupt Power Failure...... APC Normal Exec. Dispatch/DPC Device X...... Interrupt Dispatch Table 1 Device interrupts Interrupt Service Routine DPC Routine Device Driver DPC DPC FIFO Queue DPC DPC FIFO Queue user-level threads Interrupts DPC

18. 9/9/2015 13:17 18 Servicing an interrupt Power Failure...... APC Normal Exec. Dispatch/DPC Device X...... Interrupt Dispatch Table 1 Device interrupts 2 Transfer control to ISR Interrupt Service Routine DPC Routine Device Driver DPC DPC FIFO Queue DPC DPC FIFO Queue user-level threads Interrupts DPC

19. 9/9/2015 13:17 19 Servicing an interrupt Power Failure...... APC Normal Exec. Dispatch/DPC Device X...... Interrupt Dispatch Table 1 Device interrupts 2 Transfer control to ISR Interrupt Service Routine DPC Routine Device Driver DPC DPC FIFO Queue DPC DPC FIFO Queue user-level threads Interrupts DPC

20. 9/9/2015 13:17 20 Servicing an interrupt Power Failure...... APC Normal Exec. Dispatch/DPC Device X...... Interrupt Dispatch Table 1 Device interrupts 2 Transfer control to ISR Interrupt Service Routine DPC Routine Device Driver DPC DPC FIFO Queue DPC DPC FIFO Queue user-level threads Interrupts DPC 3 Stop int.. and queue DPC

21. 9/9/2015 13:17 21 Servicing an interrupt Power Failure...... APC Normal Exec. Dispatch/DPC Device X...... Interrupt Dispatch Table 1 Device interrupts 2 Transfer control to ISR Interrupt Service Routine DPC Routine Device Driver DPC DPC FIFO Queue DPC DPC FIFO Queue user-level threads Interrupts DPC 3 Stop int.. and queue DPC

22. 9/9/2015 13:17 22 Servicing an interrupt Power Failure...... APC Normal Exec. Dispatch/DPC Device X...... Interrupt Dispatch Table 1 Device interrupts 2 Transfer control to ISR Interrupt Service Routine DPC Routine Device Driver 3 Stop int.. and queue DPC DPC DPC FIFO Queue DPC DPC FIFO Queue 4 Task level drops and DPC can execute user-level threads Interrupts DPC

23. 9/9/2015 13:17 23 Servicing an interrupt Power Failure...... APC Normal Exec. Dispatch/DPC Device X...... Interrupt Dispatch Table 1 Device interrupts 2 Transfer control to ISR Interrupt Service Routine DPC Routine Device Driver 3 Stop int.. and queue DPC DPC DPC FIFO Queue DPC DPC FIFO Queue 4 Task level drops and DPC can execute 5 Transfer control to driver’s DPC user-level threads Interrupts DPC

24. 9/9/2015 13:17 24 Servicing an interrupt Power Failure...... APC Normal Exec. Dispatch/DPC Device X...... Interrupt Dispatch Table 1 Device interrupts 2 Transfer control to ISR Interrupt Service Routine DPC Routine Device Driver 3 Stop int.. and queue DPC DPC DPC FIFO Queue DPC DPC FIFO Queue 4 Task level drops and DPC can execute 5 Transfer control to driver’s DPC 6 Execution of DPC routine user-level threads Interrupts DPC

25. 9/9/2015 13:17 25 I/O Handling  I/O request is sent to device driver.  Device completes operation and interrupts.  Complete I/O request. Buffered I/ODirect I/O APC Device System User’s space Device System User’s space (Keyboard, mouse) (disk, network)

26. 9/9/2015 13:17 26 Prototype Software Architecture Operator Station Acquisition &Control Equipment Heartbeat Timer Command Ack. Real Video Receiver Producer Consumer Buffer Highest Priority Normal Priority Real-Time Class

27. 9/9/2015 13:17 27 Performance Metrics  Round Trip Time (RTT) as seen by the operator input.  Rate of execution of sensor data processing entities.  Quality of the video output.

28. 9/9/2015 13:17 28 Workload  Number of sensor data streams (2-20).  Period of new sensor values (10-1000 ms).  Period of control messages (10-100 ms).  Amount of work done in processing data.  One Video and audio.

29. 9/9/2015 13:17 29 Operator Command Performance Two 1KB Sensor data streams 1 second update rate 30 ms period for control messages No work

30. 9/9/2015 13:17 30 Operator command Performance NT Scheduler - same RT priority 16 Sensor data streams (1KB) Update rate (100, 200, 500, 1000 ms) 90 ms period for control messages work

31. 9/9/2015 13:17 31 Operator command Performance (user-level scheduling) Rate Monotonic with limited levels 16 Sensor data streams (1Kb) Update rate (100, 200, 500, 1000 ms) 50 ms period for control messages work Time-cognizant dispatcher + plan 8 Sensor data streams (1Kb) Update rate (100, 200, 500, 1000 ms) 100 ms period for control messages work

32. 9/9/2015 13:17 32 Design principles and recommendations:  Do not depend on NT scheduler to accomplish timing behavior in interactive applications.  Utilize user-level scheduling to achieve higher predictability.  If possible, characterize duration of I/O activity and its frequency.  Lock pages in memory for real-time threads.  Manage and control utilization of systems resources.

33. 9/9/2015 13:17 33 Use a General Purpose OS for RT?  It is possible to improve the predictability of real-time tasks.  It is not possible to “mask” all sources of unpredictability within NT “as is”.  Designer needs to be aware of the effects DPC queue on any user thread.  I/O handling.  Prototype application demonstrates the uses of the recommendations. K. Ramamritham, C. Shen, O. González, S. Sen, S.B. Shirgurkar. Using Windows NT for Real-Time Applications. In Proc. of the 4th IEEE Real-Time Technology and Applications Symposium, Denver, CO, June 1998.

34. 9/9/2015 13:17 34 Talk Outline  Using off-the-shelf components for Real-Time applications?  Future application characteristics  Problems and challenges  Characteristics and limitations of OTS OSs  OTS component based solutions for distributed Real-Time applications  User-level scheduling of communicating RT tasks  Conclusions, recommendations, future

35. 9/9/2015 13:17 35 Challenges in Supporting Communicating Real-Time Tasks Network Architecture Network Interface Design Resource Manager Scheduling Algorithms Scheduler Middleware Services Traffic Management Rate Control

36. 9/9/2015 13:17 36 Communications Middleware  Transporting multiple media types, control, data, visual, image, alarms, video and audio  Integrating control with visual monitoring  Enabling plug-and-play  QoS management

37. 9/9/2015 13:17 37 Writer DPA_1 DPA_n Reader_i Network Node 1 Node 2 ReMA ReMA = Reflective Memory Area Real-Time Channel-based Reflective Memory (RT-CRM) DPA = Data Push Agent DRA = Data Receive Agent DRA_n

38. 9/9/2015 13:17 38 RT-CRM Operation Modes Synchronous/Asynchronous Writer DPA_1 DPA_n Reader_i Network Node 1 Node 2 ReMA Blocking / Non-blocking

39. 9/9/2015 13:17 39 Network Node 1 Node 2 Using MidART services Write Cmd DPA Process Cmd DPA Result Synchronous mode

40. 9/9/2015 13:17 40 End-to-End QoS Provisions Writer DPA_1 DPA_n Reader_i Network Node 1 Node 2 ReMA Memory-to-Memory Application-to-Application

41. 9/9/2015 13:17 41 GPOS Problems  No priority inheritance  Among user processes/threads  No Priority tracking  From user to system/network threads  Limited number of priorities  No explicit support for guaranteeing timing constraints

42. 9/9/2015 13:17 42  Problem alleviated: No priority inheritance and limited priorities  No priority tracking  Hidden protocol stack  Priority inversion Server-based User Level Scheduling  Solution:  Modified Dual Priority Scheduling

43. 9/9/2015 13:17 43 Dual Priority Scheduling (York) Middle Band Low Band High Band Three Priority Bands Non-real-time tasks Real-time tasks before promotion time Real-time tasks after promotion time

44. 9/9/2015 13:17 44 Dual Priority in MidART Middle Band Low Band High Band Time Critical Band Four Priority Bands Non-real-time tasks Real-time tasks before promotion time Critical real-time tasks Real-time tasks after promotion time

45. 9/9/2015 13:17 45 Communication Server ReMA DPA ReMA DPA ReMA DPA High Promoted Middle Low Queues Rate Controllers Data Pusher To network @ promotion time  Request = actual messages  Queues limited priorities Calculation of promotion time

46. 9/9/2015 13:17 46 Admission control  To calculate the schedulability T_i: Promotion Time i = Deadline i - worst case response time i worst case response time i = worst case delay (w i ) + release jitter i w i m+1 = C i + S o + [  w i m + Jj C j ] j  hp(i) T j Computation timeBlocking time Interference of high priority tasks

47. 9/9/2015 13:17 47 Rate Control with Uniform Message Size  Bounding the priority inversion time due to non-preemptive message transmission.  The “optimal” message size is a platform dependent parameter.  Receiving node capability is the key - the known live-lock problem.  Match value between the system timer granularity and the data push latency. (E.g., in our case, 8KB)

48. 9/9/2015 13:17 48

49. 9/9/2015 13:17 49 Communication Server ReMA DPA ReMA DPA ReMA DPA High Promoted Middle Low Queues Rate Controllers Data Pusher To network @ promotion time STOP Uniform message size to allow transmission with fix preemption points

50. 9/9/2015 13:17 50 Communication Server ReMA DPA ReMA DPA ReMA DPA High Promoted Middle Low Queues Rate Controllers Data Pusher To network Uniform message size to allow transmission with fix preemption points

51. 9/9/2015 13:17 51  Problem alleviated:  No priority inheritance and limited priorities  No priority tracking  Hidden protocol stack  Priority inversion Server-based User Level Scheduling  Solution:  Modified Dual Priority Scheduling  Real-time data pusher  Rate control  Uniform message size  Integration of real-time and non-real-time applications on the same computing and communication platform

52. 9/9/2015 13:17 52 Industrial Control Environment EWS Network Field Devices Field Network PLC Field Devices Field Network PLC Operator Stations Periodic Monitoring (64KB @ 10 msec) Command and control (1KB @ 20 msec) Server

53. 9/9/2015 13:17 53

54. 9/9/2015 13:17 54

55. 9/9/2015 13:17 55 Summary: User Level Communication  Communication Scheduler  End-to-End Task Model  Local scheduling of precedence constrained distributed tasks  Integration of real-time and non-real-time tasks C. Shen, O. González, K. Ramamritham, and I. Mizunuma. User Level Scheduling of Communicating Real- Time Tasks. In Proc. Of fifth IEEE Real-Time Technology and Applications Symposium, Vancouver, Canada, June 1999.

56. 9/9/2015 13:17 56 Conclusion  Characterization and understanding of limitations imposed by GPOS.  Development and evaluation of an efficient user-level scheduling scheme for communicating real-time tasks  Support coexistence of real-time and non-real-time using general purpose operating systems and networks  Ongoing work for QoS management support at the middleware level

57. 9/9/2015 13:17 57 Incorporation of Quality of Service  QoS management is needed to support multiple real-time applications on single platform  Operating system  Application  Dynamic QoS management.  Extend QoS guarantees provided by network to end point applications  Predictable performance for real-time and multimedia applications according to user QoS parameters.

58. 9/9/2015 13:17 58 System Research to Meet the Challenge Network Architecture Network Interface Design Resource Manager Scheduling Algorithms Scheduler Middleware Services Traffic Management Rate Control Qos Management

59. 9/9/2015 13:17 59 Research Questions  What type of support is needed to optimize resource use?  How to implement efficient QoS adaptation at the middleware level? The real problem  Integrated end-to-end Quality of Service management  From the mouse to the wire (User to Network)  From the wire to the glass (Network to Display)

60. 9/9/2015 13:17 60 Problems  Specification of user QoS requirements  Efficient adaptation mechanisms  Mode change  Integrated scheduling of multiple resources  Resource reservation  Account for interrupt in a GPOS  Support for management of trade-offs between different types of requirements  dependability vs. performance