©SCILabs 2002www.SCILabs.es Real-Time CORBA ©SCILabs Ingenieros S.L Miguel Segarra José Antonio Clavijo HRTC Vienna Plenary Meeting

©SCILabs 2002www.SCILabs.es Speech Contents Introduction Real-Time/Minimum CORBA development The DOTS Project Conclusions

©SCILabs 2002www.SCILabs.es Introduction What is CORBA? Useful URLs  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es What is CORBA It is an specification of the OMG CORBA specifies an architecture for building distributed object systems, common services for those systems and protocols for interoperability among CORBA implementations.  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es What is CORBA?  Introduction  RT CORBA Dev.  DOTS Project  Conclusions Horizontal CORBA Facilities Object Request Broker CORBA Services Vertical CORBA Facilities Application Objects The CORBA “software bus”

©SCILabs 2002www.SCILabs.es What is CORBA? UML Modeling CORBAservices CORBAfacilities Business Object Facility Common Business Objects CORBA Domains CORBA Domains CORBA Domains Meta-Object Facility SECURITY IDL Interfaces, Mappings, & ORB Realtime, Embedded options Interoperability: IIOP, Asynch Components, Scripting  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es What is CORBA? How does it work?  Introduction  RT CORBA Dev.  DOTS Project  Conclusions Server Side Client Side Object Request Broker IDLIDL IDLIDL POA CLIENTCLIENT SERVANTSERVANT IDL

©SCILabs 2002www.SCILabs.es What is CORBA? Object Request Broker IDLIDL IDLIDL POA CLIENTCLIENT SERVANTSERVANT  Introduction  RT CORBA Dev.  DOTS Project  Conclusions How does it work? Client.GetValue(); Servant.GetValue();

©SCILabs 2002www.SCILabs.es What is CORBA? Object Request Broker IDLIDL IDLIDL POA CLIENTCLIENT SERVANTSERVANT Object Request Broker IDLIDL IDLIDL POA CLIENTCLIENT SERVANTSERVANT  Introduction  RT CORBA Dev.  DOTS Project  Conclusions GIOP Client.GetValue(); Servant.GetValue();

©SCILabs 2002www.SCILabs.es Useful URLs Our Home Page: – OMG Home Page: – OMG Specs: – spec_catalog.htm OMG News: – Lots of OO/CORBA URLs –  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Speech Contents Introduction Real-Time/Minimum CORBA Development The DOTS Project Conclusions

©SCILabs 2002www.SCILabs.es RT/Minimum CORBA Development CORBA features for real- time/embedded development Specifications vs. Reality RT CORBA Tools for System Design Some Lessons Learnt Industry Trends  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Real-Time/Minimum CORBA Features Characteristics of RT CORBA systems –End-to-end system predictability (not high performance) –Allocation of resources Characteristics of Minimum CORBA systems –Limited resources –Full CORBA interoperability  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Real-Time/Minimum CORBA Features End-to-end predictability –Respecting thread priorities between clients and servers –Bounding the duration of thread priority inversion –Bounding latencies of operation invocations  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Real-Time/Minimum CORBA Features Management of resources –Process, storage and communication Threadpools Priorities Number of concurrent requests Transport connections  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Real-Time/Minimum CORBA Features Real-Time CORBA 1.0 addresses fixed-priority real-time systems –Priority-based scheduling rather than e.g. deadline based Real-Time CORBA 2.0 addresses dynamic scheduling  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Real-Time/Minimum CORBA Features Minimum CORBA defines a cut down profile of CORBA Dynamic aspects of CORBA are removed  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es RT/Minimum CORBA Development CORBA features for real- time/embedded development Specifications vs. Reality RT CORBA Tools for System Design Some Lessons Learnt Industry Trends  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Real-Time CORBA Specifications vs. Reality  Introduction  RT CORBA Dev.  DOTS Project  Conclusions Messaging CORBA Minimum CORBA Time Service But close to the process resources are very limited!

©SCILabs 2002www.SCILabs.es Specifications vs. Reality  Introduction  RT CORBA Dev.  DOTS Project  Conclusions CORBA Minimum CORBA Real-Time CORBA Messaging Time Service Dynamic part of CORBA In embedded/real-time systems a lot of decisions are made at design time!

©SCILabs 2002www.SCILabs.es RT/Minimum CORBA Development CORBA features for real- time/embedded development Specifications vs. Reality RT CORBA Tools for System Design Some Lessons Learnt Industry Trends  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Stub Client Real-Time CORBA Tools For System Design  Introduction  RT CORBA Dev.  DOTS Project  Conclusions thread priority Provide predictability by controlling ORB behavior ORB connection ORB Invoke priority protocol Skeleton POA thread priority concurrency Servant

©SCILabs 2002www.SCILabs.es Stub Client Real-Time CORBA Tools For System Design  Introduction  RT CORBA Dev.  DOTS Project  Conclusions Impact on ORB architecture of predictability issues ORB buffering ORB delay Skeleton POA thread dispatching Servant buffering request dispatching marhalling Memory mgmt

©SCILabs 2002www.SCILabs.es Real-Time CORBA Tools For System Design Real-time CORBA features –RT ORB and RT POA –RT CORBA priority and priority mappings –RT CORBA priority models –Priority transforms –RT CORBA mutex –Threadpools –Connection management –Protocol selection –Invocation timeout –Bounding of ORB priorities  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Real-Time CORBA Tools For System Design RTCORBA::RTORB –It is an extension of the CORBA::ORB with operations to create new real-time entities (e.g. threadpool, mutex and new RT policies) RTPortableServer::POA –Add to the PortableServer::POA operations to set priority on an object reference basis  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Real-Time CORBA Tools For System Design  Introduction  RT CORBA Dev.  DOTS Project  Conclusions 0 RTCORBA::Priority OS #1 native priority model 4 25 OS #2 native priority model Real-Time CORBA Priority An OS independent priority scheme. The same range of CORBA priorities for all platforms. Custom priority mappings are allowed.

©SCILabs 2002www.SCILabs.es Real-Time CORBA Tools For System Design Real-Time CORBA Priority Models –Client propagated –Server declared –Per object reference basis  Introduction  RT CORBA Dev.  DOTS Project  Conclusions Stub Client ORB Skeleton POA Servant What is the invocation Priority?

©SCILabs 2002www.SCILabs.es Real-Time CORBA Tools For System Design Client propagated priority model  Introduction  RT CORBA Dev.  DOTS Project  Conclusions Stub Client ORB Skeleton POA Servant client running at priority 100 Invocation executed at priority 100 Client’s priority propagated along the path to server in a service context

©SCILabs 2002www.SCILabs.es Real-Time CORBA Tools For System Design Server declared priority model  Introduction  RT CORBA Dev.  DOTS Project  Conclusions Stub Client ORB Skeleton POA Servant client running at priority 100 Invocation executed at priority 3347 Client’s priority IS NOT propagated along the path to server in a service context

©SCILabs 2002www.SCILabs.es Real-Time CORBA Tools For System Design Priority transforms –Inbound –Outbound  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Real-Time CORBA Tools For System Design Real-Time CORBA mutex –An API to give access to the mutex implementation that the ORB uses –This gives consistency to the priority protocol used –Basic tool for resource protection  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Real-Time CORBA Tools For System Design Threadpools –Server-side tool to handle invocations –Provide concurrency control –Allows thread pre-creation and reuse –Setup of idle threads priority Two flavors of threadpools –Without lanes –With lanes  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Real-Time CORBA Tools For System Design Threadpools  Introduction  RT CORBA Dev.  DOTS Project  Conclusions POA APOA B ORB threadpool POA APOA B ORB Threadpool AThreadpool B

©SCILabs 2002www.SCILabs.es Real-Time CORBA Tools For System Design Threadpools  Introduction  RT CORBA Dev.  DOTS Project  Conclusions Threadpool Lane ALane B Without lanes - stacksize prio Static 5 - Dynamic 3 - Req buff Yes - Max Reqs 3 - Req Buf Sz 5120 With lanes - Stacksize Lanes 2 - Borrowing Yes - Req buff Yes - Max Reqs 3 - Req Buf Sz Lane A -Prio Static 2 -Dynamic 1

©SCILabs 2002www.SCILabs.es Real-Time CORBA Tools For System Design Handling connections –Implicit and explicit binding –Multiplexed connections –Private connections –Banded connections  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Real-Time CORBA Tools For System Design ORB client server Multiplexed: Resource contention by shared connection ORB client server Private connection: Exclusive use of a connection by a client  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es ORB Real-Time CORBA Tools For System Design ORB client server Banded connections: Invocation priority determines connection to use  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Real-Time CORBA Tools For System Design Protocol configuration –It can be done at the client or server side of the ORB –Available protocols are exposed to clients in IORs when the configuration is set from the server side  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Real-Time CORBA Tools For System Design  Introduction  RT CORBA Dev.  DOTS Project  Conclusions RTOS A Client ORB AORB B Skeleton RTOS B Servant Stub TCP/IPATM OtherATMTCP/IP Object Reference Invocation

©SCILabs 2002www.SCILabs.es Real-Time CORBA Tools For System Design  Introduction  RT CORBA Dev.  DOTS Project  Conclusions RTOS A Client ORB AORB B Skeleton RTOS B Servant Stub TCP/IPATM OtherATMTCP/IP  Invocation timeout

©SCILabs 2002www.SCILabs.es RT/Minimum CORBA Development CORBA features for real- time/embedded development Specifications vs. Reality RT CORBA Tools for System Design Some Lessons Learnt Industry Trends  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Some Lessons Learnt In general, sources of unsatisfactory results are not easy to find. Pattern-based development provides good results but... there is no ORB pattern. Improving the ORB by experimenting is of great help. Ask “Why...?”. Run lots of tests.  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Some Lessons Learnt Performance –Avoid new/delete. Expensive and leads to memory fragmentation. –Avoid linear searches Unmarshalling IORs DeMux to object implementation and operation invocation. –Copying of data: From client to ORB to network. From network to ORB to server.  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Some Lessons Learnt Performance. –Buffer resizing strategy. –Threads life-cycle. Keep threads alive but idle. –Avoid overhead of first invocation. Early binding. –Compare to raw-socket application to find ORB overhead. –Measure ORB behavior with varying data sizes. (must be linear)  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Some Lessons Learnt Predictability –Worst time case is hard to find: several sources of non determinism (application, ORB, OS, network). –Configure the ORB in a simpler manner to reduce the number of “changing things” (e.g. Use static buffers). –Being predictable introduces ORB overhead (priority service contexts)  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Some Lessons Learnt Footprint –What is too much? –Performance vs. Footprint –Performance vs. Services provided –Memory left on board by the ORB for the application –Bound memory usage by the ORB –Size of the ORB library  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es RT/Minimum CORBA Development CORBA features for real- time/embedded development Specifications vs. Reality RT CORBA Tools for System Design Some Lessons Learnt Industry Trends  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Industry Trends The embedded industry is not used to object solutions Embedded systems have issues not considered in the RT CORBA specification There is no embedded CORBA specification Let the user configure default behavior of the ORB (e.g. Default multiplexing of connections) Preallocation and control of ORB resources  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Industry Trends Usage of multiple protocol stacks: adopt the OCI framework The ORB architecture must be open to developers: configure ORB in a “plug-in” fashion. Tracing execution and debugging tools Specialised services for real-time  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Speech Contents Introduction Real-Time/Minimum CORBA Development The DOTS Project Conclusions

©SCILabs 2002www.SCILabs.es The DOTS Project Overview Participants Objectives Domain Elitel-5: an embedded device Some results  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Overview –To establish an open software model, built upon real-time distributed object technologies and emergent telecontrol standards to allow the optimum exploitation of the interoperation capabilities of devices and systems in the distributed context of an electric power grid.  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Participants  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Objectives Distributed Object Telecontrol Systems & Networks IST DOTS Real-Time CORBA Minimum CORBA IEC for control systems in electrical substations  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Objectives From the IEC draft 1.The functions of the substation automation system (SAS) and their communication behaviour shall be described device independent. 2.The functions shall be described as far as necessary only to identify the information to be exchanged. 3.The free allocation of functions to devices shall be supported. 4.The functions used today are well known but the standard shall be open for future requirements. 5.The interaction of device independent distributed functions shall be described by the logical interfaces in between which may be freely allocated to physical interfaces or LANs. 6.The standard shall define generic information to be communicated and the communication behaviour of the functions to provide for planned and future functional extensions of substation automation system.  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es The DOTS Project  Introduction  RT CORBA Dev.  DOTS Project  Conclusions Domain of application

©SCILabs 2002www.SCILabs.es The DOTS Project The ELITEL-5 IEDs  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es The DOTS Project  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es The DOTS Project The pilot application  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es The DOTS Project Elitel 5 RTORB Real-Time POA Client Propagated Priority Model Threadpools ICa MinRT VisiBroker OIT IIOP Process IEC  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es The DOTS Project Physical devices –OIT: Operator Interface Tool –IED1: Elitel 5 with blocking control –IED2: Elitel 5 with circuit breaker and switchs Logical Nodes –IHMI: Human-Machine Interface –XCBR: Circuit Breaker –XSWI: Switch –CILO: Inter-locking Supervision of analog and digital signals and digital refresh rate of 100ms  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es The DOTS Project  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es The DOTS Project  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es The DOTS Project Some results –DOTS is the first implementation of the IEC standard –Contribution to the adoption of the IEC standard General Model Definition Specific Communication Service Mapping-Mapping to CORBA –Embedded IEC SW for IED and RTU over the SCILabs’ ICa ORB  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es The DOTS Project Real-time ICa An SCT and OIT capable to manage IEC objects  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es The DOTS Project Some results of SCILabs  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Speech Contents Introduction Real-Time/Minimum CORBA Development The DOTS Project Conclusions

©SCILabs 2002www.SCILabs.es Conclusions Real-Time CORBA is beginning to be used by the RT/embedded systems community There is need for an embedded CORBA specification Battle to improve performance, footprint and ORB determinism  Introduction  RT CORBA Dev.  DOTS Project  Conclusions

©SCILabs 2002www.SCILabs.es Conclusions Thank you!  Introduction  RT CORBA Dev.  DOTS Project  Conclusions