1. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology Flight Software Through the Looking Glass Scott Burleigh Jet Propulsion Laboratory California Institute of Technology 12 December 2013 This research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration. (c) 2013 California Institute of Technology. Government sponsorship acknowledged.

2. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology Flight Software is TWO systems We all know that real-time flight software is indispensable to spacecraft operations. It drives the hardware of a robotic vehicle and its instruments, and it has been present since deployment of the first flight computer. As missions become more capable, a second system emerges. It’s a non-real-time system that manages data. 12 December 20132

3. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology The second system The software we usually think of when we say “flight software” is the real-time, safety-critical, interrupt- driven foreground system. The other part of “flight software” – which we typically don’t recognize as being different in nature – is the non-real-time, discretionary, time-sharing background system. 12 December 20133

4. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology Where it lives Scheduling theory tells us that the real-time tasks’ time allocations should add up to no more than about 60% of CPU time, to ensure they never miss deadlines. The remaining 40% of CPU – “idle time” – isn’t necessarily idle. It’s time that is available for interruptible tasks, the second system. That second system should include everything that is not truly real-time. 12 December 20134

5. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology Second system overview All tasks run at the same priority, the lowest priority supported by the O/S. – Possibly multiple background subsystems. – To the real-time FSW they all look like “idle”. Tasks have no deadlines. – Locking one another out for prolonged periods is okay, so long as everybody gets a chance to run eventually. – Each task must release the CPU on finishing a unit of work, usually blocks on something. 12 December 20135

6. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology A Practical Example The Flight Software architecture of the Curiosity rover is clearly an excellent approach to the design of a FSW real-time system (but other good approaches are possible). Postulate: the Interplanetary Overlay Network (ION) architecture, an implementation of Delay-Tolerant Networking, is a good approach to the design of a FSW non-real-time system (but again other good approaches are possible). 12 December 20136

7. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology ION Design Overview Application Object database Routing SenderReceiver issuedeliver transmitreceivedispatch bundles for delivery bundles to be forwarded 12 December 20137

8. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology Use shared memory. – Often there’s no protected memory, so we have no option. – But this can be turned to advantage: shared memory is a highly efficient way to pass data between flight software tasks. ION Design Principles senderreceiver Zero-copy procedures: leverage shared memory to minimize processing overhead. –Encapsulation in layers of protocol overhead (headers and trailers) can be done by reference rather than by copy. –The same data object can be shared by multiple tasks, provided reference counting prevents premature deletion. Portability: this is an unfamiliar programming model, so we must make it easy to develop in an environment with good programming support (e.g., Linux) and then deploy – without change – in the target RTOS environment. S linked list givetake enqueuedequeue 12 December 20138

9. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology Through the looking glass The foreground and background systems of a spacecraft’s FSW share many qualities that distinguish them – both – from workstation or PC software. But in some ways the character of the background system is the exact inverse of the character of the real-time FSW. And the two can coexist in perfect compatibility. It’s been demonstrated. 12 December 20139

10. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology What they have in common… …is most of the flight project cultural values: – Project schedule can’t be jeopardized. – Module coupling must be minimized. – No dynamic system memory allocation. – Make maximum use of available resources. – Contain and, as possible, tolerate faults. – Test as you’ll fly, fly as you tested. – Formal, controlled development process. 12 December 201310

11. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology Where they differ What is optimized – Determinism – Portability Mutual exclusion Data sharing Message passing Memory management 12 December 201311

12. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology What is optimized The foreground system must be reliable, else you lose the spacecraft. – Fixed scope enables minimal, mission-specific design which enables comprehensive testing. The background system must be efficient, else not enough work gets done. – Minimize wasted space and cycles. – This adds complexity, so it’s important to support portability: reliability comes from extensive multi- mission testing history. 12 December 201312

13. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology Mutual exclusion Real-time FSW can’t tolerate lengthy mutual exclusions: tasks miss deadlines. In the background system, no problem. You can serialize an entire subsystem on a single mutex, because there are no deadlines. – Only one task runs at a time anyway. (For now; multi-core flight computers are on the horizon.) – Loops and function calls in critical sections are okay, so long as they eventually terminate and let other tasks run. – Minimize cycles spent on task switching. 12 December 201313

14. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology Data sharing Data sharing requires mutual exclusion, so it’s no good in the foreground system. In the background system, lengthy mutual exclusion is okay – so shared access to data is okay. Which is good, because shared access is also the fastest way multiple tasks can operate on the same data. 12 December 201314

15. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology Message passing Because shared access is excluded, the real-time system uses message passing to enable multiple tasks to operate on common data. But because shared access is okay in the background system, message passing is not needed. – No cycles wasted in copying anything. – Signal “data ready” by giving a semaphore. 12 December 201315

16. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology Memory management Because mission scope is fixed and is known at launch time, all foreground FSW memory can be in fixed-length arrays, each with margin. But the background system gains reliability from being multi-mission, hence portable and evolvable. – Management of private common heap: pooled resource, pooled margin, efficient use of space. – Automatically adapts to mission scope change. 12 December 201316

17. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology How they collaborate The foreground system never calls the background system’s library functions – never blocks, ignores background tasks. Background tasks are interrupted whenever foreground tasks need to run. For communication between the two: operating system (e.g., VxWorks) message queues. – Non-blocking at the foreground end. – Blocking at the background end. 12 December 201317

18. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology cfdp_manager pxlso pxlsi PX LTP LTP segs S S PxVdb libfdput_px.so To telecom system and earth Out FPDU In FPDU S S pxout pxin PxDb Outbound FPDUs Inbound FPDUs How we know this works 12 December 201318 The Deep Impact Network (DINET) project validated ION in flight, on the EPOXI (formerly Deep Impact flyby) spacecraft in October-November 2008. The ION non-real-time data management system was integrated with DI’s real-time flight system, written by Ball Aerospace. No modification to either the Ball flight software or the ION multi-mission software was required; a small “link service adapter” system, named “PX”, provided the interface between the foreground and background systems. The ION software enabled EPOXI to function as a DTN router in deep space, about 10 million miles from Earth, for four weeks while the spacecraft was in cruise to comet Hartley 2. About 14.5 MB of data (292 images) were routed through the spacecraft over 8 low-rate DSN tracking passes, without data loss and without software failure.

19. National Aeronautics and Space Administration Jet Propulsion Laboratory, California Institute of Technology Thanks! 12 December 201319