The System-Level Simplex Architecture Stanley Bak Olugbemiga Adekunle Deepti Kumar Chivukula Mu Sun Marco Caccamo Lui Sha

Building Reliable Software Use best practices from industry: –Software review –“Safe” programming languages –Extensive testing Cost: –$100-$1000 per line of code And worst of all…

“Reliable” Software Still Has Bugs! In 2007, 12 F-22s were going from Hawaii to Japan After crossing the IDL, all 12 experienced multiple system crashes –No navigation –No fuel subsystems –Limited communications –Rebooting didn’t help Formal Verification? –F-22 has 1.7 million lines of code

Our Contribution Our contribution is threefold: We present the System-Level Simplex Architecture that provides reliability for large, safety-critical systems. We formalize and verify our architecture in an AADL model, which can be immediately instantiated for applications requiring reliability. We demonstrate the System-Level Simplex Architecture in an Inverted Pendulum control system, and empirically verify its safe functionality in spite of controller/OS/middleware bugs.

Trends in Cross-Layer Systems Reliability is a cross-layer property Other examples of cross-layer properties include security, and real-time Hardware Operating System Software

Trends in Security Security protocols at the software-level (SSL) trust the operating system and hardware is secure An operating system can be compromised by a kernel rootkit or VMBR Solution: Use the hardware for security checks (Secure Boot, TPMs)‏ Hardware - TPM Operating System - SELinux Software - SSL

Trends in Real-Time Systems A real-time software application requires the operating system be aware of real-time requirements and the hardware be predictable A real-time operating system can use a real-time scheduling algorithm, but can do nothing in the face of unpredictable hardware Ideal Solution: Design hardware predictably (ASICs, deterministic hardware)‏ Practical Solution: Enforce predictable behaviour (bus monitoring and cutoff)‏ Hardware – Deterministic Processor Operating System - VxWorks Software – Flight Controller

Trends in Reliability Designing 100% correct, complex control software is infeasible Operating systems can provide isolation (microkernel) and power through abstractions, but are often large, complex, and unverified Ideal Solution: Design hardware reliably; verify all software Practical Solution: Reject OS abstractions? Accept failure? Hardware – ??? Operating System – MINIX 3 (microkernel)‏ Software – Complex Flight Controller

Trends in Reliability Designing 100% correct, complex control software is infeasible Operating systems can provide isolation (microkernel) and power through abstractions, but are often large, complex, and unverified Ideal Solution: Design hardware reliably; verify all software Practical Solution: System-Level Simplex Hardware – System-Level Simplex Operating System – MINIX 3 (microkernel)‏ Software – Complex Flight Controller

System-Level Simplex System-Level Simplex works components off the shelf (COTS) is compatible with existing engineering practices (triple modular redundancy)‏ First, develop a simple, safe controller in hardware (on a Field Programmable Gate Array [FPGA])‏ Next, develop a complex controller that can take advantage of the power of software (COTS processor + hardware)‏ Then use the complex controller when possible, but switch to the simple one to preserve system liveliness

Physical Components Motherboard CPU Memory Ram Field Programmable Gate Array (Xilinx ML505)‏ sensors actuators sensors actuators Complex Controller Decision Module Safety Controller PCIe Bus North Bridge Bus Front Side Bus Logical Mapping

Proof of Safety AADL is an architecture description language designed for real-time, embedded systems –Used by European Space Agency, Rockwell-Collins, Lockheed Martin, Airbus, and others Systems can be instantiated from an AADL Model Safety properties of a model can be proven using model checking

System-Level Simplex Model We provide a System-Level Simplex AADL Model generator to generate an initial architecture design This model is modified as the design evolves The final AADL design can then be checked for violation of System-Level Simplex requirements

System-Level Simplex: Inverted Pendulum Testbed

Overview Complex Controller CPU with Linux OS PCIe Bus IO Module Analog Input FPGA Safety Controller Decision Module A/D Converter D/A Converter Analog Output Bus module Type checker

Inverted Pendulum An inverted pendulum is an unstable system that tries to maintain an upright rod by moving the base along a track (video)‏video We used the Quanser Q4 IP04 inverted pendulum for our testbed The pendulum tells us the angle and track position which we convert to a digital signal with an A/D Converter (ADS7812P)‏ We output the digital voltage for the motor to use, which is converted to an analog using an A/D converter (DAC714P)‏

Hardware Components Our hardware components run on a Xilinx ML505 Field Programmable Gate Array (FPGA)‏ The safety controller code can be generating in Matlab given the physical properties of the inverted pendulum The decision module switches controllers when the pendulum is in danger of collapse. We can compute this state region with a Lyapunov stability function.

Decision Module Recoverable Region: Based on the dynamics of the controlled physical system, we can derive a stability envelope. Here, any state inside the red region is recoverable if we use the safety controller. Safe Region: When the state is in the green region, the system can tolerate aggressive action without immediately losing stability (we can use the complex controller). FPGA Safety Controller Envelope Calc A/DD/A Bus module Type checker PCIe Bus Software State Space

Software Components Our complex controller runs on a x86 PC with Linux RK (a real-time Linux variant)‏ The software components are interfaced with the FPGA through the PCIe bus Communication occurs through memory mapped I/O, where sensor and actuation values are viewed as memory on the FPGA The complex controller is a modified version of the safe controller with various bugs to test the System-Level Simplex design

Implementation Results By introducing bugs in the complex controller, we were able to verify that the System-Level Simplex Architecture protected the system from several potential failures

Conclusions and Future Work We proposed a system-level simplex architecture which provides reliability at the lowest (hardware) level We provide an AADL architecture generator and checker We demonstrated the System-Level Simplex Architecture on an Inverted Pendulum Control System, and empirically verified its functionality. The System-Level Simplex Architecture has fostered a funded collaboration with John Deere applied towards autonomous tractor control