1. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 20111 Unique Software Considerations in Military and Avionic Embedded Systems Embedded Systems Conference Boston 2011 ESC-402 Dave Stewart, PhD Director of Research and Systems Integration InHand Electronics dstewart@inhand.com www.inhand.com

2. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 20112 Overview Most software presentations on Military and Avionic Systems focus on two key items: Reliability and Fault Tolerance Security This presentation focuses on many other aspects of software that are somewhat unique to military and avionic systems, yet are rarely discussed

3. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 20113 Outline Is that REALLY the Requirement? Self Destruct! Development Environment != Deployed System Sorry, no debug I/O allowed, but debug it quick! Error detection: Why bother? It Uses Too Much Power!

4. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 20114 Is that REALLY the Requirement? Understanding the Real Requirements Hundreds of pages of requirements This is typical for military and avionic applications If only tens of pages, first step is likely a need to refine the requirements! Separate between wish-list and really really required Many requirement documents are the culmination of the wish- lists of everyone involved in prior projects Given fixed cost contracts, understand which ones have flexibility, and which do not The systems engineer should look closely for any conflicting requirements; address those first!

5. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 20115 Is that REALLY the Requirement? The Impossible Requirements It is not unusual to find requirements that seem like they will be very difficult to meet When this happens, investigate the requirement carefully, and make sure what the customer wants is properly captured Understand every Use-Case Scenario. Often the documented requirement does not actually meet the needs of the use cases Yet, a simpler requirement might actually meet the needs of all use cases Review how that requirement was implemented in previous or competitor projects Some requirements are designed to capture what was implemented on a prior system, whether in-house or by a competitor Review those other implementations closely, as more often than not, the customer really wants the same thing, not something new that is substantially more complex

6. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 20116 Self Destruct! Zeroization Many requirements call for the zeroization, but the actual meaning is left to interpretation. The interpretation is often erase flash upon either user activation or detection of tamper This is often overkill. The real requirement imposed by military and security agencies is usually that an enemy who gains hold of the device is unable to access any classified data Therefore unencrypted classified code and data must be erased quickly and reliably Ask the customer Do you really need to erase an operating system that is readily available in the public domain?

7. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 20117 Self Destruct! Understand the Use Cases If the customer insists they mean erase all flash, challenge their understanding of the implications This is often not what the end user really wants, even if they are asking for it During requirements reviews, understand ALL the use-cases. For example, what if there is a false alarm? Erasing all flash will erase the OS and Bootloader, meaning the device becomes useless and must be returned to manufacturer. This simply cant happen if system is deployed in the war zone. When such use-cases are discussed with the customer, better (aka easier to implement) solutions are usually found.

8. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 20118 Self Destruct! Erasing the Code and Data Unencrypted classified code and data must be erased before an enemy could get to it Erase Times: NOR Flash: Minutes, needs power NAND Flash: Tens of seconds, needs power DRAM: Under 5 seconds, does not need power Crowbar DRAM: Milliseconds, does not need power Simplest solution to meet requirements: put unencrypted classified code and data in crowbar DRAM only! Only store strongly encrypted data in flash Decrypt data into DRAM only Which encryption to use: a topic beyond the scope of this talk. This is one of the topics that is covered often by others

9. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 20119 Self Destruct! Crowbar DRAM Cheapest, fastest, and most reliable means of erasing Even if power is removed from system, method works reliably Simple Solution: DRAM hardware is modified so that a switch turns off Vcc to DRAM, and at the same time shorts Vcc to ground. In cases of false alarms does not render the device into a brick

10. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201110 Self Destruct Erasing Flash A software-only solution will not work. All it takes is removal of power to stop the erasing, at which time other techniques can be used, such as removing flash from PCB then reading it independently. Some secure flash drives and memories will latch internal controller Once erasing begins, it cannot be stopped. Removing power only pauses the erase. Any attempt to power the chip to read will only resume the erase operations. Caveats: These are expensive flash memory options! Understand the implications: what happens to device if feature is triggered?

11. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201111 Development Environment != Deployed System What do you need for software development? It is typical for hardware requirements to drive the selection, design, and fabrication of hardware From the first revision, hardware will incorporate all the required features needed in the end product, such as no user-accessible debug ports, no simple unsecure high-speed access, no keyboard or mouse, etc. How is software developed for these systems?

12. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201112 Development Environment != Deployed System Software Team: Be part of Hardware Development! The software team must be part of the requirement reviews and early design meetings. Assume that every required feature will have a bug and needs to be debugged and fixed. Ensure that common tools used to troubleshoot such errors are available, otherwise software costs will go through the roof, and high levels of reliability will be unachievable Coordinate with hardware designers a plan for scaling back development features as the product matures and gets closer to production

13. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201113 Development Environment != Deployed System Back Door Mechanisms Software engineers like to put in back doors to allow easy access to troubleshooting and debug utilities. Such features, however, are frowned upon in military and avionic systems, because they are prime entry points for hacking, security breaches, and often contain unreliable code. Presence of these mechanisms will usually cause a system to not meet certification requirements. But absense of the mechanisms can greatly increase software maintenance costs.

14. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201114 Development Environment != Deployed System Front Door Mechanisms Instead, consider front door mechanisms. Dont try to hide these entry points, where they tend to be unmanaged and not well secured. Rather, design them into the system from the start, to undergo the same review scrutiny as any other feature Share the planned usage and lockouts with the customer and end user. Often, they will want to use the features too during early development. Include the necessary features to disable them when necessary, and securely re-enable them when they are needed. Include a means to componentize the front door mechanisms, so that it can also be easily removed in final production versions.

15. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201115 Development Environment != Deployed System Debug Boards Work with hardware designers to bring all the desired interfaces that customer does not want to a high-density debug connector e.g. debug serial ports, USB ports, JTAG, spare GPIOs, LEDs, inter-chip communication signals, or anything else that would be useful to the software developer Create a separate debug board that expands the signals into standard connectors Hardware team might balk at cost of such a board, which is probably in the $5000 range However, existence of such a board can save 10 to 100 times that cost in reduced software development effort Standardize the high-density debug connector, thus making debug board reusable across multiple projects

16. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201116 Sorry, No Debug Allowed! Once unit is packaged, say bye to the debug board The debug board helps reduce initial development, and can be used for troubleshooting any issues that can be replicated on an open system Once the system is packaged, that key development interface is no longer available. Alternate mechanisms are needed to troubleshoot issues.

17. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201117 Sorry, No Debug Allowed! Log Files are the key! Continuously collect data on every aspect of the system Budget the amount of space needed to store log files early during system design Store data on flash, for retrieval either by opening unit, or through an existing customer communications link Do not store any classified data in the log files!

18. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201118 Sorry, No Debug Allowed! Not all log files are equal Quality of log files is of extreme importance! It can eliminate the reliance on customer descriptions of the problems. Often, their observations are inaccurate, or they provide their interpretation of the problem. With just the right information in a log file, many issues occurring in the field can be identified This is absolutely essential, especially in avionics Avoids the need to have to replicate the issue to debug it Carefully review what goes into the log file A log file contains poor information if every time a problem occurs, the log file only provides minimal clues to the actual problem

19. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201119 Sorry, No Debug Allowed! Concise Log Files Memory reserved for log files can fill up quick Optimize memory usage as part of the design. Be cryptic, thats fine as long as there is easy way to decipher the log file Average of velocity for past 30 seconds = 3.49231 uses 40 bytes of memory av30=3.49231 uses 12 bytes of memory Both the above provide equal amount of information, but second one is more concise, and log is searchable by unique acronym av30 Store in binary: 0xaa30 0x405F8202 Uses 6 bytes of memory 16-bit number as a key allows for 65536 separate types of log entries 32-bit floating point number is only 4 bytes of data Create scripts to quickly search or filter desired data If this is not easy, then the uniqueness of keys is not adequate

20. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201120 Sorry, No Debug Allowed! What information should go into Log Files? The more information stored, the more likely a single capture of a log file is sufficient to pinpoint a problem that occurred in field Timestamp EVERYTHING! Microsecond resolution is desirable. Millisecond resolution is essential. Relative time, stored as a 16-bit or 32-bit number is usually acceptable. Dont use more than 4 bytes per timestamp, or memory will fill up too quickly.

21. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201121 Sorry, No Debug Allowed! What information should go into Log Files? Periodically log the state of the system, including the following: Any code that is built as a finite state machine, include the mode or state The current value and setting of every GPIO pin Current value of every key register Current value of any key global variables Current value of any sensor reading Current value of any controller output

22. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201122 Sorry, No Debug Allowed! What information should go into Log Files? Summary of every communication message that is sent or received: Includes Ethernet, WiFi, USB, SPI, etc. Save key information, such as sender or receiver, size of message, command or message type. For short messages (i.e. a few bytes) include entire message. For longer messages, include first few data bytes.

23. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201123 Sorry, No Debug Allowed! What information should go into Log Files? Every Output I/O Device Driver Call: E.g. Instrument the stream interface, so that every Write operation is captured Save timestamp, device, port, and value written Dont need to log inputs or reads unless it is a message (see previous slide). For sensors, the state data logs the values periodically, instead of every time the device driver is called.

24. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201124 Sorry, No Debug Allowed! What information should go into Log Files? Every Interrupt Timestamp of entry into and exit from ISR IRQ number Running counter of how many interrupts of this type received If this is a shared IRQ, which device was interrupting

25. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201125 Sorry, No Debug Allowed! What information should go into Log Files? Every Context Switch Timestamp of entry into each thread Thread ID Timestamp of any entry into idle state or thread The line of code to log this event needs to be in the operating system scheduler, right before the context switch occurs

26. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201126 Sorry, No Debug Allowed! How can we guarantee all software logs what it needs? Log files are most effective when there are no gaps in information, resulting from one or more modules not logging like they should, or disabling the logs if they have the ability When possible, put code to log information into kernel code, such as drivers or operating system functions, where they cannot be touched or changed by application programmers. Logging is then enabled always for everything, with no ability to disable it.

27. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201127 Sorry, No Debug Allowed! Wow, isnt that too much to log! Estimate log file size based on typical usage. Here is an example State of system, 400 bytes, once per second Messages, 16 bytes per message, 500 messages per second I/O Driver Writes, 12 bytes per call, 200 times per second Interrupts, 12 bytes per interrupt, about 800 times per second Context Switches, 8 bytes per switch, 200 times per second Above scenario = 400+12*800+12*200+8*200+16*500 = 22 kBytes per second = 1.3 Mbytes per minute = 80 Mbytes per hour = 2 GBytes per day

28. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201128 Sorry, No Debug Allowed! Wow, isnt that too much to log! In past systems, this was a problem. In todays system, this is easy. E.g. add a 2 GB micro-SD card solely for logging, and the full log for any 24-hour period is possible. This is worst case might only need to log for 8-hour days, and only a fraction of the above, as system might only really be in use 10% of the time during the day and idle or in a sleep mode rest of the day, thus bringing actual log requirements down to under 100 Mbytes per day, or 20 days of storage with the same 2GB card. Use a circular buffer discard oldest data when maximum data usage has been reached. Store multiple small files instead of one big file. E.g. start a new file every 5 minutes. Makes it easier to retrieve only partial logs within specific time periods.

29. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201129 Sorry, No Debug Allowed! Cant afford the performance loss! With proper optimization of log code, the logged data should not use more than 2% of the processing power. E.g. Can write SD at 1MByte/sec, only need to write 22KByte/sec worst case, thus SD write is active less than 2% of the time. Use a DRAM FIFO most logging is done in kernel space, where file access is not available Write to DRAM FIFO, overhead is negligible. Once per second, have a high-priority system-level thread read FIFO and write to log file. Only systems that are 98% loaded or more wont be able to support this added overhead. If the system is that loaded, use a faster processor, or optimize one thread to reduce CPU usage by 2% Consider the cost of NOT having this data when a problem occurs in a field unit that needs to be fixed. The 2% overhead is well justified!

30. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201130 Error Detection, Why Bother? What is right amount of error detection and handling? What percentage of your code is error detection and handling? Do you think that is too little? Too much? Just right? How is more or less error detection and handling code justified?

31. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201131 Error Detection, Why Bother? Do I need to detect all errors? Dont waste time detecting errors for which the error handling is blue screen of death More often than not, this elevates simple errors into fatal errors Only detect errors when it is possible to create a handler that recovers gracefully from the error Otherwise, let the logging mechanism simply log the information as part of its normal duties Error handling code must be part of the design documents Adding ad-hoc error handling often causes more harm than good, because the code is rarely tested adequately, and only gets used when there are already problems in the system.

32. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201132 It Uses Too Much Power! Battery and Power Management Mobile military applications and avionic applications have one thing in common: Power consumption is ALWAYS too much, so reduce it more! Aircraft need low power to reduce energy needs, and often just as important to reduce heat production within the closed body Military applications need devices to run on battery for entire missions, as the ability to recharge is often at best a few hours each night

33. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201133 It Uses Too Much Power! Battery Life Capacity is specified in mAh mAh = milli-Amp-Hours E.g. 1400 mAh means the battery can drain 100mA for 14 continuous hours Or it can draw 1.4A for one hour By measuring current when battery is discharging, can compute power consumption Power = Battery_Voltage * Discharge_Current

34. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201134 It Uses Too Much Power! Battery Life Battery capacity only guaranteed for new batteries After a battery has been recharged a few dozen times, it starts to age As it ages, capacity decreases. The faster you discharge the batteries, typically the shorter its lifespan Check battery datasheet. Some batteries may have capacity reduced by 50% in just a few hundred recharge cycles. This means less than a year if recharged daily.

35. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201135 It Uses Too Much Power! Battery Level For Alkaline batteries, voltage is a reasonable gauge of remaining capacity For Lithium type batteries, voltage level of battery decreases relatively slowly thus voltage is not indicative of remaining life

36. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201136 It Uses Too Much Power! Battery Level Li-Ion and Li-Polymer batteries typically use a fuel gauge to measure battery level The gauge counts coulombs When charging, the count increases When discharging, the count decreases Battery level is only accurate when the capacity is calibrated Once a battery starts aging, a battery level algorithm may require the battery to be fully discharged, then reset the algorithm to indicate a new battery (even though it is old), then fully charge. This allows the new capacity to be re-learned. Some drivers auto-calibrate anytime a full charge occurs Actual algorithm is hardware dependent If battery level does not seem to be accurate, verify that drivers are properly reading the fuel gauge, and that there is some form of recalibration for aging batteries

37. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201137 It Uses Too Much Power! Reduce Power: Go For Idle Idle systems can offer best power savings Sometimes, running FASTER can reduce power, by getting back to idle quicker E.g. SPI driver with busy-wait loops. Run SPI clock as fast as possible, to get back to idle more quickly For processor, running faster usually means more power Slow down periodic threads Read an A/D 50 times per second instead of 100 times per second, if application allows it Eliminate unnecessary services and interrupts If system is not being used for 1 minute, capture every interrupt that occurs or every service that executes. Ask WHY they execute, if system is idle? Coming out of idle often is costly to battery life.

38. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201138 It Uses Too Much Power! Reduce Power: Go For Idle Idle Implementations Idle Thread Lowest-priority thread has an empty while(1) loop. Anything that becomes active preempts this thread. Hardware-independent, thus easily portable to any processor Power-hungry; processor is still running at full tilt CPU Idle Mode Most modern processors have an Idle Mode By installing hardware-dependent support to use this mode, significant power savings can be achieved when idle Deep Idle Turn off the processor oscillator when idle May take 100 usec extra to wakeup But the power savings can be significant Extra care needed if DMA running in background, e.g. for Display If turning off oscillator not an option because it breaks some devices, consider just slowing down clock when idle

39. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201139 It Uses Too Much Power Suspend and Shelf Modes Suspend (aka Sleep) Turn off processor and most peripherals Keep DRAM powered State of each I/O pin controlled by software Enables resuming within a second Shelf (aka Deep Sleep) Turn off processor and all peripherals Turn off DRAM State of each I/O pin is undefined Resuming requires rebooting Used to maintain enough power for a soft power-switch, and perhaps maintain an RTC Hibernate is a form of shelf mode, except that the contents of DRAM are first saved to the file system

40. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201140 It Uses Too Much Power Suspend and Shelf Modes Suspend Mode Power Software has significant impact on power consumption during this mode, even though CPU not running Devices not properly turned off or put into low power modes may continue to draw power I/O pins not placed in the right state may cause power drain Shelf Mode Power Software has no control of this. If this power consumption is too high, hardware engineers need to investigate

41. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201141 It Uses Too Much Power Suspend and Shelf Modes Manage entry into suspend mode Use the xxx_power_off() of the driver Start by saving the context In addition to saving context: if the device has a low-power mode, enter it; or if the device has a power-enable, disable it if turning off power, hardware engineers should verify there is no backfeed, as that will unnecessarily consume power during the sleep mode Ensure that any output signal to that device is placed into its inactive state Note that for active-low signals that typically have a pull-up on the line, this means set the output HI When exiting suspend mode Do opposite steps in xxx_power_on(), in reverse order Exit low-power mode or re-enable power End by restoring the context If device was powered off instead of a low-power mode It might be necessary to re-initialize the device, as any internal state would have been lost.

42. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201142 It Uses Too Much Power Device Low-Power Modes When a device is not needed, it should be placed into a low-power mode In theory, the xxx_power_off() function of the driver is what should be called to turn off that device However, that only gets called during a suspend operation For high-powered drivers, create an IOCTL to execute the commands of the power-off function under user control Create a second IOCTL to re-enable the power

43. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201143 It Uses Too Much Power Duty Cycling Duty cycling high-powered items can lower average power consumption A 50% Duty Cycle of a device can reduce consumption in half for that device This means ON half the time, then either OFF or in low- power mode the other half Look at each individual hardware item to determine if there is an opportunity. LEDs Analog Inputs Communications

44. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201144 Summary Is that REALLY the Requirement? Review requirements early! Unrealistic requirements can often be negotiated Self Destruct! Dont erase everything, that is overkill! Put unencrypted classified data into DRAM Development Environment != Deployed System Just because end requirements means no debug does not mean prototypes cant have it Create a debug board with favorite signals going to a high- density connector

45. (c) 2011 Dave Stewart - InHand Electronics - www.inhand.comEmbedded Systems Conference Boston, Sept 201145 Summary Sorry, no debug I/O allowed, but debug it quick! Continuously generate log files Design the content to manage memory needs Error detection: Why bother? Only detect errors that can be handled gracefully No blue screens of death! It Uses Too Much Power! Understand how battery power really works Leverage idle, sleep modes, and duty cycling