1. © 2010 InHand Electronics, Inc. – www.inhand.com Troubleshooting Linux Performance and Battery Consumption Issues Dave Stewart Director of Software Engineering InHand Electronics dstewart@inhand.com www.inhand.com Embedded Systems Conference Chicago 2010 Class ESC 361

2. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 2 Contents Overview Go for Idle Performance Debugging Sleep and Shelf Modes Device Low Power Modes Dynamic Frequency and Voltage Scaling (DFVS) Duty Cycling

3. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 3 Overview Typical Characteristics of Embedded Hardware Environments that support Linux: 32-bit Environment, E.g. ARM (e.g. OMAP, PXA320) x86 (e.g. Intel Atom) PowerPC Numerous desktop-like devices E.g. USB, Ethernet, WiFi, Bluetooth Human-Machine Interfaces E.g. Display, Touchscreen, Keyboard, Keypad, Mouse Possibly HMI is via network, e.g. Web Page Some custom devices E.g. camera, display, sensors, actuators, wireless radio, GPS

4. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 4 General Concepts Performance vs. Battery Consumption Increased performance == more power Longer battery life == performance compromise One Exception: Optimize code == Less power AND better performance Only Power what you Need Most hardware have various modes that enable turning off or going into low power mode Most drivers, by default, dont make use of these modes

5. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 5 Battery Life Capacity is specified in mAh mAh = milli-Amp-Hours E.g. 1400 mAh means the battery can drain 100mA for 14 continuous hours Or it can draw 1.4A for one hour By measuring current when battery is discharging, can compute power consumption Power = Battery_Voltage * Discharge_Current

6. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 6 Battery Life Battery capacity only guaranteed for new batteries After a battery has been recharged a few dozen times, it starts to age As it ages, capacity decreases. The faster you discharge the batteries, typically the shorter its lifespan Check battery datasheet. Some batteries may have capacity reduced by 50% in just a few hundred recharge cycles. –This means less than a year if recharged daily.

7. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 7 Battery Level For Alkaline batteries, voltage is a reasonable gauge of remaining capacity For Lithium type batteries, voltage level of battery decreases relatively slowly thus voltage is not indicative of remaining life

8. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 8 Battery Level Li-Ion and Li-Polymer batteries typically use a fuel gauge to measure battery level The gauge counts coulombs –When charging, the count increases –When discharging, the count decreases Battery level is only accurate when the capacity is calibrated –Once a battery starts aging, a battery level algorithm may require the battery to be fully discharged, then reset the algorithm to indicate a new battery (even though it is old), then fully charge. This allows the new capacity to be re-learned. –Some drivers auto-calibrate anytime a full charge occurs Actual algorithm is hardware dependent If battery level does not seem to be accurate, verify that drivers are properly reading the fuel gauge, and that there is some form of recalibration for aging batteries

9. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 9 Reduce Power: Go For Idle Idle systems can offer best power savings Sometimes, running FASTER can reduce power, by getting back to idle quicker E.g. SPI driver with busy-wait loops. Run SPI clock as fast as possible, to get back to idle more quickly For processor, running faster usually means more power Slow down periodic threads Read an A/D 50 times per second instead of 100 times per second, if application allows it Eliminate unnecessary services and interrupts If system is not being used for 1 minute, capture every interrupt that occurs or every service that executes. Ask WHY they execute, if system is idle? Coming out of idle often is costly to battery life.

10. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 10 Go For Idle Idle Implementations Idle Thread Lowest-priority thread has an empty while(1) loop. Anything that becomes active preempts this thread. Hardware-independent, thus easily portable to any processor Power-hungry; processor is still running at full tilt CPU Idle Mode Most modern processors have an Idle Mode By installing hardware-dependent support to use this mode, significant power savings can be achieved when idle Deep Idle Turn off the processor oscillator when idle May take 100 usec extra to wakeup But the power savings can be significant Extra care needed if DMA running in background, e.g. for Display If turning off oscillator not an option because it breaks some devices, consider just slowing down clock when idle

11. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 11 Go For Idle Look at arch/$ARCH/kernel cpu_idle() is starting point grep cpu_idle *.c to find which file has the function –Depending on architecture, several possibilities default_idle() is used if you dont override it arch_idle() may already utilize the CPU instruction, check what it does pm_idle() is your hook To create your own idle, define an idle function, and attach it to the pm_idle() hook Use pm_idle() to adjust clock speed or oscillator upon enterring and exiting idle Refer to idle section of your processor architecture for power- saving options that are available This code might require a call to assembly language functions

12. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 12 Performance Debugging Identify threads that use too much execution time On many systems, one or two threads end up using the bulk of execution time Optimize these one or two threads, and quickly see power and performance improvements How to identify the culprits? If really obvious, ps command will show it Many times not so obvious, use logic analyzer Enables identifying issues with both threads and interrupts

13. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 13 Performance Debugging Scatter logic analyzer debug statements throughout code Start and stop of each function and thread is a good starting point. Often, just keep all the prior debug statements in place, they may provide the data needed. Run the code, and monitor on the logic analyzer. Look for patterns of behavior. Identify patterns that use too much CPU. Verify what is happening when analyzer shows periods of inactivity. Monitor execution time of interrupts. Some examples on next few slides. PDEBUG_HEX8(x) sends an 8-bit code x to the logic analyzer via pre-defined digital outputs. See ESC-203, Troubleshooting Real-Time Software Issues using a Logic Analyzer for specific details

14. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 14 Performance Debugging Green is commmunication. White/Yellow is PDEBUG_HEX8(). We can clearly see communication stopped for a period of time. Why? Is that normal? From ESC-203 – Troubleshooting Real-Time Software Issues using a Logic Analyzer

15. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 15 Performance Debugging See the Pattern? Four short communication packets, one long one, then none for what seems like two or three system cycles. From ESC-203 – Troubleshooting Real-Time Software Issues using a Logic Analyzer

16. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 16 Performance Debugging What line of code executes immediately after PDEBUG_HEX8(0x9F)? Whatever it is, it takes a LONG time. If there is a performance issue, focus on that code first. From ESC-203 – Troubleshooting Real-Time Software Issues using a Logic Analyzer

17. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 17 Performance Debugging This is an example of code executed with PDEBUG_HEX8() What is happening right after the code 0x33 is sent? From ESC-203 – Troubleshooting Real-Time Software Issues using a Logic Analyzer

18. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 18 Performance Debugging How do we capture EVERYTHING in Linux? Go straight for the context switch and ISR code For context switch, look at kernel/sched.c context switch code is in function schedule() After call to sched_info_switch(), send process ID of new task to logic analyzer Often it is fine to just send lower 8 bits of ID For ISR, look at kernel/irq/handle.c At beginning and end of handle_IRQ_event(), send the IRQ number to logic analyzer Identify which threads or ISRs are using the most execution time Focus on the longest ones, and determine if the amount of execution time is reasonable If not, optimize it!

19. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 19 Suspend and Shelf Modes Suspend (aka Sleep) Turn off processor and most peripherals Keep DRAM powered State of each I/O pin controlled by software Enables resuming within a second Shelf (aka Deep Sleep) Turn off processor and all peripherals Turn off DRAM State of each I/O pin is undefined Resuming requires rebooting Used to maintain enough power for a soft power-switch, and perhaps maintain an RTC Hibernate is a form of shelf mode, except that the contents of DRAM are first saved to the file system

20. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 20 Suspend and Shelf Modes Suspend Mode Power Software has significant impact on power consumption during this mode, even though CPU not running Devices not properly turned off or put into low power modes may continue to draw power I/O pins not placed in the right state may cause power drain Shelf Mode Power Software has no control of this. If this power consumption is too high, hardware engineers need to investigate

21. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 21 Suspend and Shelf Modes Manage entry into suspend mode Use the xxx_power_off() of the driver Start by saving the context In addition to saving context: if the device has a low-power mode, enter it; or if the device has a power-enable, disable it if turning off power, hardware engineers should verify there is no backfeed, as that will unnecessarily consume power during the sleep mode Ensure that any output signal to that device is placed into its inactive state Note that for active-low signals that typically have a pull-up on the line, this means set the output HI When exiting suspend mode Do opposite steps in xxx_power_on(), in reverse order Exit low-power mode or re-enable power End by restoring the context If device was powered off instead of a low-power mode It might be necessary to re-initialize the device, as any internal state would have been lost.

22. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 22 Device Low-Power Modes When a device is not needed, it should be placed into a low-power mode In theory, the xxx_power_off() function of the driver is what should be called to turn off that device However, that only gets called during a suspend operation For high-powered drivers, create an IOCTL to execute the commands of the power-off function under user control Create a second IOCTL to re-enable the power

23. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 23 Dynamic Frequency and Voltage Scaling (DFVS) Higher frequency requires higher voltage The two need to be adjusted in sync, in order to achieve best power savings If CPU is not fully loaded, consider running at a lower frequency E.g. PXA270, CPU Core Only Running at 200MHz = 279mW Running at 500MHz = 432mW This is a 35% savings In this case, running slower is better

24. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 24 Dynamic Frequency and Voltage Scaling (DFVS) Affect on performance Sometimes no effect For example, if memory bus is 200MHz, but processor is 500MHz, a memory-intensive activity might not run any slower if processor slows down to 200MHz In this case, utilization is same regardless of processor speed, so go for lower power Sometimes it does affect it A CPU-bound job based completely out of cache may run 100% of the time at 200MHz, but only 40% at 500MHz. In this case, when running at 500MHz, task is in idle for 60% of the time, thus power savings of idle mode can be leveraged. However, if execution used 100% at 500MHz, performance would be reduced if CPU clock is lowered to 200MHz Check your processor manual or linux distribution for possible support for DFVS

25. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 25 DFVS Example Logic Analyzer Screen Captures Follow Next few pages show a logic analyzer setup of a DFVS system Key signals to look at: PDEBUG_HEX8(): 8-bit debug port (yellow) Indicates where in the code we are executing See ESC-203 class for more details on this macro DFVS 1-bit trigger (magenta) 3-bit Frequency (green) 1=104MHz, 2=208MHz, … 5=520MHz Instantaneous current consumption (green analog) Reading is inverted, scale is 0mA to 400mA, lower reading means higher current draw. Battery voltage (red). Each page is a zoom of the prior page. The zoom area is shown by the magenta rectangle.

26. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 26 Setup Extras Example zoom

27. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 27 Setup Extras Example zoom

28. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 28 Setup Extras Example zoom

29. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 29 Setup Extras Example

30. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 30 Duty Cycling Duty cycling high-powered items can lower average power consumption A 50% Duty Cycle of a device can reduce consumption in half for that device This means ON half the time, then either OFF or in low-power mode the other half Look at each individual hardware item to determine if there is an opportunity. Some are not obvious. A few examples given on following slides

31. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 31 Duty Cycling LEDs LEDs can consume 0.1W each if on all the time A 50% duty cycle at a fast enough rate will still light them albeit more dimly, but consume one-fourth the power. Essential LEDs, like the power indicator, can be more dimly lit at night or indoors than if used outdoors Consider using a heartbeat LED instead Turn LED ON for 200msec every 5 seconds This consumes only 4% of the power as compared to leaving that LED ON all the time Reduce that to 100msec every 10 seconds, then it consumes only 1% of the power as compared to always on

32. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 32 Duty Cycling Networking and Serial The receiver of most radios and serial or network links often consume more power than the transmitter This is because the transmitter knows when it transmits. It is easy to turn on transmitter, send data, then turn it off. It is more difficult for the receiver, because it has to be listening all the time just in case a message comes in If wake-on-activity features exist in the hardware, use them! Drivers may support the feature, but high-level protocols dont utilize it by default If a device has the feature, but driver doesnt, then modify the driver

33. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 33 Duty Cycling Networking and Serial (continued) Consider the transmitting sides retry mechanism If the system could be idle for extended periods of time, then suddenly receive data, and the transmitter retries if it doesnt succeed, then it may be possible to turn off the receiver. Suppose the transmitter has a 1-second timeout, and will retry 4 times before giving up The receiver can wakeup once every 4 seconds, stay on for 1 second, then turn off. It may not get first message, but it will get it at least once. If transmitter is not on battery power, then consider increasing the retries E.g. transmit every 0.25 seconds for 10 seconds until an Ack. In this case, receiver only needs to wakeup for 0.5 seconds every 5 seconds, and still get redundancy in case of transmission errors.

34. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 34 Duty Cycling Analog Inputs Review every analog input, and determine how often the data is really needed Many applications throw away unnecessary samples If data is being sampled at 100Hz, but the application using it is only running once a second, then most the data might be unnecessary Some data changes very slow. E.g. Battery Voltage. Sampling it once every 20 seconds instead of every second is likely sufficient. Bump up rate to every second when charging and on AC power Reducing sampling rate can enable CPU going into idle more often As discussed earlier, this is a win Many A/D devices consume less power when they are idle than when they are scanning Check datasheets; in some cases this may be automatic; in other cases the device might need to be placed into low-power mode explicitly

35. Dave Stewart, ESC Chicago – June 2010 Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 35 Summary Find ways to make the system idle more often, for the greatest savings Use a Logic Analyzer for Performance Debugging Leverage Sleep and Shelf Modes for extended battery life; ensure proper device and I/O settings during these modes Use individual device low power modes when a device is not being used Enable Dynamic Frequency and Voltage Scaling (DFVS) Look for opportunities to use duty cycling to reduce power