1. IBM India Software Labs FFDC Using Crash Dumps – BoF07/09/04 © 2004 IBM Corporation First Failure Data Capture Using Crash Dumps: A BoF session Linux Symposium 2004 Ottawa, Canada

2. IBM India Software Labs © 2004 IBM Corporation 1FFDC Using Crash Dumps – BoF 22 July 2004 Topics for Discussion ✔ Existing Crash Dump Tools ✔ Approaches taken ✔ Problems ✔ Other FFDC Tools ✔ Kexec based Dumping ✔ Demo ✔ Dump Analyzing Tools

3. IBM India Software Labs © 2004 IBM Corporation 1FFDC Using Crash Dumps – BoF 22 July 2004 Current Dumping Solutions ✔ LKCD, Netdump ✔ None have made it to mainline ✔ Known limitations in each ✔ Invasiveness into kernel a major bottleneck ✔ Diskdump – Newer alternative, driver specific code needed ✔ Firmware based solutions ✔ S390 dumping solution ✔ Has worked in other Oses ✔ Solution available largely distro specific ✔ SLES9 has picked up LKCD ✔ RedHat prefers netdump/diskdump

4. IBM India Software Labs © 2004 IBM Corporation 1FFDC Using Crash Dumps – BoF 22 July 2004 Other FFDC tools ✔ Printk ✔ Still the most preferred debugging mechanism ✔ Is it powerful enough ✔ Oops ✔ Will not suffice for problems such as data corruption, hangs, etc ✔ Limited data for PD/PSI ✔ Kmsgdump ✔ Effective method to capture panic information

5. IBM India Software Labs © 2004 IBM Corporation 1FFDC Using Crash Dumps – BoF 22 July 2004 Kexec based crash dumping ✔ Reboots with “less” memory ✔ Rest of memory treated as a device ✔ No dump device required ✔ Dump exported as an ELF format file ✔ Greatly simplifies dump write-out ✔ Uses “standard” core dump format ✔ Standard dump analyzers can be used ✔ Multiple “views” will be provided ✔ Full-view, Kernel-pages only, etc

6. IBM India Software Labs © 2004 IBM Corporation 1FFDC Using Crash Dumps – BoF 22 July 2004 Dump Analyzing Tools ✔ Numerous tools exist ✔ lcrash, crash, gdb ✔ Issues to ponder ✔ Is there a need for a standard “dump format” ✔ Is compatibility between tools a necessity ✔ What's the best way to keep up with kernel changes ✔ What are the deficiencies in the current tools? ✔ Commands and features needed

7. IBM India Software Labs © 2004 IBM Corporation 1FFDC Using Crash Dumps – BoF 22 July 2004 Discussions please...

8. IBM India Software Labs © 2004 IBM Corporation 1FFDC Using Crash Dumps – BoF 22 July 2004 Disclaimers and Trademarks ✔ This work represents the view of the author and does not necessarily represent the view of IBM. ✔ IBM and IBM(logo) are trademarks of International Business Machines Corporation in the United States, other countries, or both. ✔ Linux is a registered trademark of Linus Torvalds in the U.S., other countries, or both. ✔ Red Hat is a trademark of Red Hat, Inc. in the U.S., other countries, or both. ✔ Other company, product, or service names may be trademarks or service marks of others.