hp education services education.hp.com 85 System Crash Dump Version C.00 H4264S Module 9 Slides

© 2001 Hewlett-Packard Company H4264S C System Panic An HP-UX system will panic when it encounters a situation in which it does not know how to respond. In these situations, the system halts (it panics). There are three types of system panics: - HPMC - ToC - PANIC

© 2001 Hewlett-Packard Company H4264S C High Priority Machine Checks Hardware initiated failures Usually indicate a hardware problem BUT Can also be caused by software.

© 2001 Hewlett-Packard Company H4264S C Transfer of Control Can be initiated by the user, usually because the system has hung Can be caused by MC/ServiceGuard, while attempting to ensure data integrity

© 2001 Hewlett-Packard Company H4264S C Panics Panics are software failures in kernel mode. The HP-UX kernel has detected an internal problem. The system is “stopping,” because continuing to process could cause more problems. Stopping is safer.

© 2001 Hewlett-Packard Company H4264S C System Crash Dump /var/adm/crash crash.0 image.0.1INDEX crash.1 image.0.2vmunix.0 A crash dump is a core image stored to the dump area upon system panic. The dump area is usually primary swap, but it is configurable. The /sbin/savecrash command causes the dump area contents to be saved to dump files. It must be executable. A crash dump consists of the following files in the following directories:

© 2001 Hewlett-Packard Company H4264S C Big Picture Memory CPU /var OS Tables Kernel Dump/Swap Disks Dump/Swap #2 1 2 /var/adm/crash/crash.0 image.0.1INDEXimage.0.2vmunix.0

© 2001 Hewlett-Packard Company H4264S C Memory Classes Memory BC Text Data Text Data Unused Pages Text Data Kernel text Kernel data (static and dynamic tables) F/S Data As of HP-UX 11.00, physical memory is divided into 8 classes: kcode ksdata kddata fsdata bcache userpg ustack unused Use the crashconf -v command to view the configured memory classes.

© 2001 Hewlett-Packard Company H4264S C Configuring Memory Classes and Dump Devices # crashconf -v CLASS PAGES INCLUDED IN DUMP DESCRIPTION UNUSED no, by default unused pages USERPG 7911 no, by default user process pages BCACHE no, by default buffer cache pages KCODE 1507 no, by default kernel code pages USTACK 191 yes, by default user process stacks FSDATA 89 yes, by default file system metadata KDDATA 4093 yes, by default kernel dynamic data KSDATA 4179 yes, by default kernel static data Total pages on system: Total pages included in dump: 8552 DEVICE OFFSET(kB) SIZE (kB) LOGICAL VOL. NAME :0x :0x /dev/vg00/lvol The crashconf command allows configuration of dynamic memory classes and dump devices while the system is up and running: # crashconf -i userpg # crashconf -e fsdata # crashconf /dev/vg01/dumplvol # crashconf -v

© 2001 Hewlett-Packard Company H4264S C How Much Dump Space Is Needed? Buffer Cache Text Data Text Data Free Pages Text data Kernel text Kernel data (OS tables) Memory (2 GB) Selective items are dumped! Disk Swap (1.5 GB)

© 2001 Hewlett-Packard Company H4264S C Dump Configuration Startup Files Pool of startup/shutdown scripts Data Configuration Files /etc/rc.config.d /sbin/init.d rc script crashconf savecrash crashconf savecrash /sbin/rc1.d S080crashconf S440savecore start

© 2001 Hewlett-Packard Company H4264S C Obtaining the Crash Dump Message /var/adm/crash/crash.0 image.0.1INDEXvmunix.0 corefile Procedure for Extracting the Dump Message from the Crash Files /tmp kernfile # crashutil -v COREFILE /var/adm/crash/crash.0 \ /tmp/corefile /tmp/kernfile # cd /tmp # adb -k kernfile corefile msgbuf+8/s... Panic: Data page fault... $q # Note: Enter $ q to quit/terminate

© 2001 Hewlett-Packard Company H4264S C The Interactive Editor — ied Provides command rerun, recall, search, and editing in any interactive utility Uses the EDITOR shell variable Creates utility command history in -h filename #export EDITOR=vi #ied -h ~root/.q4_history interactive utility or command

© 2001 Hewlett-Packard Company H4264S C Another Debugger — q4 Displays kernel, process, and thread symbols, structures, and unions Uses Perl scripts Includes commands in /usr/contrib/bin Includes documentation in /usr/contrib/do c Installs from compressed tarball in /usr/contrib/lib –cd /usr/contrib/lib –uncompress Q4Lib.tar.Z –tar xvf Q4Lib.tar Prepare the kernel for debugging with q4: –q4pxdb /stand/vmunix # Live Kernel –q4pxdb /var/adm/crash/crash.0/vmunix # Crash dump Kernel

© 2001 Hewlett-Packard Company H4264S C q4 Documentation, Commands, and Syntax q4 help lists, commands, topics, and syntax q4> help q4> help topic q4> help examine Commands to retrieve basic system information and status q4> load struct utsname from &utsname; print -t System versions q4> examine panicstr using s Panic string q4> examine &msgbuf+8 using s Console message buffer q4> examine &time using Y System time at the crash q4> examine &boot_string using s Bootstring (device path and kernel name) q4> examine &nproc using D Maximum # of processes (full word decimal) q4> examine &maxfiles_lim using D Maximum # of file descriptors/proc q4> examine &maxdsiz using D Maximum amount of private data q4> examine &maxvgs using D Maximum # of volume groups exit or quit or bye to exit the debugger