1. April 2-5 Orlando, Florida Database Disaster Recovery and More Presented by: Stephen Rea University of Arkansas Cooperative Extension Service April 3, 2006 Evaluation Code 082

2. 2 Session Rules of Etiquette  Please turn off your cell phone/pager  If you must leave the session early, please do so as discreetly as possible  Please avoid side conversation during the session Thank you for your cooperation!

3. Evaluation Code 082 3 Introduction “Backups? We don’ NEED no steenkin’ backups!” - Famous last words of an ex-DBA  Stephen Rea - Oracle ® Database Administrator  University of Arkansas Cooperative Extension Service

4. Evaluation Code 082 4 Introduction  This session will show you step-by-step instructions for:  Jaunt: Bulletproofing your database against data loss  Jaunt: Creating database backups (cold and hot backups)  Recovering from various disaster scenarios  Bonus: Implementing a standby database (if time permits) Examples shown are for Oracle 9i (and 8i) and AIX UNIX 5.2 (not using RMAN here; originally developed under Oracle 7.3.4 and AIX UNIX 4.1.5). Oracle Backup and Recovery Classes (instructor led training):  Enterprise DBA Part 1B: Backup and Recovery (8i)  Oracle9i Database Administration Fundamentals II  4-5 days  $2,000-$2,500 registration  $???? transportation, hotel, meals, etc.

5. Evaluation Code 082 5 Benefits of Attending  After this session you will be able to:  Minimize the potential for loss of your data.  Recognize Oracle problems that you may encounter.  Know step-by-step how to recover from those problems.  Gain confidence in your disaster recovery ability.  Increase your worth as a DBA.  And, you’ve saved a bundle of money ($2,000+) as well as several days of your valuable time.

6. Evaluation Code 082 6 Topics of Discussion  Jaunt: Bulletproofing against data loss  Jaunt: Creating backups (cold and hot)  Recovering from disaster scenarios  Bonus: Implementing a standby database

7. Evaluation Code 082 7 Bulletproofing  To lessen the possibility of data loss, you can:  Enable archivelog mode (to reapply the changes during recovery; required by most disaster recovery scenarios)  Separate the archive logs from the redo logs (allocate to separate drives; likewise for the following items)  Separate the redo logs and archive logs from the datafiles  Multiplex (mirror) the redo log groups and members  Multiplex (mirror) the control file  Multiplex (mirror) the archive log files Oracle 9i Database Administrator's Guide (chapters 6 - 8, 12)

8. Evaluation Code 082 8 The Basics - Shutting Down Your Database Position to the correct database (setting the Oracle SID): For example, in UNIX:Or, in NT: $. oraenv c:\> set ORACLE_SID=PROD PROD c:\> set ORACLE_HOME=d:\oracle\v9204 Connect as sysdba (from a dba group user login, such as oracle): Oracle 8.1.5 and above:Prior to Oracle 9i, use Server Manager: $ sqlplus “/ as sysdba” $ svrmgrl ( SQL> connect / as sysdba) SVRMGR> connect internal Shut down the database (connected as sysdba): SQL> shutdown immediate (kills sessions, rolls back pending changes) If “shutdown immediate” fails or hangs (do this in another session): SQL> shutdown abort SQL> startup SQL> shutdown immediate

9. Evaluation Code 082 9 The Basics - Starting Up Your Database in Various Stages Database Startup Sequence (from shutdown state; connected as sysdba in sqlplus):  nomount – reads init.ora (or spfile), allocates SGA memory, starts background processes (such as pmon - process monitor) SQL> startup nomount (from shutdown state)  mount – opens control files SQL> startup mount (from shutdown state, or, …) SQL> alter database mount; (from nomount state)  open – opens datafiles and online redo logs, performs crash recovery SQL> startup [open] (from shutdown state, or, …) SQL> alter database open; (from nomount or mount state) Note: The pfile=init.ora option on the startup command can be used to specify the database’s init.ora file pathname.

10. Evaluation Code 082 10 Enabling Archiving (to recover database changes) Steps: Update init.ora, create archive log directory, shutdown, mount, set archivelog mode, open: Edit the init.ora file to add the archive log parameters: $ vi $ORACLE_HOME/dbs/initPROD.ora log_archive_dest = /u01/oradata/PROD/archivelogs/ log_archive_format = arch_PROD_%S.arc log_archive_start = true $ mkdir /u01/oradata/PROD/archivelogs From the mount state after shutdown, start up archiving and open the database: SQL> alter database archivelog; SQL> alter database open; SQL> archive log list (or: select * from v$database;)

11. Evaluation Code 082 11 Moving Datafiles (with shutdown) Steps: Shutdown, move files, mount, rename files, open: After shutdown, use an O/S command to move the datafile (done from within the unix sqlplus session here; “!” is the host command for unix; “$” is the host command for NT): SQL> !mv /u03/oradata/PROD/devl_PROD_01.dbf /u04/oradata/PROD From the mount state, rename the datafile that you moved: SQL> alter database rename file '/u03/oradata/PROD/devl_PROD_01.dbf' to '/u04/oradata/PROD/devl_PROD_01.dbf'; Then, open the database and look at the change made: SQL> alter database open; SQL> select * from v$datafile;

12. Evaluation Code 082 12 Moving Datafiles (without shutdown) Steps: Offline tablespace, move files, rename files, online tablespace: $ sqlplus "/ as sysdba" SQL> alter tablespace development offline; SQL> !mv /u03/oradata/PROD/devl_PROD_01.dbf /u04/oradata/PROD SQL> alter database rename file '/u03/oradata/PROD/devl_PROD_01.dbf' to '/u04/oradata/PROD/devl_PROD_01.dbf'; SQL> alter tablespace development online; SQL> select * from v$datafile;

13. Evaluation Code 082 13 Moving Datafiles (using control file) Steps: Create textual control file, shutdown, move files, update datafile pathnames in control file, run control file: $ sqlplus "/ as sysdba" SQL> alter database backup controlfile to trace; SQL> show parameter user_dump SQL> shutdown immediate SQL> exit $ mv /u03/oradata/PROD/*PROD*.dbf /u04/oradata/PROD (continued …)

14. Evaluation Code 082 14 Moving Datafiles (using control file) $ cd /u00/oracle/admin/PROD/udump $ ls -ltr *.trc $ vi prod_ora_16060.trc (or similar name, like ora_16060.trc) Delete the lines before the STARTUP NOMOUNT line Rename the datafiles (:g/DATAFILE/,/;/s/u03/u04/) # RECOVER (comment out the RECOVER line) Execute the edited control file, which starts up the database: $ sqlplus "/ as sysdba" SQL> @prod_ora_16060.trc SQL> select * from v$datafile;

15. Evaluation Code 082 15 Adding Redo Log Members (mirror across drives) Steps: Add member file to given group: SQL> alter database add logfile member '/u04/oradata/PROD/log_PROD_1C.rdo' to group 1; SQL> select * from v$logfile;

16. Evaluation Code 082 16 Increasing Maximum Number of Members (to add) Steps: Create textual control file, update maxlogmembers, shutdown, run control file: $ sqlplus "/ as sysdba" SQL> alter database backup controlfile to trace; SQL> !ls -ltr /u00/oracle/admin/PROD/udump SQL> !vi /u00/oracle/admin/PROD/udump/prod_ora_16060.trc Delete the lines before the STARTUP NOMOUNT line Edit maxlogmembers value (such as changing from 2 to 3) # RECOVER (comment out the RECOVER line) SQL> shutdown immediate SQL> @/u00/oracle/admin/PROD/udump/prod_ora_16060.trc

17. Evaluation Code 082 17 Adding Redo Log Groups (to limit log switch waits) Steps: Add group with member files of given size: SQL> alter database add logfile group 4 ('/u00/oradata/PROD/log_PROD_4A.rdo', '/u01/oradata/PROD/log_PROD_4B.rdo') size 500K; SQL> select * from v$logfile;

18. Evaluation Code 082 18 Multiplexing Control Files (to mirror across drives) Steps: Shutdown, make control file copies, update init.ora, startup: $ sqlplus "/ as sysdba" SQL> shutdown immediate SQL> !cp -p /u03/oradata/PROD/ctrl_PROD_01.ctl /u01/oradata/PROD/ctrl_PROD_02.ctl SQL> !vi $ORACLE_HOME/dbs/initPROD.ora control_files = (/u03/oradata/PROD/ctrl_PROD_01.ctl, /u01/oradata/PROD/ctrl_PROD_02.ctl) SQL> startup SQL> select * from v$controlfile;

19. Evaluation Code 082 19 Topics of Discussion  Jaunt: Bulletproofing against data loss  Jaunt: Creating backups (cold and hot)  Recovering from disaster scenarios  Bonus: Implementing a standby database

20. Evaluation Code 082 20 What To Back Up  Files to back up during one backup cycle:  Datafiles (for all tablespaces)  Control Files (binary and textual versions)  Redo Log Files (cold backups only, not hot backups)  Archive Log Files (archived redo logs, if archivelog mode is enabled)  Parameter File (init.ora (and/or spfile); init.ora is not in the database; like $ORACLE_HOME/dbs/initPROD.ora)  Password File (if used; it is not in the database; named like $ORACLE_HOME/dbs/orapwdPROD) Oracle9i User-Managed Backup and Recovery Guide (chapter 2)

21. Evaluation Code 082 21 What To Back Up (DB-based SQL) Getting names of datafiles, temp files, control files, and redo log files: SQL> select name from v$datafile; SQL> select name from v$tempfile; SQL> select name from v$controlfile; SQL> select member from v$logfile; Getting names of datafile and temp file tablespaces: SQL> select tablespace_name,file_name from dba_data_files order by tablespace_name; SQL> select tablespace_name,file_name from dba_temp_files;

22. Evaluation Code 082 22 What To Back Up (DB-based SQL) Getting locations of archive logs and archive parameters: SQL> select name,value from v$parameter where name in ('log_archive_dest', 'log_archive_format','log_archive_start'); SQL> show parameter archive SQL> archive log list For just the most recent archive log file names: SQL> select name from v$archived_log where trunc(completion_time) >= trunc(sysdate)-5;

23. Evaluation Code 082 23 What To Back Up (SID-based - preferred) Getting files using SID-based standard naming convention (preferred): $ find / -name '*PROD*' ! -type d 2>/dev/null >backemup.dat  Scripts for getting the list of backup files:  backup_list.shl (SID-based; best for script)  backup_list.sql (DB-based - but, don’t use this in an actual backup script for cold backups)

24. Evaluation Code 082 24 Cold Backups (with database down)  Cold Backup Requirements:  Database is shut down (shutdown immediate)  Complete backup from same point in time  Database not available for queries or updates  Either archivelog mode or noarchivelog mode  All associated files are backed up: Datafiles Control Files Redo Log Files Archive Log Files Parameter File Password File

25. Evaluation Code 082 25 Cold Backups - Getting the files to back up Create the textual control file (then, shutdown): $ sqlplus "/ as sysdba" SQL> alter database backup controlfile to trace; SQL> shutdown immediate SQL> exit $ ls -ltr /u00/oracle/admin/PROD/udump/* | tail -1 | sed 's/^.* \//\//' >>backemup.dat Get the SID-based file list with archivelogs at end of list: $ find / -name '*PROD*' ! -type d 2>/dev/null | grep -v 'arc$' | grep -v 'gz$' >>backemup.dat $ ls /u01/oradata/PROD/archivelogs/*.arc.gz >>backemup.dat $ ls /u01/oradata/PROD/archivelogs/*.arc >>backemup.dat

26. Evaluation Code 082 26 Cold Backups - File Backup Options  Copy the files to a staging (backup) directory: $ cat backemup.dat | sed 's/\(^.*\)\/\(.*\)$/ cp -p \1\/\2 \/u03\/backups\/prod1\/\2/' | sh  Or, compress (zip) the files to a staging directory: $ cat backemup.dat | sed 's/\(^.*\)\/\(.*\)$/ gzip -cv1 \1\/\2 \/u03\/backups\/prod1\/\2.gz; touch -r \1\/\2 \/u03\/backups\/prod1\/\2.gz/' | sh  Then, start the database and back up the staging directory: SQL> startup $ ls /u03/backups/prod1/* | cpio -ovC64 >/dev/rmt0  Or just copy the files directly to tape (no staging directory): $ cat backemup.dat | cpio -ovC64 >/dev/rmt0

27. Evaluation Code 082 27 Hot Backups (with database up)  Hot Backup Requirements:  Database is up (tablespace begin/end backup)  Backup of datafiles per tablespace over time  Database is available for queries and updates  Archivelog mode required  Special backup processing required for files: Datafiles (by ts) BACKUP Control Files NO Redo Logs Archive Logs (switch) Parameter File Password File

28. Evaluation Code 082 28 Hot Backups of Tablespaces Steps: For each tablespace: begin tablespace backup, copy (or zip) datafiles to a staging (backup) directory, end tablespace backup: SQL> alter tablespace development begin backup; SQL> !cp -p /u03/oradata/PROD/devl_PROD_*.dbf /u03/backups/prod1 SQL> alter tablespace development end backup; Note that no shutdown or startup is needed

29. Evaluation Code 082 29 Hot Backups of Log Files Steps: Switch logfile, get list of archivelogs, copy log files: SQL> alter system switch logfile; $ ls /u01/oradata/PROD/archivelogs/*.arc.gz >logfile.lst $ ls /u01/oradata/PROD/archivelogs/*.arc >>logfile.lst $ sleep 5 (for script) $ cat logfile.lst | sed "s/\(.*\/\)\([^\/].*\)/ cp -p \1\2 \/u03\/backups\/prod1\/\2/" >logfile.shl $ sh logfile.shl

30. Evaluation Code 082 30 Hot Backups of Control Files Steps: Create binary and textual control files, find and copy textual control (back up the control file last): SQL> alter database backup controlfile to '/u03/backups/prod1/controlfile.ctl'; SQL> alter database backup controlfile to trace; $ ls -ltr /u00/oracle/admin/PROD/udump $ cp -p /u00/oracle/admin/PROD/udump/prod_ora_16060.trc /u03/backups/prod1

31. Evaluation Code 082 31 Backing Up the Staging (Backup) Directory Steps: Change to the staging directory, compress or zip the files (optional), copy resulting files to tape : $ cd /u03/backups/prod1 $ gzip -vN1 * $ find /u03/backups/prod1 | cpio -ovC64 >/dev/rmt0 Compressing while Exporting using UNIX Pipes: $ mknod /tmp/exp_pipe p $ gzip -cNf prod.dmp.gz & $ exp " '/ as sysdba' " file=/tmp/exp_pipe full=y compress=n log=prod.dmp.log $ rm -f /tmp/exp_pipe

32. Evaluation Code 082 32 Additional Nightly Processing (optional)  Create and keep the current copy of your textual control file  Back up the current copy of your textual init.ora file (if you are using a server parameter file (spfile), use the "create pfile from spfile;" command to create it)  Make a full database export for quick table restores and to check datafile integrity  Generate sql definitions for all of your tables and indexes (using indexfile option of the import command)  Gather statistics on datafile and index space usage and table extent growth for proactive maintenance (shown on next slide)

33. Evaluation Code 082 33 Other Nightly Processing Gathering space usage statistics: SQL> select segment_name,segment_type,extents,max_extents from sys.dba_segments where extents + 5 > max_extents or extents > 50 order by segment_name,segment_type desc; SQL> col "Free" format 9,999,999,999 SQL> col "Total Size" format 9,999,999,999 SQL> select tablespace_name,sum(bytes) "Total Size" from dba_data_files group by tablespace_name; SQL> select tablespace_name,sum(bytes) "Free" from dba_free_space group by tablespace_name; For each index, find the number of deleted rows (may rebuild it): SQL> validate index ; SQL> select name,del_lf_rows_len from index_stats;

34. Evaluation Code 082 34 Topics of Discussion  Jaunt: Bulletproofing against data loss  Jaunt: Creating backups (cold and hot)  Recovering from disaster scenarios  Bonus: Implementing a standby database

35. Evaluation Code 082 35 Suddenly Your Database Dies SQL> $ ORA-01115: IO error reading block from file 13 (block # 66771) ORA-01110: data file 13: '/ndxs/oradata/PROD/medi_PROD_01.dbf' ORA-27091: skgfqio: unable to queue I/O ORA-27072: skgfdisp: I/O error IBM AIX RISC System/6000 Error: 5: I/O error Additional information: 66770 @mywebstats.sql ls: /ndxs/oradata/PROD: There is an input or output error. total 0 ls -ltr /ndxs/oradata/PROD

36. Evaluation Code 082 36 Disaster Recovery – What Happened?  Things to check to determine the problem:  System error messages during normal processing  Messages from database startup (backup first)  Alert log in the database's bdump directory (/u00/oracle/admin/PROD/bdump/alert_PROD.log)  Recent bdump and udump trace (*.trc) files (background_dump_dest, user_dump_dest)  Oracle processes ($ ps -ef | grep ora, including: ora_smon_PROD, *pmon*, *dbwr*, *lgwr*)  Recent /home/jobsub/*.lis and *.log files

37. Evaluation Code 082 37 What To Restore (from backup)  ONLY the affected datafile for full recovery  ALL datafiles for incomplete recovery  All archivelog files generated since the datafile backup was made  DON'T restore control files (unless all of them are lost)  DON'T restore online redo log files (unless they are not used during recovery - then must resetlogs:) SQL> connect / as sysdba SQL> startup mount SQL> alter database open resetlogs;

38. Evaluation Code 082 38 Disaster Recovery Overview  Backup first, then try startup (shutdown after)  Primary recovery options:  Recover database (complete or incomplete recovery)  Recover datafile (one datafile; complete recovery only)  Recover tablespace (all of its datafiles; complete only)  Generic recovery steps:  Shutdown database or offline tablespace or datafile  Restore datafile(s) (and archivelogs) from backup  Issue recover command (database, datafile, or tablespace)  Bring tablespace or datafiles online Oracle9i User-Managed Backup and Recovery Guide

39. Evaluation Code 082 39 Basic Recover Database – Complete Recovery State: mount (from shutdown state), datafiles online Steps: Restore datafile(s), mount, online the datafile(s) if needed, recover database, open: $ cp -p /u03/backups/prod1/devl_PROD_01.dbf /u03/oradata/PROD (restore datafile(s)) $ sqlplus "/ as sysdba" SQL> startup mount SQL> select * from v$datafile; if any offline, then: SQL> alter database datafile ' ' online; SQL> set autorecovery on (or recover automatic below) SQL> recover database; SQL> alter database open;

40. Evaluation Code 082 40 Basic Recover Database – Incomplete Recovery State: mount, datafiles online Steps: Restore ALL datafiles, mount, online the datafiles, recover database until..., open. Like complete recovery, except for "recover database" command (use only one): SQL> recover automatic database until time '2005-02-14:15:45:00'; SQL> recover database until cancel; (no autorecovery; steps through logs until CANCEL entered) SQL> recover automatic database until change 43047423; (SCN (system change number) for archivelogs is in v$archived_log as first_change# and next_change# - 1; online redo logs is in v$log as first_change# (subtract 1)) SQL> alter database open resetlogs; (then, backup)

41. Evaluation Code 082 41 Basic Recover Datafile – From Mount State State: mount, datafile online or offline Steps: Restore datafile, mount, recover datafile, online the datafile if needed, open: $ cp -p /u03/backups/prod1/devl_PROD_01.dbf /u03/oradata/PROD (restore datafile) $ sqlplus "/ as sysdba" SQL> startup mount SQL> recover automatic datafile '/u03/oradata/PROD/devl_PROD_01.dbf'; SQL> select * from v$datafile; if datafile is offline, then: SQL> alter database datafile '/u03/oradata/PROD/devl_PROD_01.dbf' online; SQL> alter database open;

42. Evaluation Code 082 42 Basic Recover Datafile – From Open State State: open (with database up), datafile offline Steps: Offline the datafile, restore datafile, recover datafile, online the datafile (not for system datafile): $ sqlplus "/ as sysdba" SQL> alter database datafile '/u03/oradata/PROD/devl_PROD_01.dbf' offline; SQL> !cp -p /u03/backups/prod1/devl_PROD_01.dbf /u03/oradata/PROD (restore datafile) SQL> recover automatic datafile '/u03/oradata/PROD/devl_PROD_01.dbf'; SQL> alter database datafile '/u03/oradata/PROD/devl_PROD_01.dbf' online;

43. Evaluation Code 082 43 Basic Recover Datafile – From Open After Startup State: open, datafile offline Steps: Mount, offline the datafile, open, restore datafile, recover datafile, online the datafile (not for system datafile): $ sqlplus "/ as sysdba" SQL> startup mount SQL> alter database datafile '/u03/oradata/PROD/devl_PROD_01.dbf' offline; SQL> alter database open; SQL> !cp -p /u03/backups/prod1/devl_PROD_01.dbf /u03/oradata/PROD (restore datafile) SQL> recover automatic datafile '/u03/oradata/PROD/devl_PROD_01.dbf'; SQL> alter database datafile '/u03/oradata/PROD/devl_PROD_01.dbf' online;

44. Evaluation Code 082 44 Basic Recover Tablespace – From Open State State: open, tablespace offline Steps: Offline the tablespace, restore tablespace’s datafiles, recover tablespace, online the tablespace (not for system tablespace): $ sqlplus "/ as sysdba" SQL> alter tablespace development offline immediate; (immediate rolls back currently pending transactions) SQL> !cp -p /u03/backups/prod1/devl_PROD* /u03/oradata/PROD (restore datafiles) SQL> recover automatic tablespace development; SQL> alter tablespace development online;

45. Evaluation Code 082 45 Basic Recover Tablespace – From Open After Startup State: open, tablespace offline Steps: Mount, offline any bad datafiles, open, offline the tablespace,restore tablespace’s datafiles, recover tablespace, online the tablespace (not for system tablespace): $ sqlplus "/ as sysdba" SQL> startup mount SQL> alter database datafile '/u03/oradata/PROD/devl_PROD_01.dbf' offline; SQL> alter database open; SQL> alter tablespace development offline; SQL> !cp -p /u03/backups/prod1/devl_PROD* /u03/oradata/PROD (restore datafiles) SQL> recover automatic tablespace development; SQL> alter tablespace development online;

46. Evaluation Code 082 46 Checking Logs and Trace Files Checking alert log and trace files for pmon, lgwr, dbwr, arch: SQL> select value from v$parameter where name = 'background_dump_dest'; $ grep background_dump_dest $ORACLE_HOME/dbs/initPROD.ora $ cd /u00/oracle/admin/PROD/bdump $ tail -200 alert_PROD.log $ ls -ltr *.trc (look at latest pmon, lgwr, dbwr, arch trace files, if any) $ cat prod_pmon_13612.trc(process monitor trace) $ cat prod_lgwr_32306.trc(redo log writer trace) $ cat prod_dbwr_43213.trc(database writer trace) $ cat prod_arch_22882.trc(archiver trace)

47. Evaluation Code 082 47 Archivelogs Disk Volume Filled Up  Symptoms: Database freezes. No space for archivelogs (df -k).  Messages: Users: None (sessions freeze). Logins: ERROR: ORA-00257: archiver error. connect internal only, until freed.  Logs: Alert log: ORA-00272: error writing archive log. Arch trace: ORA-00272: error writing archive log. Note: Use oerr for error description, such as: $ oerr ora 257

48. Evaluation Code 082 48 Archivelogs Disk Volume Filled Up Steps: Free up space on archivelogs volume by moving files or older archivelogs off, or, by deleting old archive log files that have already been backed up: # File: remove_old_logs.shl echo "You must be logged in as user Oracle to run this script," echo "which removes all archivelog files older than X days." echo "Enter number of days to keep: \c" read DAYS_KP; export DAYS_KP find /u01/oradata/PROD/archivelogs -name '*.arc.gz' -mtime +$DAYS_KP -exec rm {} \; find /u01/oradata/PROD/archivelogs -name '*.arc' -mtime +$DAYS_KP -exec rm {} \; echo "Results after deletions:" du -k df -k

49. Evaluation Code 082 49 Loss of Control Files  Symptoms: May be none until shutdown and/or startup.  Messages: Shutdown: If deleted: ORA-00210: cannot open control file '/u03/oradata/PROD/ctrl_PROD_01.ctl'. If overwritten: ORA-00201: control file version incompatible with ORACLE version. (May have to shutdown abort.) Startup: ORA-00205: error in identifying control file '/u03/oradata/PROD/ctrl_PROD_01.ctl'. Also, if overwritten: ORA-07366: sfifi: invalid file, file does not have valid header block.

50. Evaluation Code 082 50 Loss of Control Files – Textual Control File Recovery Steps: Edit out header from latest textual control file you have available (in udump directory), run control file (as long as no datafiles have been added): $ sqlplus "/ as sysdba" SQL> shutdown abort SQL> !ls -ltr /u00/oracle/admin/PROD/udump/*.trc SQL> !vi /u00/oracle/admin/PROD/udump/prod_ora_31494.trc Delete the lines before the STARTUP NOMOUNT line SQL> @/u00/oracle/admin/PROD/udump/prod_ora_31494.trc

51. Evaluation Code 082 51 Loss of Control Files – Backup Control File Recovery Steps: Restore ALL datafiles AND control files (NOT online redo log files), recover using backup controlfile: $ sqlplus "/ as sysdba" SQL> shutdown abort At this point, restore ALL datafiles AND control files from the last backup, but, NOT the online redo log files. SQL> connect / as sysdba SQL> startup mount SQL> recover automatic database using backup controlfile; (AUTO on "ORA-00308: cannot open archived log...") SQL> alter database open resetlogs; (then, backup)

52. Evaluation Code 082 52 Loss of TEMP Datafile  Symptoms: Error message on large sorts (select distinct, order by, group by, union).  Messages: Loss during: ORA-01157: cannot identify data file 3 – file not found. Loss before: ORA-01116: error in opening database file 3. Both Followed by: ORA-01110: data file 3: '/u03/oradata/PROD/temp_PROD_01.dbf'.  Logs: No alert.log entries or trace files generated.

53. Evaluation Code 082 53 Loss of TEMP Datafile State: open or mount Steps: Offline datafile, drop and recreate TEMP tablespace (here we are using locally managed temporary tablespace): SQL> alter database datafile '/u03/oradata/PROD/temp_PROD_01.dbf' offline; If from mount state: SQL> alter database open; SQL> select file_name,bytes/1024 kbytes from dba_temp_files; SQL> select initial_extent/1024 kbytes from dba_tablespaces where tablespace_name = 'TEMP'; SQL> drop tablespace temp; SQL> !rm /u03/oradata/PROD/temp_PROD_01.dbf SQL> create temporary tablespace temp tempfile '/u03/oradata/PROD/temp_PROD_01.dbf' size 40064K extent management local uniform size 640K;

54. Evaluation Code 082 54 TEMP Datafile Offline  Symptoms: Error message on large sorts (select distinct, order by, group by, union). (Like Loss of TEMP Datafile.)  Messages: ORA-00376: file 3 cannot be read at this time. ORA-01110: data file 3: '/u03/oradata/PROD/temp_PROD_01.dbf'. Steps: Recover datafile (DB open), online the datafile: SQL> recover automatic datafile '/u03/oradata/PROD/temp_PROD_01.dbf'; SQL> alter database datafile '/u03/oradata/PROD/temp_PROD_01.dbf' online;

55. Evaluation Code 082 55 Loss of INACTIVE Online Redo Log Group (Archived)  Symptoms: Database crashes switching to lost log group.  Messages: Users: ORA-01092: ORACLE instance terminated. Disconnection forced. Logins: ERROR: ORA-03114: not connected to ORACLE. ERROR: ORA-00472: PMON process terminated with error.  Logs: Alert log: No indication. Pmon trace: ORA-00470: LGWR process terminated with error. Lgwr trace: ORA-00313: open failed for members of log group 3 of thread 1. ORA-00312: online log 3 thread 1: '/u03/oradata/PROD/log_PROD_3B.rdo'.

56. Evaluation Code 082 56 Loss of INACTIVE Online Redo Log Group (Archived) Steps: Mount, drop logfile group, add logfile group, open: $ sqlplus "/ as sysdba" SQL> startup mount If "ORA-01081: cannot start already-running ORACLE – shut it down first", then "startup force" (or "shutdown abort" and "startup mount"). Shows "Database mounted." and "ORA-00313: open failed for members of log group 3 of thread 1". SQL> select bytes/1024 K from v$log where group# = 3; SQL> select member from v$logfile where group# = 3; SQL> alter database drop logfile group 3; SQL> alter database add logfile group 3 ('/u03/oradata/PROD/log_PROD_3A.rdo', '/u03/oradata/PROD/log_PROD_3B.rdo') size 500K; SQL> alter database open;

57. Evaluation Code 082 57 Loss of INACTIVE Online Redo Log Group Member  Symptoms: No apparent symptoms.  Logs: Alert log: ORA-00313: open failed for members of log group 3 of thread 1. ORA-00312: online log 3 thread 1: '/u03/oradata/PROD/log_PROD_3A.rdo'. (Each switch.) Steps: Drop logfile member, add logfile member to group: SQL> select * from v$log where group# = 3; If status is active or current for group, first do: SQL> alter system switch logfile; SQL> alter database drop logfile member '/u03/oradata/PROD/log_PROD_3A.rdo'; SQL> alter database add logfile member '/u03/oradata/PROD/log_PROD_3A.rdo' to group 3;

58. Evaluation Code 082 58 Loss of CURRENT Online Redo Log Group (Unarchived)  Symptoms: Database freezes (after cycling back). Plenty of space for archivelogs (df -k).  Messages: Users: None (sessions freeze). Logins: ERROR: ORA-00257: archiver error. Connect internal only, until freed.  Logs: Alert log: ORA-00286: No members available, or no member contains valid data. ORACLE Instance PROD – Can not allocate log, archival required. Thread 1 cannot allocate new log, sequence 21. All online logs needed archiving. Arch trace: ORA-00286: No members available, or no member contains valid data.

59. Evaluation Code 082 59 Loss of CURRENT Online Redo Log Group (Unarchived) Steps: Startup to get log group number, then do an incomplete recovery (as described earlier) up to the start time for the lost log group (Note: all changes in all redo log groups will be lost): $ sqlplus "/ as sysdba" SQL> shutdown abort SQL> startup Shows: ORA-00313: open failed for members of log group 2 of thread 1. SQL> shutdown abort (continued …).

60. Evaluation Code 082 60 Loss of CURRENT Online Redo Log Group (Unarchived) At this point, restore ALL datafiles AND the lost online redo log group's files from the latest backup, but, NOT the control files. SQL> connect / as sysdba SQL> startup mount SQL> select sequence#, bytes, first_change#, to_char(first_time, 'DD-MON-YY HH24:MI:SS'), status from v$log where group# = 2; SQL> recover automatic database until time '2005-02-14:12:59:59'; (1 second before loss) SQL> alter database open resetlogs; (and backup)

61. Evaluation Code 082 61 Failure During Hot Backup Steps: Mount, determine datafiles in hot backup state, end backup (Oracle 7.2 and above), open: $ sqlplus "/ as sysdba" SQL> startup mount SQL> select df.name,bk.time from v$datafile df,v$backup bk where df.file# = bk.file# and bk.status ='ACTIVE'; SQL> alter database datafile '/u03/oradata/PROD/devl_PROD_01.dbf' end backup; SQL> alter database open;

62. Evaluation Code 082 62 Topics of Discussion  Jaunt: Bulletproofing against data loss  Jaunt: Creating backups (cold and hot)  Recovering from disaster scenarios  Bonus: Implementing a standby database

63. Evaluation Code 082 63 Bonus Topic: Data Guard - Oracle's Answer to Disaster Recovery  See how to quickly implement a Data Guard physical standby database step-by-step in a day.  Learn how to switch over or fail over to your standby database in minutes.  Possibly offload your batch reporting workload to your standby database.  Replace your forebodings about crashes with "Don't worry... be happy!" Oracle Data Guard Concepts and Administration Release 2 (9.2)

64. Evaluation Code 082 64 Data Guard Flow (Oracle 9i)

65. Evaluation Code 082 65 Data Guard Flow (Oracle 10g)

66. Evaluation Code 082 66 Data Guard Protection Modes  Maximum Performance  Updates committed to primary and sent to standby without waiting to see if they were applied to standby  Pros: Little or no effect on performance of primary  Cons: Slight chance of lost transactions on failover  Maximum Availability (we will implement this one)  Attempts to apply updates to standby before committed to primary  Lowers protection to Maximum Performance temporarily if updates can't be applied to standby  Pros: Primary continues unaffected if connection to standby is lost or the updates are delayed  Cons: Slight performance hit on primary; lost transactions on failover possible only if the standby has been unreachable

67. Evaluation Code 082 67 Data Guard Protection Modes  Maximum Protection  Assures updates are applied to standby before committed to primary  Pros: No chance of lost transactions  Cons: Primary will freeze if connection to standby is lost or the updates are delayed

68. Evaluation Code 082 68 Primary Database Requirements for Data Guard  FORCE LOGGING must be enabled: SQL> select force_logging from v$database; SQL> alter database force logging;  ARCHIVELOG mode and automatic archiving must be enabled: SQL> archive log list  MAXLOGFILES >= (2 * Current Redo Log Groups) + 1: SQL> select records_used "Current Groups", records_total "Max Groups" from v$controlfile_record_section where type = 'REDO LOG';

69. Evaluation Code 082 69 listener.ora Additions  Define the standby database SID on the standby site: (SID_DESC= (SID_NAME=PROD2) (ORACLE_HOME=/pgms/oracle/product/v9204) ) (in $ORACLE_HOME/network/admin/listener.ora)

70. Evaluation Code 082 70 tnsnames.ora Additions  Define the standby database connect string on the primary site: myserver_prod2 = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP) (Host = 123.45.67.89) -- whatever host IP has PROD2 (Port = 1521) ) (CONNECT_DATA = (SID = PROD2) ) (define myserver_prod and myserver_prod2 on both primary and standby sites for quick switchovers)

71. Evaluation Code 082 71 sqlnet.ora and /etc/oratab Additions  Enable dead connection detection on the primary and standby sites: sqlnet.expire_time=2 (in $ORACLE_HOME/network/admin/sqlnet.ora)  Add the standby database's entry to /etc/oratab on the standby site: PROD2:/pgms/oracle/product/v9204:N

72. Evaluation Code 082 72 Standby Database Parameter File  Create the initPROD2.ora parameter file to be used for the standby database (done from primary database):  If your primary is using an spfile: $ sqlplus "/ as sysdba" SQL> create pfile='$ORACLE_HOME/dbs/initPROD2.ora' from spfile;  Else, if your primary is using a pfile: $ cp -p $ORACLE_HOME/dbs/initPROD.ora $ORACLE_HOME/dbs/initPROD2.ora  Note: We will be modifying both the primary and standby parameter files to handle being in either the primary or the standby mode for quick switchovers.

73. Evaluation Code 082 73 Standby Database Parameters (changes in copy of primary's values) Change pathnames, such as control_files, background_dump_dest, core_dump_dest, user_dump_dest, and audit_file_dest, and add: # log_archive_dest = /orcl/oradata/PROD2/archivelogs log_archive_dest_1 = 'LOCATION=/orcl/oradata/PROD2/archivelogs MANDATORY' # for switchover log_archive_dest_state_1 = ENABLE # for switchover log_archive_dest_2 = 'SERVICE=myserver_prod LGWR SYNC' # for switchover log_archive_dest_state_2 = ENABLE # for switchover standby_archive_dest = /orcl/oradata/PROD2/archivelogs standby_file_management = AUTO # or MANUAL for raw devices remote_archive_enable = TRUE # TRUE or RECEIVE, change RECEIVE to SEND on switchover instance_name = PROD2 lock_name_space = PROD2 # use when primary and standby on same system; same as instance_name fal_server = myserver_prod # "fal" is Fetch Archive Log, for log gap resolution fal_client = myserver_prod2 db_file_name_convert = ('/PROD/','/PROD2/') log_file_name_convert = ('/PROD/','/PROD2/')

74. Evaluation Code 082 74 Primary Database Parameters (changes in primary's values) #log_archive_dest = /orcl/oradata/PROD/archivelogs log_archive_dest_1 = 'LOCATION=/orcl/oradata/PROD/archivelogs MANDATORY' log_archive_dest_state_1 = ENABLE log_archive_dest_2 = 'SERVICE=myserver_prod2 LGWR SYNC' log_archive_dest_state_2 = ENABLE standby_archive_dest = /orcl/oradata/PROD/archivelogs # for switchover standby_file_management = AUTO # for switchover; or MANUAL for raw devices remote_archive_enable = TRUE # TRUE or SEND, change SEND to RECEIVE on switchover instance_name = PROD lock_name_space = PROD # use when primary and standby on same system; same as instance_name fal_server = myserver_prod2 # for switchover fal_client = myserver_prod # for switchover db_file_name_convert = ('/PROD2/','/PROD/') # for switchover log_file_name_convert = ('/PROD2/','/PROD/') # for switchover (If primary uses spfile, wait until after the standby database files are copied/created to make these parameter changes.)

75. Evaluation Code 082 75 Standby Database Datafiles, etc.  Create the standby control file from the primary database: SQL> alter database create standby controlfile as '/orcl/oradata/PROD2/ctrl_PROD_01.ctl';  Shut down the primary database and copy or FTP its datafiles, redo log files, and the just-created standby parameter file and standby control file, to the standby site.

76. Evaluation Code 082 76 Standby Database Datafiles, etc.  Copy the standby control file on the standby site to the other file names listed in the control_files init.ora parameter.  Create the standby's password file, if needed, on the standby site: $ orapwd file=$ORACLE_HOME/dbs/orapwPROD2 password= entries=5  Reload the listener on the primary and standby sites: $ lsnrctl reload

77. Evaluation Code 082 77 Standby Database Startup  Start the standby database in nomount mode, create the spfile if wanted, mount the standby database, and change to managed recovery: $. oraenv PROD2 $ sqlplus "/ as sysdba" SQL> create spfile from pfile; SQL> startup nomount SQL> alter database mount standby database; SQL> alter database recover managed standby database disconnect from session; SQL> exit

78. Evaluation Code 082 78 Primary Database Startup  If your primary is using an spfile, set the primary database parameters in the spfile as listed earlier: SQL> startup nomount SQL> alter system reset log_archive_dest scope=spfile sid='*'; SQL> alter system set log_archive_dest_1 = 'LOCATION=/orcl/oradata/PROD/archivelogs MANDATORY' scope=spfile; … etc … SQL> shutdown

79. Evaluation Code 082 79 Primary Database Startup  Start up the primary database with the new parameters: SQL> startup  Start archiving to the standby database by issuing a log switch: SQL> alter system switch logfile; Congratulations! You now have a working standby database for your primary database. But wait … There's more …

80. Evaluation Code 082 80 Add Standby Redo Log Groups to Standby Database  Create standby redo log groups on standby database (one more than current redo log groups): $ sqlplus "/ as sysdba" SQL> alter database recover managed standby database cancel; SQL> alter database open read only; SQL> select max(group#) maxgroup from v$logfile; SQL> select max(bytes) / 1024 "size (K)" from v$log; SQL> alter database add standby logfile group 4 ( '/orcl/oradata/PROD2/stby_log_PROD_4A.rdo', '/orcl/oradata/PROD2/stby_log_PROD_4B.rdo') size 4096K; … etc … SQL> column member format a55 SQL> select vs.group#,vs.bytes,vl.member from v$standby_log vs, v$logfile vl where vs.group# = vl.group# order by vs.group#,vl.member;

81. Evaluation Code 082 81 Add Tempfile To Standby  Add a tempfile to the standby database for switchover or read-only access, then, switch back to managed recovery: SQL> alter tablespace temp add tempfile '/data/oradata/PROD2/temp_PROD_01.dbf' size 400064K reuse; SQL> alter database recover managed standby database disconnect from session; SQL> select * from v$tempfile; SQL> exit

82. Evaluation Code 082 82 Add Standby Redo Log Groups to Primary Database  Create standby logfile groups on the primary database for switchovers (one more than current redo log groups): $ sqlplus "/ as sysdba" SQL> select max(group#) maxgroup from v$logfile; SQL> select max(bytes) / 1024 "size (K)" from v$log; SQL> alter database add standby logfile group 4 ( '/orcl/oradata/PROD/stby_log_PROD_4A.rdo', '/orcl/oradata/PROD/stby_log_PROD_4B.rdo') size 4096K; … etc … SQL> column member format a55 SQL> select vs.group#,vs.bytes,vl.member from v$standby_log vs,v$logfile vl where vs.group# = vl.group# order by vs.group#,vl.member;

83. Evaluation Code 082 83 Switch To Maximum Availability Protection Mode  Switch to the desired "maximum availability" protection mode on the primary database (from the default "maximum performance"): SQL> select value from v$parameter where name = 'log_archive_dest_2'; -- must show LGWR SYNC SQL> shutdown normal SQL> startup mount SQL> alter database set standby database to maximize availability; SQL> alter database open; SQL> select protection_mode from v$database;

84. Evaluation Code 082 84 Test Updates Propagating to Standby  Try some edits on the primary and check to see that the changes made it to the standby:  On the primary: SQL> update spriden set spriden_first_name = 'James' where spriden_pidm = 1234 and spriden_change_ind is null; SQL> commit; SQL> alter system switch logfile;  On the standby (wait a few seconds first): SQL> alter database recover managed standby database cancel; SQL> alter database open read only; SQL> select * from spriden where spriden_pidm = 1234 and spriden_change_ind is null; SQL> alter database recover managed standby database disconnect from session;

85. Evaluation Code 082 85 Running Reports with a Standby  Set standby to Read Only to run reports: SQL> alter database recover managed standby database cancel; SQL> alter database open read only; SQL> @myreport.sql SQL> alter database recover managed standby database disconnect from session;

86. Evaluation Code 082 86 Shutdown and Startup for Standby Database  To shut down a standby database:  If in read-only access, switch back to managed recovery (after terminating any other active sessions): SQL> alter database recover managed standby database disconnect from session;  Cancel managed recovery and shutdown: SQL> alter database recover managed standby database cancel; SQL> shutdown immediate  To start up a standby database: SQL> startup mount SQL> alter database mount standby database; SQL> alter database recover managed standby database disconnect from session;

87. Evaluation Code 082 87 Switchover - Swapping Primary and Standby  End all activities on the primary and standby database.  On the primary (switchover status "TO STANDBY"): SQL> select database_role,switchover_status from v$database; SQL> alter database commit to switchover to physical standby; SQL> shutdown immediate SQL> startup nomount SQL> alter database mount standby database;  On the standby (switchover status "SWITCHOVER PENDING"): SQL> select database_role,switchover_status from v$database; SQL> alter database commit to switchover to primary; SQL> shutdown normal SQL> startup  On the primary: SQL> alter database recover managed standby database disconnect from session;  On the standby: SQL> alter system archive log current;  Change tnsnames.ora entry on all servers to swap the connect strings (myserver_prod and myserver_prod2).

88. Evaluation Code 082 88 Failover - Standby Becomes Primary  End all activities on the standby database.  May need to resolve redo log gaps (not shown here).  On the standby: SQL> alter database recover managed standby database finish; SQL> alter database commit to switchover to primary; SQL> shutdown immediate SQL> startup  Change tnsnames.ora entry on all servers to point the primary connect string to the standby database.  New standby needs to be created. Old primary is no longer functional.

89. Evaluation Code 082 89 Summary  Jaunt: Bulletproofing against data loss  Enabling archiving of redo log files  Separating types of database files  Moving datafiles  Adding redo log members  Mirroring log files and control files  Jaunt: Creating backups (cold and hot)  Identifying files to back up  Performing cold backups (DB down)  Performing hot backups (DB up)  Statistics, exports, and other tasks

90. Evaluation Code 082 90 Summary  Recovering from disaster scenarios  Determining cause of failure  Identifying files to restore  Basic disaster recovery scenarios  Complete and incomplete  Database, datafile, tablespace  Specific file type recovery scenarios  Step-by-step recovery commands

91. Evaluation Code 082 91 Summary  Bonus Topic: Implementing a standby database  Data Guard provides an automated standby database which can essentially eliminate downtime of your production data.  Setup is easy and fairly straightforward.  Maintenance is minimal.  Switchovers and failovers can be done within a few minutes.  Reporting can be offloaded to the standby to ease the workload on the primary.  And … It's Free! (Included with Enterprise Edition)

92. Evaluation Code 082 92 Summary http://www.uaex.edu/srea/bkupreco.htm http://www.uaex.edu/srea/dataguard.htm  Backup, backup, backup  Practice, practice, practice your disaster recovery plans  Don’t Panic! Follow your plan

93. Evaluation Code 082 93 Questions and Answers Whew! Glad That’s Over! Any Questions?

94. Evaluation Code 082 94 Thank You! Stephen Rea srea@uaex.edu More Oracle and Banner information can be found at my Oracle Tips, Tricks, and Scripts web site: http://www.uaex.edu/srea Please complete the on-line Evaluation Form Evaluation Code 082