1. 1 Hüsnü Şensoy Turkcell Telecommunication Services VLDB Expert Oracle ACE Director Member of Global DWH Leaders & Oracle CAB Oracle DBA of 2009

2. 2 Agenda Backup & Recovery Strategies for Oracle Databases – Motivation behind those strategies – Revisiting “Incrementally Updated Backup” – Revisiting “FRA” – How to bring your database back without restore ? – Sick backup will not work – Centralized scheduling & monitoring 11g Release 2 Backup & Recovery New Features with real Telco data warehouse data – Brand new compression algorithms Summary

3. 3 Turkcell Overview Leading GSM operator of Turkey established in February 1994. Third GSM operator in Europe in terms of subscriber (+36 million). First and only Turkish company ever to be listed on New York Stock Exchange. Member of Board of Directors of GSMA since 2003. 25th company of INFOTECH 100 list.

4. 4 BACKUP & RECOVERY STRATEGIES FOR ORACLE DATABASES

5. 5 Design Considerations Define your backup & recovery policies upfront – A well documented strategy that can be used to bring everything back – KISS: – KISS: Even a junior DBA should be able to bring your database back. – Standardize, standardize, standardize… – Be prepared to justify the cost in terms of business impact of downtime

6. 6 Design Considerations Proactively validate database and backup integrity – Physical errors – Logical inconsistencies – Transmission errors Do you perform regular full recoveries to separate host and storage?

7. 7 Design Considerations Centralized backup reporting: – Is there a single point of access for all my databases’ backup logs ? – What is the average backup duration for database X ? – How do brand new tape drives affect backup performance ?

8. 8 What’s in there ? 7 RAC databases More than 20 services What’s in there ? 7 RAC databases More than 20 services 20 Gbit/s 120 Intel Cores 640 GB Memory 25 TB DATA ARCHIVEFRA What type of Architecture ? VASRESVASRES VASNIFVASNIF VASCMTVASCMT APPDBAPPDB BSSARCHBSSARCH BSSOSSBSSOSS VASSEVASSE

9. 9 How Do We Backup ? Incrementally Updated Backup Strategy Initial image copy backup to FRA Fast incremental backups thereafter Image copy is rolled forward with incremental backup on regular basis to create full on- disk backup Full database backup times only depend on the amount of blocks changed since last incremental backup. The longest backup time is only ~30 minutes, with ZLIB backup compression and logical block checking turned on. run{ backup as compressed backupset check logical incremental level 1 for recover of copy with tag DAILY_COPY database filesperset 1; recover copy of database with tag DAILY_COPY; } This is the shortest, cleanest, and most elegant backup script that I have seen in all my years at Turkcell.

10. 10 Setting Up F Setting Up Flash Recovery Area (Oracle Database 11g Release 1) Self managed & organized logical storage area. Setup as part of Universal Installer wizard. – Redo log copy, control file copy, archived logs, and Flashback logs are automatically stored there. – RMAN automatically utilizes FRA for all disk backups. Or, just enable it by setting two init.ora parameters : – db_recovery_file_dest_size – db_recovery_file_dest

11. 11 FRA Flash Recovery Area ASM is the best infrastructure to be used as FRA destination: – Raw device performance. – No other solution (except Sun ZFS file system with its online FS check capability) will practically let you implement large storage pools as ASM does. – Ease of management. ASM allows you to provision the same diskgroup to multiple FRA destinations. ASM Diskgroup (+FRA) DB1 FRA DB2 FRA DB3 FRA DB4 FRA

12. 12 Restore-Free Recovery Create a pfile whose control_file parameter just points the FRA copy of controlfile Mount DB Switch database to copy Recover databaseOpen database

13. 13 What Are the Commands? Step1 SQL> startup pfile=/home/oracle/init.ora nomount; ORACLE instance started. … Step 2 RMAN> switch database to copy; using target database control file instead of recovery catalog datafile 1 switched to datafile copy "+FRA/disaster/datafile/system.503.678209167" … datafile 9 switched to datafile copy "+FRA/disaster/datafile/undotbs5.510.678209175" Step 3 RMAN> recover database; Starting recover at 07-FEB-09 using channel ORA_DISK_1 starting media recovery media recovery complete, elapsed time: 00:00:03 Finished recover at 07-FEB-09 Step 4 RMAN> alter database open; database opened From hours to minutes

14. 14 Backups on disk or tape might be damaged due to – Physical problems on media (fabric problems, dust, cosmic rays, etc) – Media library errors (error in checksum computation) How you can increase the probability that your backups are healthy ? Backup Validation

15. 15 Protective action for possible problems. It will slightly effect your backup time ( check logical ) Prevent Errors at Backup Time Good if the error is a temporary/stochastic one. It has an additional cost of time and media. Multiplex Backups Cost effective method for the majority of the problems Not 100% coverage for any kind of errors. Probe Backups Most effective one among all Most costly one among all. Restore them Possible Solutions

16. 16 RMAN Backup Validation RMAN> backup check logical validate datafilecopy all filesperset 1; This will report – For any inconsistent data, index, or other type of blocks. – Number of total and empty blocks examined. – Highest change number of each datafile copy.

17. 17 Centralized Scheduling & Monitoring Develop standard backup job scheduling and monitoring routines. This enables you to: – See all backup schedules at once – Check details of previously completed backups (duration, logs, etc.) – Easily modify backup scripts and bulk deploy them.

18. 18 Grid Control Backup Jobs Manage backup of all databases of the cluster by using just one screen

19. 19

20. 20

21. 21 11G RELEASE 2 RMAN COMPRESSION

22. 22 11gR2 RMAN Compression RMAN Compression Pre-Compression Block Processing Binary CompressionBasicAdvancedHIGHMEDIUMLOW

23. 23 Test Setup Data Marketing data from Turkcell data warehouse 2.2 billion records (140G) No segment compression PCTFREE 1 16K block size tablespace Number of Channels 8 RMAN Channels Compression Types NONE BASIC LOW MEDIUM HIGH Collected Metrics Compression Ratio Duration I/O Throughput CPU Utilization

24. 24 Backup Compression Summary In Oracle Database 11g Release 2, RMAN extends its compression capabilities to fit any CPU power and I/O throughput combination. MEDIUM compression level can backup faster than BASIC with the same compression ratio and 3X faster with 50% less CPU utilization. Even if you don’t have need to reduce backup sizes, LOW/MEDIUM compression level might be faster than uncompressed backup depending on your I/O throughput, by significantly reducing the amount of data/sec written by RMAN.

25. 25 Best Practices Summary A well defined, documented, standard, manageable, and fast backup & recovery strategy is a MUST if you manage tens (even hundreds) of databases. Whatever solution you pick, the indicator of a good backup & recovery strategy is simple: – It shouldn’t depend on the size of database. FRA over ASM and RMAN satisfies these requirements with zero cost.