1. Backup and restore of Oracle databases: introducing a disk layerby
Ruben Gaspar
IT-DB-DBB
BR evolution: Backup to disk

2. Agenda CERN Oracle databases & Oracle backup basics
Backup to disk implementation details
Recovery platform
Some bits of backup to disk backend
Summary
BR evolution: Backup to disk- 2

3. Agenda CERN Oracle databases & Oracle backup basics
Backup to disk implementation details
Recovery platform
Some bits of backup to disk backend
Summary
BR evolution: Backup to disk- 3

4. Target Oracle databases for backup to disk
~70 Oracle databases, most of them running
Oracle clusterware (RAC)
49 are being backed up to disk and then tape
21 are just backed up with snapshots. Test and development instances.
15 Data Guard RAC clusters in Prod
Active Data Guard since upgrade to 11g
They are just backed up to tape
10 Oracle single instance in DBaaS also backed up using snapshots.
Redo Transport
BR evolution: Backup to disk- 4

5. Oracle backup basics The Oracle clock: System Change Number (SCN)
It will take 544 years to run out of SCN at 16K/s
smon_scn_time tracks time versus SCN
Type of backups
Consistent: taken while database has been cleanly shutdown. All redo applied to data files. Archive logs are not produced.
Inconsistent: taken while database is running. Database must be in archivelog mode. It means archive logs will be produced. Point in Time Recoveries (PITR) are possible. Drawback: clean-up of archivelogs is critical to avoid that database blocks → TSM was playing a critical role here
Backup sets: Oracle proprietary format for backups. Binary files.
Backup sets are containers for one or several backup pieces
Backup pieces contain blocks of 1 or several data files (multiplexing)
RMAN channels: disk or tape or proxy, read data files and write back to the backup media. We use SBT: serial backup to tape API, using IBM Tivoli Data Protection 6.3 (provided by TSM support)
BR evolution: Backup to disk- 5

6. Oracle backup basics (II)
Backup jobs based on templates. Recovery Manager API
--Full
backup incremental level 0 database;
--comulative
backup incremental level 2 cumulative database;
--Incremental
backup incremental level 1 database;
--Archivelogs
backup tag 'BR_TAG' archivelog all delete all input;
Retention policy from 60 to 90 days, depending on DB.
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 90 DAYS;
e.g. LEMONRAC → [1xfull + 6xdifferential + archivelogs] * 13 weeks
Controlfile backup, automatically taken by each backup
CONFIGURE CONTROLFILE AUTOBACKUP ON;
e.g. LHCBSTG → [2xfull + 5xdifferential + 24x4 archivelogs] *13 weeks = 934GB 1 2
Arch
Full
Cum.
Inc
PITR
Fulls (GB)
Inc (GB)
Archived logs
Total
LEMONRAC
857.52
BR evolution: Backup to disk- 6

7. BR evolution: Backup to disk- 7

8. What is there to be backed up ?
Backup jobs using RMAN API take care of :
Database files: user and system files
Control files: contain structure and status of data files. They have also all backup history
Archived logs: backup of redo logs. Needed for inconsistent backup strategies. They need to be backed up and removed from the active file system otherwise if running out of space, database freezes/stops.
5.1TB redo logs produced per day
ALL THREE ARE CRITICAL FOR A BACKUP/RECOVERY strategy
BR evolution: Backup to disk- 8

9. Agenda CERN Oracle databases & Oracle backup basics
Backup to disk implementation details
Recovery platform
Some bits of backup to disk backend
Summary
BR evolution: Backup to disk- 9

10. Backup architecture
Custom solution: about 15k lines of code, Perl + Bash
Flexible: easy to adapt to new Oracle release, backup media
Based on Oracle Recovery Manager (RMAN) templates
Central logging
Easy to extend via Perl plug-ins: snapshot, exports, RO tablespaces,…
We send compressed:
1 out of 4 full backups
All archivelogs
BR evolution: Backup to disk- 10

11. Impact on TSM Savings depend on database workload,
e.g.: backup sets on disk for three databases DB
Full (GB)
Inc (GB)
Archived logs (GB)
Savings
EDHP
70%
CASTORNS
71%
ATLASSTG
45% +
x 1/4
Sent to tape
+ backup sets are compressed (see later)
Source: TSM support
Savings ~ 71% 17 5
BR evolution: Backup to disk- 11

12. Impact on TSM (II) ~70% savings ~47% savings
15 accounts: alicestg,atlasstg,cmsstg,castorns,..
Source: TSM support
~70% savings
29 accounts: pdb,wcernp,ITCORE,AISDBP,…
~47% savings
BR evolution: Backup to disk- 12

13. Workflow for disk/tape backups
Same workflow as per tape backups → to ease maintenance
Disk or Tape templates are almost identical, just channel allocation differs
Disk channel allocation calculated on the fly considering available space in aggregate and file system: using Netapp management API called ZAPI
About 75 templates to adapt to all type of backup strategies
Tape and disk backup strategies co-exist
Reversible changing from one to another is a matter of changing templates.
DISK
BR evolution: Backup to disk- 13

14. Typical DB architecture
LAN
Public interface
Public interface
10GbE
RAC
10GbE
10GbE
Interconnect
10GbE
1 GbE
C-mode
Cluster interconnect
Private network
10GbE
1 GbE
10GbE
7-mode
6Gb/s
Media
Manager
Server
mgmt network
6Gb/s 01 02 03 04
backup01
backup02
IBM TSM
Archivelogs controfile
datafiles
At least 2 file systems for backup to disk:
/backup/dbsXX/DBNAME
BR evolution: Backup to disk- 14

15. New C-mode features Transparent file system movements:
cluster01::> volume move start -destination-aggregate aggr1_c01n02 -vserver vs1 -volume castorns03 -cutover-window 10
DNS load balancing inside the cluster
Automatic virtual IP rebalancing (based on failover groups)
Access security via “export-policy” joins firewall + different authentication mechanisms: sys, krb5, ntlm
Global namespace
Compression and Deduplication
We strongly rely on compression as the way to satisfy 2.3PB of backup set storage needs using 1.1PB of disk
The DNS load balancing zone dynamically calculates the load on all LIFs. Every LIF is assigned a
weight based on its port load and CPU utilization of its home node. LIFs that are on less-loaded ports
have a higher probability of being returned in a DNS query. Weights can also be manually assigned.
The commands to manually assign weights are available only at the advanced privilege level.
* Automatic LIF rebalancing gets disabled if a LIF is enabled for automatically reverting to the
home-port (by enabling the auto-revert option in the network interface modify
command).
BR evolution: Backup to disk- 15

16. Backup to disk configuration on database servers
Global namespace in use: /backup/dbsXX
Ease management: mount point unchanged as data moves. It’s a Netapp C-mode feature (see later)
7-mode: mount –o … priv-controllerIP:/vol/castorns03 /ORA/dbs03/CASTOR
C-mode: mount -o … public-ip-cluster:/backup/dbs01/CASTORNS /backup/dbs01/CASTORNS
/backup/dbs01/<DBNAME> → autobackup controlfile + backupsets
/backup/dbsXX/<DBNAME> → backupsets
RMAN configuration parameters: minimal change
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/backup/dbs01/<DBNAME>/<DBNAME>_%F';
BR evolution: Backup to disk- 16

17. for users’ access and for disaster recovery
Particular cases
Solution also operational in a Data Guard configuration: full and incremental taken on standby (more while talking about restores)
Multiple channels: rman_channels_connect in order to distribute backup load
Plug-in for RO tablespaces backup (ACCLOG: size about 170TB, growth 70TB/year)
Automatic clean-up in case of tablespace state change
One backup set per tablespace
Extension to allow special mount points (ACCLOG)
rman_mounts_readonly
full + incremental + controlfile
archivelogs + controlfile
Redo Transport
Primary Database
Active Data Guard
for users’ access and for disaster recovery
BR evolution: Backup to disk- 17

18. Backup to disk performance
Backups run faster ~ 50% than on tape
ACCLOG full backup 5TB
34 hours ~ 35 MB/s
Tape
14 hours ~ 100MB/s
Disk
Sending backup sets from disk to tape needs optimisation
Work on progress with TSM support
BR evolution: Backup to disk- 18

19. Backup to Disk space consumption
Channels order is important → storage management
Space distribution should be according planning to avoid miss balance. File systems should grow at same pace.
Emptiest volume is always selected on top
Automatic size extension
BR evolution: Backup to disk- 19

20. Agenda CERN Oracle databases & Oracle backup basics
Backup to disk implementation details
Recovery platform
Some bits of backup to disk backend
Summary
BR evolution: Backup to disk- 20

21. Recovery platform Only reliable proof of truth: run a recovery
Any change introduce in backup platform/backup strategy is always validated via test recoveries
Isolation
Run independently of the production database
Cant access any other system (database network links)
No user jobs must run
Flexible and easy to customize
Maximize recovery server: several recoveries at the same time
Exports taken after a successful recovery → help in support cases: mainly logical errors
Open source:
BR evolution: Backup to disk- 21

22. Recovery platform (II)
Introducing disk buffer highly improves our recovery testing
Also tested with Data Guard configurations:
Data Guard: Oracle support ID
RMAN> set backup files for device type disk to accessible
Restore from disk are usually 50% faster
More recoveries can be run, nowadays about 40 recoveries per week
No blocking of tape resources that could be used by backups
BR evolution: Backup to disk- 22

23. Agenda CERN Oracle databases & Oracle backup basics
Backup to disk implementation details
Recovery platform
Some bits of backup to disk backend
Summary
BR evolution: Backup to disk- 23

24. Backup to disk cluster 2xFAS6240 Netapp controllers
24xdiskshelf DS4243
24x3TB SATA disks each (576 disks)
raid_dp (raid6) → 1.1 PB usable space split into 8 aggregates ~ 135TB each
2xquad core 64bit Intel(R) Xeon(R) CPU E5540 @ 2.53GHz
10gbps connectivity
Multipath SAS loops 3 gbps
Flash cache 512GB per node
BR evolution: Backup to disk- 24

25. How fast, How compressed
Compression (datafiles)
Online compression of datafiles ~55% (saved by compression)
Backupsets compression of a 501 GB tablespace of random alphanumeric strings, dbms_random.
*Ontap fas6240, 72x 3TB SATA disks.
no-compressed (t)
basic
low
medium
high
No-compressed-fs
Cron- compression
Netapp 8.1.1
Inline-compression
501GB
83GB (6h21’)
116GB (49’)
88GB (07h23’)
82GB (11h02’)
459GB(41’)
188GB
188GB(46’)
Percentage saved (%)
83%
76,8%
82,4%
83,6%
8,3%
62%
BR evolution: Backup to disk- 25

26. Compression: real values
Used(GB)*
Saved (GB)
%saved-by-compression
AISDB_PROD
24719
25941 52
CASTORNS
3629
3448 49
CMSSTG
6510
6395 50
CSR
20636
32008 61
ITCORE
16387
23552 60
EDHP
9631
24913 66
LEMONRAC
47104
49152 51
*Space used on controller side
Logical space used: Used + Saved
BR evolution: Backup to disk- 26

27. NAS controllers throughput
net_data_recv
disk_data_written
compression ratio
BR evolution: Backup to disk- 27

28. Deduplication
When combined with compression, it doesn’t provide good results
Due to the way compression works: compression group: 32k, our Oracle block is 8k, Wafl block is 4k
Control files are a different story. Block size of 16k 4k 4k
DB         Type                      Location               Size(GB)
PAYP     archives               /backup/dbs
PAYP     archives               /backup/dbs
PAYP     controlfile           /backup/dbs
PAYP     fullinc    /backup/dbs
PAYP     fullinc    /backup/dbs
Checksum
BR evolution: Backup to disk- 28

29. Agenda CERN Oracle databases & Oracle backup basics
Backup to disk implementation details
Recovery platform
Some bits of backup to disk backend
Summary
BR evolution: Backup to disk- 29

30. Summary Backup and Recovery testing is critical
Tape copies are essential but TSM became a critical point of failure for DB services
Adding a disk buffer
Removes TSM criticality
Reduces DB volume in TSM
Speeds up backups and restores
Better response time
Better resource utilization
Disk buffer plug-ins were easily integrated in our backup framework
First system to exploit Ontap C-mode features
Valuable experience for the future
BR evolution: Backup to disk- 30

31. Questions ?
BR evolution: Backup to disk- 31