1. Oracle ASM Reduces Cost of VLDB Deployment
Hanan Hit, Principal Database Architect
Lina Shabelsky, Senior Application and Database Engineer
NOCOUG Winter Conference February 08, 2007

2. Confidential and Proprietary
Agenda
Application description
What ASM is and is not
Protocol etc.
Deployment models
Network design
Physical architecture
Oracle operations to reduce required I/O
Index creation stats
Best practices
Lessons learned during VLDB implementation
Backup and recovery
Confidential and Proprietary

3. Application Description

4. Logical Database Structure
Management Db
Operational
Data Store
Historical
Archive
(SATA)
OLAP
Confidential and Proprietary

5. General Loading Schema
Staging
ODS –
Last Hour
Data Warehouse
OLAP
Bus
Loader
Aggregators
Confidential and Proprietary

6. High Level – Physical Layer
Confidential and Proprietary

7. What ASM is and is not

8. ASM - Flexible, Efficient, Time Saver
Vertically integrated file system
Easy file management system
Management of:
Raw device volume
Oracle data files
Online and archive logs
RMAN backup
Multi platform availability
Stripes data across all raw volumes
Hot spot detection and correction
Optional mirroring (best with SAN/NAS) but striping is NOT
Oracle 10g Release 1 and above
Confidential and Proprietary

9. Confidential and Proprietary
What ASM is NOT
A cluster file system
Available for non Oracle files
Available for OCR and voting disks
General file system
Confidential and Proprietary

10. Confidential and Proprietary
Options prior to ASM
Raw
With LVM
Without LVM
Cooked
With LVM
Without LVM
Confidential and Proprietary

11. Why We Chose ASM in the First Place?
New implementation using 10g
Linux RHEL shop
Mid range storage array - full enterprise business needs
“Just trust the hardware to handle it”
Not a workable solution (DBA’s)
Expected thousands of data files
Didn’t want to use the BIG File option (YET).
RAC & non RAC implementations
Obvious need for storage growth with unpredictable limit size
Do more with less
Very few DBAs & sys/storage admin
Block level access to storage
Easy storage provisioning
Confidential and Proprietary

12. Protocol

13. Confidential and Proprietary
Which Protocol to Use?
FC – SAN (2/4 GBit/s)
iSCSI – IP-SAN (2/4 GBit/s)
NFS – NAS (2/4 GBit/s)
FCIP
Confidential and Proprietary

14. Confidential and Proprietary
What is iSCSI Protocol?
A network protocol standard that allows the use of the SCSI protocol over TCP/IP networks
A transport layer protocol in the SCSI-3 specifications framework
Expected to capture more than 10% of storage systems revenue and an even greater percentage of capacity by Hot technologies for 2007 (Storage Magazine)
“iSCSI SAN is definitely happening”
- Stephen Foskett, GlassHouse Technologies Inc., Framingham, MA
Confidential and Proprietary

15. Wikipedia iSCSI Definitions
iSCSI initiator in client/server terminology, is akin to a client device that connects to some service offered by the server (in this case an iSCSI target).
An iSCSI target is akin to a server, in that it provides block level access to its storage media (usually a hard drive, but can be other types of SCSI devices).
Only one iSCSI initiator can talk to a given iSCSI target at a time (one-to-one).
Confidential and Proprietary

16. Confidential and Proprietary
Simple iSCSI system
Confidential and Proprietary

17. Deployment Models

18. Confidential and Proprietary
Deployment Models
Separate Disk Groups
Use storage base features for ASM deployment data management
Backup/recovery, cloning etc.
Shared Disk Groups
Use Oracle tools exclusively for all data management methods
Confidential and Proprietary

19. Confidential and Proprietary
Separate Disk Groups
Confidential and Proprietary

20. Confidential and Proprietary
Shared Disk Groups
Confidential and Proprietary

21. Shared Disk Group- Multiple RAC Instances
Confidential and Proprietary

22. Network Design

23. Typical Performance in MB/s
System Component
Throughput /Performance
Bits
Bytes
16 Port Switch8 *
2GBit/s
1200 MB/s
Fibre Channel
2 GBit/s
200 MB/s
Disk Controller
180 MB/s
GigE NIC
80 MB/s
1 Gbit HBA
1 GBit/s
100 MB/s
2 Gbit HBA
Confidential and Proprietary

24. iSCSI End Point Options
Confidential and Proprietary

25. Network Configuration
Confidential and Proprietary

26. Physical Architecture

27. Confidential and Proprietary
FAS3050 #1
Confidential and Proprietary

28. Confidential and Proprietary
FAS3050 #2
Confidential and Proprietary

29. Confidential and Proprietary
FAS3020
Confidential and Proprietary

30. Confidential and Proprietary
Database size
Confidential and Proprietary

31. Oracle operations to reduce required I/O

32. Oracle Operations - Reduce Disk I/O
Index creation on many billions of rows
New software version while supporting legacy data
Gathering Oracle stats
Confidential and Proprietary

33. Index creation statistics

34. Index Creation – New App Version
Set/unset the SKIP_UNUSABLE_INDEXES
When set to TRUE Oracle will not attempt to use or report errors when an index is marked as unusable
System Level - ALTER SYSTEM SET SKIP_UNUSABLE_INDEXES = TRUE;
Session Level ALTER SESSION SET SKIP_UNUSABLE_INDEXES = TRUE;
Confidential and Proprietary

35. Index Creation – New App Version
Step 1 - Create the new index with the UNUSABLE attribute
Step 2 - System Level - ALTER SYSTEM SET SKIP_UNUSABLE_INDEXES = TRUE;
Step 3 - ALTER INDEX <index name> MODIFY PARTITION <partition name> UNUSABLE;
On all partitions. If using a Global Index then mark the entire index as UNUSABLE
Step 4 - ALTER INDEX <index name> REBUILD PARTITION <new partition name> NOLOGGING;
On the most fresh partition this will enable the application to run with the new index ASAP.
Step 5 - ALTER INDEX <index name> REBUILD PARTITION <partition name> NOLOGGING;
On all the legacy (non hotspot) partitions. This will be rebuild according the system workload.
Final Step - System Level - ALTER SYSTEM SET SKIP_UNUSABLE_INDEXES = FALSE;
Confidential and Proprietary

36. Gathering Oracle Stats
Gather partitions statistics (First time)
Exec dbms_stats.gather_table_stats(ownname=>‘<schema name>', tabname => ‘<tab name>', PARTNAME=> ‘<part name>' , estimate_percent => DBMS_STATS.AUTO_SAMPLE_SIZE, cascade=>TRUE, method_opt=>'FOR ALL COLUMNS SIZE AUTO');
Exec dbms_stats.gather_table_stats (ownname=>‘<schema name>' , tabname=>‘<tab name>' , PARTNAME=> '<part name>' , estimate_percent=>1,CASCADE=> true);
Unlock Statistics (If previously locked)
Exec dbms_stats.unlock_table_stats(OWNNAME=>‘<schema name>' , TABNAME=>‘<tab name>');
Confidential and Proprietary

37. Gathering Oracle Stats – More
Copy statistics
Execdbms_stats.COPY_TABLE_STATS(OWNNAME=>‘<schema name>' , TABNAME=>‘<tab name> ' , SRCPARTNAME=>‘<N’th partition name>' , DSTPARTNAME=>'<N’th+1 partition name>');
Lock table statistics
Exec dbms_stats.lock_table_stats(OWNNAME=>‘<schema name>' , TABNAME=>‘<tab name>');
Confidential and Proprietary

38. Index Creation Example
Confidential and Proprietary

39. Index Creation Example – More
Confidential and Proprietary

40. Best Practices

41. NetApp/ASM – Our Best Practices
Single aggregate
FlexVol
ASM external redundancy
Separate disk groups
Maximum volume size allowed 16TB while recommended is not more then 3TB
Use RAID-DP with maximum 16 drives in a single RAID group
Set minra to off – even on DSS
Jumbo frames – 9K MTU
Single mode VIF
Confidential and Proprietary

42. Lessons learned during VLDB implementation

43. RAC 10gR2 Implementation Learnings
Implementation of VLDB in RHEL 4.3, software iSCSI, NetApp and ASM environment
Procedure of provisioning additional storage
Backup procedures with NetApp SnapManager for Oracle
Confidential and Proprietary

44. Network Implementation
Isolate your database from the rest of the IP traffic
Use separate VLAN for access to the storage
Use oifcfg to verify the setup of your interfaces
Use IP bonding on both – server and filer to achieve network stability and as an alternative solution to multipathing
IP bonding allows you to aggregate multiple network interfaces into higher performance network link and provides failover solution
Confidential and Proprietary

45. Confidential and Proprietary
Oracle Clusterware
Carefully plan your file system layout
If you use ASMLIB with iSCSI disks, don’t map OCR and voting disk to raw devices in RAC environment – use OCFS or NFS
Raw devices are not aware of ASMLIB
In RedHat 4 update 3, the use of /etc/sysconfig/rawdevices is deprecated
Confidential and Proprietary

46. Confidential and Proprietary
Storage
When planning for the large database consult your storage vendor about RAID configuration, storage provisioning and backup procedures.
As an example – using NetApp as storage appliance allows DBAs to take advantage of RAID-DP (Double Parity) – fault tolerance of RAID 1 at the price of RAID 4.
NetApp flexible architecture allows DBAs to provision additional storage with minimal downtime.
SnapManager for Oracle allows DBAs to combine NetApp snapshot technology with Oracle RMAN to dramatically decrease time taken to backup and restore database.
Confidential and Proprietary

47. Storage Provisioning with ASM
Adding 1TB of storage takes no more than
30 minutes with Netapp
Step1 – Ask your system administrator to plug in Netapp shelves
Step 2 – login to the filer and add new disks to the aggregate leaving 2 spares per array
Step 3 – create new volume and LUNs, add them to the initiator group
Confidential and Proprietary

48. Confidential and Proprietary
Filer view
Confidential and Proprietary

49. Adding Storage to the Database
Step 4 - After new LUNs created on the filer, reboot the database server to automatically discover new LUNS. You can verify that LUNs are accessible from the database server using NetApp host utilities:
Confidential and Proprietary

50. Confidential and Proprietary
Creating ASM Disks
Step 5 – run fdisk on new devices.
Step 6 – run /etc/init.d/oracleasm createdisk.
Last step – connect to ASM instance and execute “create diskgroup…” command. You can also use Enterprise Manager db control:
Confidential and Proprietary

51. Backup and Recovery

52. SnapManager for Oracle Backups
A management tool with a GUI and command-line interface for Oracle Database administrators that simplifies backup, recovery, and cloning for Oracle Databases
Designed to leverage Oracle Database 10g data management and grid features such as ASM, RAC, and RMAN
Utilizes NetApp Snapshot technology to create extremely fast and space-efficient backups
Snapshot copies are point-in-time copies of a database that are created nearly instantaneously.
These backups can also be registered with Oracle RMAN, which facilitates the use of RMAN to restore and recover the database at finer granularities such as blocks.
Confidential and Proprietary

53. Confidential and Proprietary
How SnapManager Works
To create a successful backup, SnapManager performs the following operations:
Determines the list of data files, control files, and log files that make up the database. These files can be located on one or more ASM disk groups.
Puts the database in hot backup mode
Creates an atomic Snapshot copy of all the filer volumes that make up the ASM disk group
Ends the hot backup mode
Clones the Snapshot copy and then renames the cloned ASM disk group
Verifies the backup
Registers the cloned ASM disk group and storage information in the RMAN repository
Confidential and Proprietary

54. Disk Backup vs. SnapManager Backup
To demonstrate the benefit of snapshot backup technology we measured the time taken to backup 70GB database to the disk. Here’s what we found:
Backup to the disk:
Backup set complete, elapsed time: 00:25:20
It would take about 18 hours to backup 3 TB database.
Confidential and Proprietary

55. Time Log for SnapManager Backup
:43:45,396 [INFO ]: SMO-07100: Placing database into online backup mode.
:46:00,724 [DEBUG]: SMO-12000: Executing SnapDriveCommand
:46:04,762 [DEBUG]:SMO-12001: Result SnapDriveResult (0:00:04.037) (took 4 seconds to take a snapshot of the volume)
:55:55,841 [INFO ]: SMO-13037: Successfully completed operation: Backup
:55:55,947 [INFO ]: SMO-13048: Operation Status: SUCCESS
:55:56,030 [INFO ]: SMO-13049: Elapsed Time: 0:12: (including all backup management procedures)
From database alter log:
Fri Nov 10 17:43: ALTER DATABASE BEGIN BACKUP
Fri Nov 10 17:46: ALTER DATABASE END BACKUP
Confidential and Proprietary

56. Confidential and Proprietary
Restore Operation
Similar to backups, SnapManager for Oracle relieves the DBA from having to worry about the underlying layout of data in terms of storage subsystems, disks, host volumes, or host file systems.
DBAs just choose the backup they want to restore from, and SnapManager does the rest.
DBAs can also specify the date and time or the Database SCN to which they would like the database restored.
Confidential and Proprietary

57. Thank you Hanan Hit, Principal Database Architect
Lina Shabelsky, Senior Application and Database Engineer