1. Oracle Clustering and Replication Technologies UK Metadata Workshop - Oxford Barbara Martelli Gianluca Peco

2. 5 July, 2006UK Metadata Workshop, Oxford2 Overview  Oracle server architecture  Oracle Real Application clusters architecture and tests  Oracle Streams technology  LFC replcation tests done with LHCb team

3. 5 July, 2006UK Metadata Workshop, Oxford3 Oracle Database Architecture The Oracle Server architecture can be divided in three categories:  User-related processes User Process Server Process  Logical memory structures that are collectively called an Oracle instance  Physical file structures that are collectively called a database Database

4. 5 July, 2006UK Metadata Workshop, Oxford4

5. 5 July, 2006UK Metadata Workshop, Oxford5 Instance Database

6. 5 July, 2006UK Metadata Workshop, Oxford6 Oracle Real Application Cluster  The Oracle Real Application Cluster technology allows to share a database amongst several servers  All datafiles, control files, PFILEs, and redo log files in RAC environments must reside on cluster- aware shared disks so that all of the cluster database instances can access them.  RAC aims to provide highly available, fault tolerant and scalable database services Network shared disks (Cluster Filesystem) Database servers

7. 5 July, 2006UK Metadata Workshop, Oxford7 RAC testbed ORA-RAC-01 ORA-RAC-02 ORA-RAC-03 ORA-RAC-04 IBM FAStT900 FC RAID Controller Fiber Channel Sw GigaSw2 Clients GigaSw1 Private network for interconnect traffic Public and VIP Network Interface 4 x Dual Xeon 2.8 GHz 4 GB RAM Red Hat Enterprise 4 on RAID-1 disks 2 x Intel PRO1000 NICs 1 QLogic 2312 FC HBA with 2 x 2Gb/s links Clients Disk I/O traffic 1.2 TB RAID-5 disk array formatted with OCFS2

8. RAC Test AS3AP 1-4 nodes Select Query 1GB cache

9. 5 July, 2006UK Metadata Workshop, Oxford9 Overview  Summarize the main plans  Explain the long-term course to follow Select Query 8GB no db cache RAC Test AS3AP 1-4 nodes

10. 5 July, 2006UK Metadata Workshop, Oxford10 RAC Test OLTP 1-2-4 nodes With OLTP applications, system scalability is lower, we argue there is a disk subsystem bottleneck 1 node 2 nodes 4 nodes

11. 5 July, 2006UK Metadata Workshop, Oxford11 RAC Test OLTP 4 nodes TransactionPerMinute workload OLTP O_DIRECT enabled ASYNC_IO enabled TransactionPerMinute workload OLTP O_DIRECT Disabled ASYNC_IO Disabled

12. 5 July, 2006UK Metadata Workshop, Oxford12 Oracle Streams CAPTURE:  Streams captures events Implicitly: log-based capture of DML and DDL Explicitly: Direct enqueue of user messages PROPAGATION:  Captured events are published in the staging area  The staging area has the following characteristics: Implemented as a queue Messages remain in staging area until consumed by all subscribers  Other staging areas can subscribe to events in same database or in a remote database  Events can be routed through a series of staging areas  Transformations can be performed as events enter, leave or propagate between staging areas Consumption PropagateCapture

13. 5 July, 2006UK Metadata Workshop, Oxford13 Oracle Streams Comsumption:  Staged events are consumed by subscribers Implicitly: Apply Process  Default Apply  User-Defined Apply Explicitly: Application dequeue via API (C++, Java…)  The default apply engine will directly apply the DML or DDL represented in the LCR apply to local Oracle table apply via DB Link to remote Oracle table  Automatic conflict detection with optional resolution unresolved conflicts placed in exception queue  Rule based configuration: expressed as “WHERE” clause Consumption PropagateCapture

14. 5 July, 2006UK Metadata Workshop, Oxford14 Streams Replication Example table1 Update table1 set field1=‘value3’ where table1id=‘id1’; Redo Log Capture table1 table1id |field1|.. id1 | value3 |… id2 | value2 |... Apply Queue ----- LCRs Queue ------ LCRs Propagation ACK Source Node Destination Node User executes an update statement at source node: update table1 set field1= ‘id3’ where table1id = ‘id1’;

15. 5 July, 2006UK Metadata Workshop, Oxford15

16. 5 July, 2006UK Metadata Workshop, Oxford16 Oracle Streams in 3D  The Oracle streams allows connecting single tables or complete schemas in different databases and keeping them up to date at Real Time.

17. 5 July, 2006UK Metadata Workshop, Oxford17

18. 5 July, 2006UK Metadata Workshop, Oxford18

19. 5 July, 2006UK Metadata Workshop, Oxford19

20. 5 July, 2006UK Metadata Workshop, Oxford20

21. 5 July, 2006UK Metadata Workshop, Oxford21 LFC Replication testbed  40 lfc clients, 40 lfc daemons threads, streams pool.  Client’s actions Control if LFN exists into the database  Select from cns_file_metadata If yes -> add a sfn for that lfn  Insert sfn into cns_file_replica If not -> add both lfn and sfn  Insert lfn into cns_file_metadata  Insert sfn into cns_file_replica For each lfn 3 sfn are inserted

22. 5 July, 2006UK Metadata Workshop, Oxford22 LFC Master HW Configuration Gigabit Switch Private LHCB link rac-lhcb-01 rac-lhcb-02 Dell 224F 14 x 73GB disks ASM Dual Xeon 3,2GHz,4GB memory 2nodes-RAC on Oracle 10gR2 RHEL 4 kernel 2.6.9-34.ELsmp 14 Fibre Channel disks (73GB each) HBA Qlogic Qla2340 – Brocade FC Switch Disk storage managed with Oracle ASM (striping and mirroring)

23. 5 July, 2006UK Metadata Workshop, Oxford23 LFC Slave Configuration  LFC Read only replica Dual Xeon 2.4, 2GB RAM Oracle 10gR2 (oracle RAC but used as single instance) RHEL 3 kernel 2.4.21 6 x 250GB disks in RAID 5 HBA Qlogic Qla2340 – Brocade FC Switch Disk storage formatted with OCFS2

24. 5 July, 2006UK Metadata Workshop, Oxford24 Performance About 75 transactions per second on each cluster node. Inserted and replicated 1700k entries in 4 hours (118 insert per second). Almost real-time replica with Oracle Streams without significant delays (<< 1s).

25. 5 July, 2006UK Metadata Workshop, Oxford25 CPU load on cluster nodes is far from being saturated.

26. 5 July, 2006UK Metadata Workshop, Oxford26 Conclusions and Future Plans  RAC technology is a good solution for scalability at DB server level. Some work is needed to tune the installation and optimize performance for a particular application. Moreover a reliable and scalable storage subsystem is needed.  Streams based replication is a good solution for scalability at “grid level”, a reliable DB infrastructure has to be distributed across many sites.  First LFC replication test results demonstrate that Streams is an interesting solution for real-time master/slave replication.  VOMS replication tests in the very near future.  Many thanks to Vincenzo Vagnoni, Eva da Fonte Perez.