1. LCG 3D and Oracle Cluster
Storage Workshop
20-21 Mar 2006
Barbara Martelli, INFN - CNAF
Gianluca Peco, INFN Bologna

2. Overview LCG 3D project LCG 3D deployment plans
Oracle technologies deployed at CNAF
Experience with Oracle Real Application Clusters
RAC tests results

3. LCG 3D Goals
LCG 3D is a joint project between the service providers (CERN and LCG sites) and the service users (experiments and grid projects) aimed at:
Defining distributed database services and application access allowing LCG applications and services to find relevant database back-ends, authenticate and use the provided data in a location independent way.
Helping to avoid the costly parallel development of data distribution, backup and high availability mechanisms in each experiment or grid site in order to limit the support costs.
Enabling a distributed deployment of an LCG database infrastructure with a minimal number of LCG database administration personnel.

4. LCG 3D Non Goals Store all database data
Experiments are free to deploy databases and replicate data under their responsibility
Setup a single monolithic distributed database system
Given constraints like WAN connections one can not assume that a single synchronously updated database would work or give sufficient availability
Setup a single vendor system
Technology independence and multi-vendor implementation will be required to minimize the long term risks and to adapt to the different requirements/constraints on different tiers.

5. LCG 3D Project Structure
WP1 - Data Inventory and Application requirements Working Group
Members are s/w providers from experiments and grid services based on RDBMS data.
Gather data properties (volume, ownership, access patterns) requirements and integrate the provided service into their software.
WP2 - Service Definition and Implementation Working Group
Members are site technology and deployment experts.
Propose an agreeable deployment setup and common deployment procedures, deploy the DB service according to 3D agreed policies.
CNAF involvement

6. Proposed Service ArchitectureM M O T0
- autonomous
T3/4
T1- db back bone
- all data replicated
- reliable service
T2 - local db cache
-subset data
-only local service O M O M
Oracle Streams
Cross vendor extract
MySQL Files
Proxy Cache
Slide by Dirk Duellmann, CERN-IT

7. Proposed Service Structure
3 separate environments with different service levels and different HW resources:
Development environment:
Shared HW setup
DBA limited support (via )
8/5 monitoring and availability
Integration:
Dedicated node for a defined slot (usually a week) to perfom performance and functionality tests
DBA support via or phone
Production:
24/7 monitoring and availability 
Backups every 10 minutes
Limited number and scheduled number interventions

8. Present CNAF DB Service Structure
3D proposed service policy will be met by steps:
At present we have:
Test environment (used for testing new Oracle based technologies such as RAC, Grid Control and Streams)
Preproduction environment composed by
2 dual RAC with 12 FC each of shared storage allocated to LHCb and Atlas
1 HP Proliant DL380G4 for Service instances such as Castor2 stager and FTS.
By the end of April preproduction environment will be moved to production.
When RAC tests will be finished, test machines will become our development/integration environment.

9. 3D Milestones
Tier-1 services starts - milestone for earlly production Tier 1 sites
CNAF will start with 2 dual RAC (LHCb,ATLAS)
Bookkeeping DB replica
LFC file catalog replica
VOMS replica
Service review workshop >> hardware defined for full production.
Experiment and site reports after first 3 month of service deployment.
Define db resource requirements for full service. Milestone for experiments and all tier 1 sites
Full LCG database service in place - milestone for all Tier 1 sites

10. 3D – Oracle Technologies
3D project leverages on several Oracle technologies to guarantee scalability and reliability of provided DB services:
Oracle Streams Replication
Oracle Real Application Clusters
Oracle Enterprise Manager / Grid Control

11. Oracle Streams Replication
Consumption
Staging
Capture
Streams captures events
Implicitly: log-based capture of DML and DDL
Explicitly: Direct enqueue of user messages
Captured events are published in the staging area
Streams publishes captured events into a staging area
Implemented as a queue
Messages remain in staging area until consumed by all subscribers
Other staging areas can subscribe to events in same database or in a remote database
Events can be routed through a series of staging areas

12. Staged events are consumed by subscribers Implicitly: Apply Process
Consumption
Staging
Capture
Transformations can be performed as events enter, leave or propagate between staging areas
Staged events are consumed by subscribers
Implicitly: Apply Process
Default Apply
User-Defined Apply
Explicitly: Application dequeue via API (C++, Java…)
The default apply engine will directly apply the DML or DDL represented in the LCR
apply to local Oracle table
apply via DB Link to non-Oracle table
Automatic conflict detection with optional resolution
unresolved conflicts placed in exception queue
Rule based configuration: expressed as “WHERE” clause

13. Example of Streams Replication
User executes an update statement at source node:
update table1 set field1= ‘value3’ where table1id = ‘id1’;
Update table1 set field1=‘value3’ where table1id=‘id1’;
table1
table1id |field1|..
id1 | value3 |…
id2 | value2 | ...
Apply
Queue
-----
LCRs
------
Propagation
ACK
table1
Source Node
Capture
Destination Node
Redo Log

14. Oracle Real Application Cluster
The Oracle Real Application Cluster technology allows to share a database amongst several database servers
All datafiles, control files, PFILEs, and redo log files in RAC environments must reside on cluster-aware shared disks so that all of the cluster database instances can access them.
RAC aims to provide highly available, fault tolerant and scalable database services
Database servers
Network shared disks (Cluster Filesystem)

15. Shared Storage on RAC
Various choices for shared storage management in RAC:
Raw Devices
Network FS
Supported only on certificated devices
Oracle Cluster File System v2
Posix Compliant, general purpose (can be used also for Oracle Homes)
Automatically configured to use Direct I/O
Enable async I/O by setting filesystemio_options = SETALL
Automatic Storage Management
Logical Volume Manager with striping and mirroring
Dynamic data distribution within and between storage arrays

16. Preproduction Environment Setup
Gigabit Switch
Gigabit Switch
Private LHCB link
Private LHCB link
Private ATLS link
Private ATLS link
Dual Xeon 3,2GHz
4GB memory
2x73GB disks in RAID1
Aggiungere l’hp proliant con castor 2 e la slide sul backup
rac-lhcb-02
rac-atlas-02
rac-lhcb-01
rac-atlas-01
Dell 224F
14 x 73GB disks
Dell 224F
14 x 73GB disks

17. 1.2 TB RAID-5 disk array formatted with OCFS2
RAC testbed
Disk I/O traffic
Fiber Channel Sw
ORA-RAC-01
GigaSw1
ORA-RAC-02
Private network for interconnect traffic
ORA-RAC-03
IBM FAStT900
FC RAID Controller
ORA-RAC-04
1.2 TB RAID-5 disk array formatted with OCFS2
4 x Dual Xeon 2.8 GHz
4 GB RAM
Red Hat Enterprise 4 on RAID-1 disks
2 x Intel PRO1000 NICs
1 QLogic 2312 FC HBA with 2 x 2Gb/s links
Public and VIP Network Interface
GigaSw2
Clients
Clients
Clients

18. RAC Test AS3AP 1-4 nodes
Select Query 1GB cache

19. Select Query 8GB no db cache
RAC Test AS3AP 1-4 nodes
Overview
Summarize the main plans
Explain the long-term course to follow
Select Query 8GB no db cache

20. RAC Test OLTP 4 nodes

21. RAC Test OLTP nodes
Con una applicazione OLTP la scalabilita' del sistema e' meno evidente Probabilmente il sistema shared disks utilizzato non e' adeguato alla tipologia di accesso ai dati da parte dell'applicazione.

22. RAC Test OLTP 4 nodes
TransactionPerMinute con workload OLTP ( Order Entry) O_DIRECT Abilitato ASYNC_IO Abilitato
TransactionPerMinute con workload OLTP ( Order Entry) O_DIRECT Disabilitato ASYNC_IO Disabilitato