1. Replication using Oracle Streams at CERN
Eva Dafonte Pérez

2. Outline Scope Oracle Streams Replication Downstream Capture
Streams Setups at CERN
Split & Merge Procedure
Streams Optimizations
Lessons Learned
Streams Monitoring
Examples and Numbers
Summary 2

3. The LHC Computing Challenge
Data volume
high rate x large number of channels x 4 experiments
15 PetaBytes of new data each year stored
much more data discarded during multi-level filtering before storage
Compute power
event complexity x Nb. events x thousands users
100 k of today's fastest CPUs
Worldwide analysis & funding
computing funding locally in major regions & countries
efficient analysis everywhere
GRID technology
The European research centre CERN currently builds the Large Hadron Collider (LHC). When it begins operations, it will produce roughly 15 Petabytes of data annually, which thousands of scientists around the world will access and analyze.
All data need to be available over the 15 year estimated lifetime of the LHC. The analysis of the data, including comparison with theoretical simulations, requires of the order of CPUs of processing power.
It uses a distributed model for data storage and analysis - computing and data GRID - where:
the costs of maintaining and upgrading the necessary resources for such a computing challenge are more easily handled, because individual institutes and participating organizations can fund local computing resources, while contributing to the global goal.
And, it reduces the points of failure. Multiple copies of data and automatic reassigning of computational tasks to available resources ensures the load balancing resources and facilitates the access to the data for all scientists involved, independently of the geographical location. 3

4. Distributed Service Architecture
The mission of the Worldwide LHC Computing Grid Project (WLCG) is to develop, build and maintain a computing infrastructure for storage and analysis of the LHC data. Data will be distributed around the globe according to a four -tiered model. A primary backup will be recorded at CERN, the "Tier-0" centre of WLCG. After initial processing, these data will be distributed to a series of Tier-1 centers, large computer centers with sufficient compute and storage capacities as well as round-the-clock support for Grid operations.
The Tier-1 centers will perform recurrent reconstruction passes and make data available to Tier-2 centers, each consisting of one or several collaborating computing facilities, which can store sufficient data and provide adequate computing power for specific analysis tasks. Individual scientists will access these facilities through Tier-3 computing resources, which can consist of local clusters in a University Department or even individual PCs. 4

5. Oracle Streams Replication
Technology for sharing information between databases
Database changes captured from the redo-log and propagated asynchronously as Logical Change Records (LCRs)
Apply
Capture
Redo Logs
Propagate
Target Database
Source Database 5

6. Downstream Capture
Downstream capture to de-couple Tier 0 production databases from destination or network problems
source database availability is highest priority
Optimizing redo log retention on downstream database to allow for sufficient re-synchronisation window
we use 5 days retention to avoid tape access
Dump fresh copy of dictionary to redo automatically
10.2 Streams recommendations (metalink note )
Apply
Capture
Propagate
Target Database
Downstream Database
Redo Logs
Source Database
Redo Transport
method 6

7. Streams Setups: ATLAS 7

8. Streams Setups: LHCb (Conditions and LFC)8

9. Split & Merge Procedure
If one site is down
LCRs are not removed from the queue
Capture process might be paused by flow control
 impact on replication performance
Objective: isolate replicas against each other
Split the capture process
(original) Real-Time capture for sites “in good shape”
(new) normal capture for site/s unstable/s
new capture queue and propagation job
original propagation job is dropped
spilled LCRs are dropped from the original capture queue
Merge the capture processes
Resynchronization
suggested by Patricia McElroy ( Principal Product Manager Distributed Systems/Replication) 9

10. Streams Optimizations
TCP and Network tuning
adjust system max TCP buffer (/etc/sysctl.conf)
parameters to reinforce the TCP tuning
DEFAULT_SDU_SIZE=32767
RECV_BUF_SIZE and SEND_BUF_SIZE
Optimal: 3 * Bandwidth Delay Product
Reduce the Oracle Streams acknowledgements
alter system set events '26749 trace name context forever, level 2'; 10

11. Streams Optimizations Rules
ATLAS Streams Replication: filter tables by prefix
Rules on the capture side caused more overhead than on the propagation side
Avoid Oracle Streams complex rules: rules with conditions that include LIKE or NOT clauses or FUNCTIONS
Complex Rule
condition => '( SUBSTR(:ddl.get_object_name(),1,7) IN (''COMP200'', ''OFLP200'', ''CMCP200'', ''TMCP200'', ’'TBDP200'', ''STRM200'')
OR SUBSTR (:ddl.get_base_table_name(),1,7) IN (''COMP200'', ''OFLP200'', ''CMCP200'', ''TMCP200'', ''TBDP200'', ''STRM200'') ) '
Simple Rule
condition => '(((:ddl.get_object_name() >= ''STRM200_A'' and :ddl.get_object_name() <= ''STRM200_Z'') OR
(:ddl.get_base_table_name() >= ''STRM200_A'' and :ddl.get_base_table_name() <= ''STRM200_Z''))
OR ((:ddl.get_object_name() >= ’'OFLP200_A'' and :ddl.get_object_name() <= ''OFLP200_Z'') OR
(:ddl.get_base_table_name() >= ’'OFLP200_A'' and :ddl.get_base_table_name() <= ''OFLP200_Z''))
Avoid complex rules:
LIKE
Functions
NOT 11

12. Streams Optimizations Rules
600 80
Simple Rules
Complex Rules 12

13. Streams Optimizations Flow Control
By default, flow control kicks when the number of messages is larger than the threshold
Buffered publisher: 5000
Capture publisher: 15000
Patch = 2 new events
10867: controls threshold for any buffered message publisher
10868: controls threshold for capture publisher
Manipulate default behavior 13

14. Lessons Learned Apply side can be significantly less efficient
Manipulate the database is slower than the redo generation
Execute LCRs serially => apply cannot keep up with the redo generation rate
SQL bulk operations (at the source db)
May map to many elementary operations at the destination side
Need to control source rates to avoid overloading
System generated names
Do not allow system generated names for constraints and indexes
Modifications will fail at the replicated site
Storage clauses also may cause some issues if the target sites are not identical 14

15. Streams Recommended Patches
Metalink Note: Recommended Patch for Streams
MLR for Streams/Logminer bugs
patches: , and
Leak in Perm Allocations with library cache comments ora-4031 generated
fix patch
ORA [KRVTADC] IN CAPTURE PARALLEL PROCESS (using compressed tables)
fix patch
DEADLOCK BETWEEN 'LIBRARY CACHE LOCK' AND 'LIBRARY CACHE PIN'
fix patch
Bug SQL apply degrades with larger transactions
Bug STREAMS 5000 LCR limit is causing unnecessary FLOW CONTROL at apply site 15

16. Streams Monitoring Features: Architecture:
Streams topology
Status of streams connections
Error notifications
Streams performance (latency, throughput, etc.)
Other resources related to the streams performance (streams pool memory, redo generation)
Architecture:
“strmmon” daemon written in Phython
End-user web application
3D monitoring and alerting integrated with WLCG procedures and tools 16

17. Streams Monitoring 17

18. Examples Streams setup for ATLAS experiment
Online  Offline  Tier1 sites (10)
Real-Time downstream capture
Oracle
Use of rules
Database size: 1.44 TB 18

19. Examples ATLAS Conditions data 2 GB per day 600 - 800 LCRs per second19

20. Examples
ATLAS PVSS tests
6 GB per day
LCRs per second 20

21. Summary
The LCG Database Deployment Project (LCG 3D) has set up a wold-wide distributed database infrastructure for LHC
some 124 RAC nodes = 450 CPU cores at CERN + several tens of nodes at 10 partner sites are in production now
Large scale tests have validated that the experiment are implemented by the RAC & streams based set-up
Backup & recovery tests have been performed to validate the operational procedures at all sites
Monitoring of database & streams performance has been implemented building on grid control and strmmon tools
key to maintain and optimise any larger system
Database infrastructure is ready for accelerator turn-on
collaboration with Oracle as part of the CERN openlab has been essential and beneficial for both partners 21

22. Questions? 22