1. Evolution of Data(base) Replication Technologies for WLCG
Zbigniew Baranowski, CERN IT-DB
CHEP2015
Okinawa 13th of April, 2015

2. Outline Why database replication for WLCG?
Replication technology evolution
Deployment
Conclusions

3. Data replication for online DBs
Key component of online-offline DB model for experiments database services
Conditions data
Controls and aquisition system archives (WinCC/PVSS)
Archiving
Replication
Online
Database
Offline
Database

4. Data replication for WLCG
World wide data distribution for WLCG
Conditions data to T1s (ATLAS, LHCb – in past)
(In past) LHCb LFC data to T1s
Consolidation of data at T0 (ATLAS)
ATLAS Metadata Interface
Muon calibration data
Replication
Offline
Databases
ATLAS conditions are replication
to a subset of Tier-1 sites

5. First implementation of replication
Technology for replication: Oracle Streams
Online – Offline replication
Offline – T1s replication
Oracle Streams architecture

6. Oracle Streams + vendor-provided solution
Support and evolution provided by Oracle
+ data filtering during extraction
No high bandwidth network needed
- coherency of entire data set is not ensured
- more replica databases impose a higher overhead on a source database

7. Reasons for evolution
Considerable effort to support and maintain Oracle Streams
Reduce the number of incidents and replica downtime
Streams does not support some data operations
Oracle introduces new replication technologies
Active Data Guard – in Oracle 11.2 (2009)
Oracle GoldenGate – in 2010
Streams is deprecated by Oracle
Oracle recommended log-based replication technology is now GoldenGate (2010)

8. Physical Standby Database
Active Data Guard
Physical Standby Database
Source Database
Redo Logs
Redo transport
Recovery
Standby
Redo
Logs

9. Active Data Guard + data consistency guaranteed
+ little maintenance effort
+ fast switchover/failover solution
o replica is in read-only mode
- default granularity is full DB replication
- no data filtering (entire data stream has to be transferred), problematic for WAN
- same database version needed for primary and replica systems

10. Oracle GoldenGate Extract Replicat Datapump Source Database
Target Database
Redo Logs
Extract
Replicat
Datapump
Trail Files
Trail Files

11. Oracle GoldenGate + improved version of Streams
better performance
lower footprint on the source database
more functionalities, data types and features
+ destination DB can be at a different version than source
+ replication granularity is at schema-level
+ installation can be centralized
- additional binaries has to be installed
- coherency of entire data set is not ensured

12. Evaluation - performance
In 2011:
-> Active Data Guard performs the best
-> Golden Gate the worst
HIGHER - BETTER
In 2013:
-> new version of GoldenGate (12c) beats
Streams
IMPROVEMENT

13. Deployment
2012: first deployment of online – offline replication based on Active Data Guard
ADG became possible with the DB upgrades from version 10g to 11g
ALICE: Q2 2012, CMS: Q2 2012
2014: replication based on GoldenGate
Selective data replication
Centralized system – for multiple replica topologies
ATLAS Q3-Q (online-offline-T1s)
LHCb: Q (online-offline)

14. ATLAS DB replication evolution 2014/2015
Downstream
Capture
Database
Conditions
GoldenGate
Online
Database
Offline
Database
Conditions
GoldenGate
REDO
Conditions
GoldenGate
PVSS
Conditions
GoldenGate
ADG
Custom
Custom
Custom
GoldenGate
Online
Physical
Standby
UMICH
(USA)
ROME
(ITALY)
MUNICH
(GERMANY)
IN2P3 AMIDB
(FRANCE)

15. Timeline 2004 2008 2010 2012 2014/2015 Oracle Streams evaluation
Online – Offline Offline – T1s
Offline – T1s
Oracle Streams in production
Oracle Streams in production
Oracle Streams in production
GoldenGate in production
Oracle Streams evaluation
GoldenGate evaluation
GoldenGate evaluation
Active Data Guard evaluation
Active Data Guard in production
Active Data Guard evaluation
Active Data Guard in production
Initial functional and
performance tests
New technology
available in Oracle 11g
Oracle buys/adopts
GoldenGate
Deployment of
Oracle 11g at CERN
and T1s
Online – Offline

16. Conclusions
Database replication is key technology to enable distribution of conditions data across T0 (and T1s)
Complex: different requirements, different topologies
Technology evolution:
Oracle Streams (initial solution) was replaced by Active Data Guard (T0) and Golden Gate (T0 and T1s)
The change improved availability and performance of the data replication services

17. Acknowledgments
Collaboration has been key for the success of this project.
Experiments
Many thanks to the experiments DB coordinators and experts
ATLAS: Gancho Dimitrov, Dario Barberis, Elizabeth Gallas, Rainer Bartoldus, Andrea Formica; ALICE: Peter Chochula; CMS: Andreas Pfeiffer, Giacomo Govi; LHCb: Marco Clemencic, Niko Neufeld
Tier 1 DBAs and database experts
Osman Aidel, Carmine Cioffi, Andrew Wong, Carlos Gamboa
Oracle (via the Openlab partnership)
Patricia McElroy, Jagdev Dhillon, Greg Doherty, Monica Marinucci
CERN IT-DB group
Especially: Eva Dafonte Perez, Lorena Lobato Pardavila
Former project members:
Maria Girone, Dirk Duellmann