1. 3D Project Status Report
Maria Girone, CERN IT
on behalf of the LCG 3D project
LHCC Comprehensive Review
19th-20th Nov 2007

2. Introduction
Set up a distributed database infrastructure for the WLCG (according to the WLGC MoU)
RAC as building-block architecture
Several 8-node clusters at Tier0
Typically 2-node clusters at Tier1
Oracle streams replication used to form a database backbone between. In production since April ‘07
Tier0 and 10 Tier1 sites
ATLAS and LHCb
Online and offline at Tier0
ATLAS, CMS and LHCb
Frontier/SQUID from Fermilab for distributing and caching database data via a web protocol is used by CMS
Database services listed among the critical services by the experiments

3. Building Block- Database Clusters
Tier 0: all networking and storage redundant
Scalability and high availability achieved
Storage and CPU scale independently
Maintenance operations w/o down-time!

4. Oracle Streams
ASGC
TRIUMF
New or updated data are detected and queued for transmission to destination databases
Database changes captured from the redo-log and propagated asynchronously as Logical Change Records (LCRs)
All changes are queued until successful application at all destinations
need to control change rate at the source in order to minimise the replication latency
2GB/day user data to Tier 1 can be sustained with the current DB setups

5. Downstream Capture & Network Optimisations
Downstream Database Capture to de-couple Tier0 production databases from Tier1 or network problems is in place for ATLAS and LHCb
TCP and Oracle protocol optimisations yielded significant throughput improvements (almost factor 10)

6. Frontier/Squid
Numerous significant performance improvements since CSA’06
experiment data model
frontier client/server software
CORAL integration
CMS is confident that possible cache coherency issues can be avoided by
Cache expiration windows for data and meta-data
Policy implemented by the client applications
Successfully used also in CSA’07
database queries encoded as URL requests and the corresponding query results transferred as html pages.
The use of http and html enables the use of standard web cache servers such as SQUID
Experiment data model resulting sometimes in too many queries addressed but CMS changing the data model
Problems seen recently for POOL-ORA caching container meta-data

7. Streams Replication Operation
Streams procedures included in the Oracle Tier0 physics database service team
Optimized the redo log retention on downstream database to allow for sufficient re- synchronisation window without recall from tape (for 5 days)
Preparing a review of the most recurrent problems for the WLCG Service Reliability Workshop, Nov 2007
Will be the input for further automation of service procedures

8. Some Issues Several bugs reported to Oracle fixed or being fixed
CERN openlab has set an excellent ground!
Some user-defined types not supported by streams
Reformatting is done with filtering rules
LFC has been hit by a known Oracle bug as it was running on on a older instant client version
Syncronization with the AA instant client established
Set up read-only replicas on other 5 sites together with code change to support user ACLs
Automate more the split-merge procedure when one site has to be dropped/re-syncronized
Progress in Oracle 11g but we need a stop-gap solution
Implement failover for the downstream capture component
ACL

9. Streams problems Workarounds found for all the open bugs:
Capture process is aborted with error ORA-01280: Fatal LogMiner
Fixed in patches and related to the capture process and the logminer
Bug SQL apply degrades with larger transactions
Fixed applying patch
Bug STREAMS 5000 LCR limit is causing unnecessary FLOW CONTROL at apply site
Fixed applying patch
Bug recyclebin=on; after child table dropped:ORA on parent
Metalink note
Fixed on and 11g
APPLY gets stuck: APPLY SERVER WAITING FOR EVENT
Generic performance issue on RAC
Fixed by applying patches for the fix for <bug: > and One off/Patch of
OEM agent blocks Streams processes
Fixed applying patch
Workarounds found for all the open bugs:
ORA-600 [KWQBMCRCPTS101] after dropping Propagation job
ORA after table is dropped when there are two streams setups between the same source and destination databases
ORA-00600: [KWQPCBK179], memory leak from propagation job
Observed after applying the fix patch
For the non-fixed problems:
ORA-600 [KWQBMCRCPTS101] after dropping Propagation job
ORA after table is dropped when there are two streams setups between the same source and destination databases
we have developed and implemented a workaround for each problem
ORA-00600: [KWQPCBK179], memory leak from propagation job
this is not a critical problem 9

10. Service Levels and Policies
DB Service level according to WLCG MoU
need more production experience to confirm manpower coverage at all T1 sites
piquet service being set-up at Tier 0 to replace existing 24x7 (best effort) service
streams interventions for now 8x5
Criticality of this service rated by most experiments (ATLAS, CMS and LHCb) as “very high” or “high”
Proposals from CERN Tier 0 have been accepted also by the collaborating Tier 1 sites
Backup and Recovery
RMAN based backups mandatory, data retention period 1 month
security patch frequency and application procedure
database software upgrade procedure
patch validation window

11. Database & Streams Monitoring
Monitoring and diagnostics has been extended and integrated in the experiments’ dashboards
Weekly/monthly database and replication performance summary has been added
extensive data and plots about replication activity, server usage, server availability available form the 3D wiki site (html or pdf)
summary plot with LCR rates during last week is show on the 3D home page and could be referenced/included into other dashboard pages
Complemented by weekly Tier 0 database usage reports which are in use already since more than one year

12. Integration with WLCG Procedures and Tools
3D monitoring and alerting has been integrated with WLCG procedures and tools
dedicated workshop at SARA in March focussing on this
Interventions announced according to the established WLCG procedures
eg EGEE broadcasts, GOCDB entries
To help reporting to the various coordination meetings we collect all 3D intervention plans also on the 3D wiki
Web based registration will be replaced as soon as a common intervention registry is in production

13. Intervention & Streams Reports
A factor 5 improvement in apply
speed by tuning of filtering rules

14. Tier 1 DB Scalability Tests
The experiments have started to evaluate/confirm also the estimated size of the server resources at Tier 1
number of DB nodes CPUs
memory, network and storage configuration
Need realistic work-load which now becomes available as experiment s/w frameworks approach complete coverage of their detectors
ATLAS conducted two larger tests with ATHENA jobs against IN2P3 (shared solaris server) and CNAF (Linux)
Total throughput of several thousand jobs/h achieved with some 50 concurrent test jobs per Tier 1
LHCb s/w framework integration done and scalability tests starting as well
Lower throughput requirements assumed than for ATLAS
Tests with several hundred concurrent jobs in progress

15. Database Resource Requests
Experiment resource requests for T1 unchanged since more than one year
2 (3) node DB cluster for LHCb (ATLAS)
fibre channel based shared storage
2 squid nodes for CMS
standard worker node with local storage
Setup shown to sustain replication throughput required for conditions data (1.7 GB/d ATLAS)
ATLAS has switched to production with the Tier1 sites
LHCb requested read-only LFC replicas at all 6 Tier1 sites as for conditions
Successful replication tests for LFC between CERN and CNAF
Updated requests for LHC startup have been collected at the WLCG workshop at CHEP’07
No major h/w extension requested at that time

16. Online-Offline Replication
Oracle Streams used by ATLAS, CMS and LHCb
Joint effort with ATLAS on replication of PVSS data between online and offline
required to allow detector groups to analyse detailed PVSS logs without adverse impact on online database
Significantly higher rates required than for COOL based conditions, which have been confirmed in extensive tests
some 6 GB of user data per day
Oracle Streams seems an appropriate technology also for this area

17. Backup & Recovery Exercise
Organised a dedicated database workshop at CNAF in June 2007 on recovery exercise
show that database implementation and procedures at each site are working
show that coordination and re-synchronisation after a site recovery works
show that replication procedures continue unaffected while some other sites are under recovery
Exercise well appreciated by all participants
several set-up problems haven been resolved during this hands on activity with all sites present
six sites have now successfully completed a full local recovery and re-synchronisation
remaining sites will be scheduled shortly after the Service Reliability Workshop

18. More Details LCG 3D wiki Recent LCG 3D workshops
interventions, performance summaries
Recent LCG 3D workshops
Monitoring and Service Procedures SARA
Backup and Recovery CNAF
LGC Service Reliability Workshop
w=standard&confId=20080

19. Approved by the MB on 13.03.07 TAGS at volunteer Tier-1 sites
(BNL, TRIUMF, ...)

20. Updated Request from ATLAS
Not yet approved. May need funding discussion
ATLAS provides more info at
Year
Total (TB)
TAGS (TB)
COOL (TB)
PVSS etc. (TB)
2007
6.4
1.0
0.4 5
2008
14.5
5.6
0.9 8
2009
27.3
13.7
1.6 12
2010
46.8
27.5
2.9
16.4 20

21. Approved by the MB on

22. Summary
The LCG 3D project has setup a world-wide distributed database infrastructure for LHC
Close collaboration between LHC experiments and LCG sites
with more than 100 DB nodes at CERN + several tens of nodes at Tier 1 sites this is one of the largest distributed database deployments world-wide
Large scale experiment tests have validated the experiment resource requests implemented by the sites
Backup & recovery tests have been performed to validate the operational procedures for error recovery
Regular monitoring of the database and streams performance is available to the experiments and sites
Tier 0+1 ready for experiment ramp-up to LHC production. Next steps:
Replication of ATLAS Muon calibrations back to CERN
Test as part of CCRC’08
Started testing 11gR1 features. Production deployment plan for 11gR2 by end of 2008