Presentation is loading. Please wait.

Presentation is loading. Please wait.

Conditions Data access using FroNTier Squid cache Server

Similar presentations


Presentation on theme: "Conditions Data access using FroNTier Squid cache Server"— Presentation transcript:

1 Conditions Data access using FroNTier Squid cache Server
dimanche 9 septembre 2018dimanche 9 septembre 2018 Conditions Data access using FroNTier Squid cache Server Farida Fassi LCG-T1 & AF

2 Outline Motivation Implementation Overview Deployment
Squid Hardware details

3 Motivation What Condition data are?
All parameters describing the detector run condition such as calibration, alignment and configuration info Data needed for Offline detector event data processing and reconstruction Vary with time and may exist in several versions A given object may be used by thousands of jobs Caching such info close to the processing activity (CMS Software application) provides significant performance gains

4 CMS Software Stack: CMSSW overview
User Application EDM EventSetup assures the correct Condition Data is accessed and available for the user application POOL-ORA (Object Relational Access) is used to map C++ objects to Relational schema A CORAL-Frontier plugin provides read-only access to the POOL DB objects via Frontier. EDM Framework EDM EventSetup POOL-ORA CORAL Oracle Frontier Frontier/Cache Database

5 Implementation: Overview
Pool and CORAL generate SQL queries from the CMSSW C++ objects The FroNTier client converts the SQL into an HTTP GET and sends it over the network to the FroNTier server The FroNTier server (a servlet under Tomcat) unpacks the SQL request, sends it to the DB server, and retrieves the needed data and then packed into an HTTP formatted stream sent back to the client. Having a Squid caching server between the FroNTier server and client caches requested objects, improving significantly performance and reducing load on the DB

6 Deployment Clients: WNs
Redundant Tomcat + Squid servers are deployed at Tier 0 (launchpad). Squids are deployed at Tier1, and TierN sites Applications needing access to conditions data are configured with a list of servers, and proxies, for example: FRONTIER_SERVER1= export FRONTIER_SERVER2= export FRONTIER_PROXY1= domain.net:10000 Tier N Squid Squid Squid Tier 1 Squid Squid Squid Squid(s) Tomcat(s) Tier 0 Launchpad DB

7 Marc Paterno, CHEP 2004 T1/T2: Squid T0: Squid T0: Tomcat T0: Oracle

8 Deployment Conditions data access via Frontier already successfully tested in CSA 2006. Many performance improvements since that time (improve cache reuse, fix many problem)

9 Cache Coherency issue CMS adapted the following policy to deal with CC: All objects that are cached have expiration times Short expiration time: metadata (Tag and IOV info) Long expiration time: data payloads (condition data) The values of the short and long times are adjusted according to where the data is being used. For example: Tier 0: calibrations change on the order of a few hours as new runs appear for reconstruction. Tier 1 +: Conditions data may be stable for weeks. The value of the short and long expirations is modified at the central FroNTier server, so they can easily be tuned as needed. For example short ~day, long ~month.

10 Service Availability Monitoring (SAM)
Available from Frontier twiki page: Heartbeat monitoring tests are run every 2 hours, 40 sites included Check integrity of local squid, and Overall Frontier system and connection to CERN

11 Local site Squid Hardware details
Minimum specs: 1GHz CPU, 1GByte mem, GBit network, 100 GB disk Multi-core machines: CPU resources under utilized because of single Squid process utilizes only one CPU Multiple network interfaces can resolve this; two solution tested Multi-homed (2 ip addresses): machine looks like multiple nodes Bonded interfaces (1 ip address): multiple NICs used together to increase throughput. Bonded approach implemented at Fermilab (FNAL) Works best with specific load balancing approach Requires 2 squids on same server machine, Two Gbit network interface ~200 MBps throughout can be achieved Many sites prefer just adding more machines to make network management simpler and reduce configuration issues


Download ppt "Conditions Data access using FroNTier Squid cache Server"

Similar presentations


Ads by Google