RUCIO Consistency Status Sarah Williams Indiana University 2015-03-23

Making Consistency Checks with RUCIO Previous version of CCC (Complete Consistency Check) using DQ2 api is still usable. However, changes brought by RUCIO have side-effects: Queries for a list of datasets at an endpoint returns only complete replicas, not incompletes. The command to list files in a dataset is slow. We have a working version of CCC using the RUCIO API. RUCIO developers would prefer that we not query each dataset to produce a list of files, as in the previous version. They are concerned that if multiple sites run the tool at once, it could cause load issues. They have agreed to produce a dump file with a list of datasets and files per dataset.

RUCIO Dump Files Two types of dump files are currently available: A list of files per endpoint. A list of datasets per endpoint A dump of files per dataset is still pending. Request for the dumps was made mid-December. ETA was a few weeks. Timeline for producing this dump has been pushed back several times. Latest news: “Waiting on non trivial changes to Rucio that unfortunately take some time. The current time frame is end of April (after CHEP)”

RUCIO rse-audit vs CCC rucio rse-audit compares the list of files per endpoint to a list of files on disk. It knows nothing about datasets CCC fetches a list of datasets, then a list of files per dataset, and compares that to a list of files on disk. Due to the change in the API, it knows nothing about incomplete datasets. When a dataset is no longer listed as having a replica at a site, the files are not deleted right away. They are instead recorded as still being at the site, and are candidates for deletion if space is needed. CCC also does not know about these files. Therefore, these report different numbers for dark data. At present the rucio rse-audit numbers are more accurate.

RUCIO rse-audit vs CCC RSE RUCIO dark file # RUCIO bytes CCC dark file # CCC Bytes MWT2_UC_PERF-JETS 200 25.98G 27.89G MWT2_UC_LOCALGROUPDISK 13.64K 730G 251.9K 13.39T MWT2_UC_TRIG-DAQ 0B 115 21.88G MWT2_UC_PRODDISK 155.4K 30.66T 378K 102.5T MWT2_UC_SCRATCHDISK 222 75.71G 232 82.3G MWT2_DATADISK 10.4K 16.86T 25.69K 53.33T MWT2_UC_USERDISK 496.4K 18.63T 2.156M 55.75T MWT2_UC_PHYS-TOP 1 183.3M MWT2_UC_PHYS-HIGGS 376 288.6G 309.9G MWT2_UC_PERF-TAU 38 117.5G 126.2G Totals 676.7K 67.36T 2.812M 225.6T

Other Monitors: rucio rse-usage and DDM Accounting Visualization RUCIO reports RSE used space as the files written to the endpoint + the files scheduled to be written. With this information a user can decide if there is enough space to subscribe a dataset to an endpoint. But, this misleading to a site manager, who wants to know how much data RUCIO thinks is at the site. The following plot from DDM monitoring compares used space used in MWT2_DATADISK as reported by SRM (green dotted line) to RUCIO (red bars). Because the amount of used data RUCIO reports is higher, it looks like there is missing data, but in fact that is the subscribed data. The total space allocated by SRM (free + used) is represented by the magenta dotted line.

Other Monitors: rucio rse-usage and DDM Accounting Visualization (cont Chart taken from the DDM Accounting Visualization site with MWT2_DATADISK selected Magenta dotted line: free + used SRM space (847TB on Mar 22) Green dotted line: used SRM space (723TB on Mar 22) Red bar: RUCIO reported used space (746TB on Mar 22)

Questions?