DELETION SERVICE ISSUES ADC Development meeting

Productivity (overall) Productivity of service: A year ago: 500K of files per day (was enough) Now: 2M per day (2,5M – estimated value, 1,5M – lowest limit) (not always enough) Monitoring: Number of operation of DS increase at least x5 Speed of on-line aggregation decrease -> slow report generation Some reports stuck (overview for last week)

Productivity (in details) Not so high as wanted (less 10k of files for one site) Slow LFC clearing Problems: Low productivity of one threat (a lot of time spent by Deletion Agent for intercommunication with backend) One LFC cleaner per LFC server -> one cleaner per cloud (last year implementation)

New version of Deletion Service In testing with production data for two endpoints in BNL (BNL-OSG2-MCDISK, BNL-OSG-USERDISK) Deletion Agent deployed on atlddm17, Deletion server deployed on dedicated instance (atlddm16) – no impact from other services. New Features Changed algorithm of intercommunication of Storage cleaner for decreasing number of calls of CC (HTTP server) Number of catalog cleaners not depend from number of LFC servers Limitation of number of resolving files (give possibility to have proper size of huge operation table in DB)

First results. It’s preliminary results: more tests with loading needed Storage cleaner performance increase (22-28K of deleted files per hour). Decreased time for intercommunication with backend (bulk operations): No significant changes for LFC Cleaner for single thread, but number of threads can be increased Idea 1: number of operations with DB didn’t change –> CC performance? Idea 2: Oracle can update data with around 100Hz (for Deletion Service) Chunk sizeOld versionNew version sec1,5-5 sec.

Next steps (very nearest plans) Continue test usage with increasing of loading (next two weeks) -> will be good to have a plan. Implement same strategy for Catalog Cleaner as for Storage Cleaner (operations across datasets, ‘bulk’ operations for update) Special processing for jumbo datasets Current implementation of resolving are not coherent for jumbo datasets. It affect DB and CC, so different implementation needed.

Needed features. Some times of data should be removed for limited period, and will be good if biggest files will be deleted from storage early then smallest -> priority of deletion needed for this case. Directory deletion Local installation (implemented in current architecture – but validation needed)

“Plan B” Reviewing of architecture, in case implemented features will not give estimated effect Decentralization for most loading operation Significant changes in DB structure Etc. Plan B will means significant changes for Deletion Monitoring too…

Deletion monitoring Decrease of performance due to a lot of aggregation. Tunes of query have limits DB structure oriented for mostly for deletion process Not all metrics shown (LFC deletion missing) Some more metrics needed for defining problems/productivity of each component (Storage, LFC, Backend)

Deletion monitoring. Nearest plan Performance increase: extension of DB schema with summary tables, filled by triggers and jobs. Interface extensions Additional metrics: already collected by Deletion Agent and put to log... But can be send someware (asynchronously ideally) Should not be collected in production Oracle Can be collected in NoSQL -> most of operation will be insert, select with aggregation