EGEE is a project funded by the European Union under contract IST Data Management Gaps Krzysztof Nienartowicz Gavin McCance EGEE JRA1 Data Management Cluster JRA1 Meeting, Oxford, 30 June

JRA1 Meeting, Oxford, 30-June Functional Gaps File pinning on SE  Who asks the SE to pin/unpin the files? For how long?  The DS or the site job manager? Optimization  Is any data optimization necessary? Using the replica catalog to inform the WMS for optimal job placement? Use replica catalog as store for current state of transfer? Use replica catalog as store for staging status? Implies continuous link between SE and catalogs.

JRA1 Meeting, Oxford, 30-June Gaps: Volume, scalability How many items: ?  Logical space in (file) catalogs  Over SEs Partitioning of data – both logical and physical dimensions  Over sites/nodes/groups/users/sites? Job limitations  Functional difference depending on the transactional logic: Is partial success of SE work acceptable by Job Manager? JDL option? Max size of transactions or bulk operations Max transfer times Max error occurrences of specific type?  Contingency logic if we hit limits Policy management, quota policy as a JDL option/hint?

JRA1 Meeting, Oxford, 30-June Gaps Distribution of catalogs  Necessary?  Propagation of changes Local transactions Model for replication, updates –Lazy updates –Publish-subscribe –Relationship with Accounting/Auditing Clashes handling –Manual –Semi-manual –Semi-automatic –Automatic

JRA1 Meeting, Oxford, 30-June Gaps- Complex Interface implementation. Internal dependencies, atomicity.

JRA1 Meeting, Oxford, 30-June Gaps- Complex Interface implementation. Cost of modularisation.