Author - Title- Date - n° 1 GDMP The European DataGrid Project Team

EDG GDMP Tutorial - n° 2  based on CMS requirements for replicating Objectivity files for High Level Trigger studies  production prototype project for evaluating Grid technologies (especially Globus)  experience for will directly be used in DataGrid n input also for PPDG and GriPhyN   implementation: Asad Samar, Heinz Stockinger

EDG GDMP Tutorial - n° 3 CMS DM Requirements n Data production (currently tens of Terabytes). n Managing the data locally at regional centres. n Replicating this data to other centres. s High speed transfers. s Secure access. s Disk management facilities s Minimizing human interference. s Fault tolerance and error recovery mechanisms. n Data integration on the destination. n Logging and book-keeping. Submit job Replicate data Replicate data Site A Site B Site C " Jobs are executed locally or remotely " Data is always written locally " Data is replicated to remote sites Job writes data locally

EDG GDMP Tutorial - n° 4 Subscription Model n All the sites that subscribe to a particular site get notified whenever there is an update in its catalog. n The sites that don’t subscribe have to poll themselves for any changes in the catalog. n Support both a pull and a push mechanism. Site 1 Site 3 Site 2 Subscriber list Subscriber list subscribe poll for changes

EDG GDMP Tutorial - n° 5 Export / Import Catalogue n Export Catalog s information about the new files produced. s is published n Import Catalog s information about the files which have been published by other sites but not yet transferred locally s As soon as the file is transferred locally, it is removed from the import catalogue. n Possible to pull the information about new files in your import catalogue. Site 1 Site 3 export catalog import catalog Site 2 export catalog 1) publish new files 2) transfer files 1) get info about new files 3) delete files

EDG GDMP Tutorial - n° 6 Usage  gdmp_server n server running at each site  gdmp_host_subscribe n first thing to be done by a site  gdmp_publish_catalogue n send information of newly created files to subscribed hosts (no real data transfer)  gdmp_replicate_file_get n get all the files from the import catalogue  gdmp_get_catalogue n creates a list of missing files; for error recovery

EDG GDMP Tutorial - n° 7 GDMP File Transfer Steps GDMP Site B GDMP Site A Host subscribe(0) Exchange of Globus Certificate Subject(-1) Trigger Publish files(1.1a) Publish Request(1.1b) Publish list of all files(1.2b) Publish list of new files(1.2a) Transfer required files(2) MSS Stage Migrate Staging request Replicate Catalog * Steps <=0 are done once per GDMP server Staging done (3)Post- processing OBJY Catalog

EDG GDMP Tutorial - n° 8 Using GDMP Site 2 Site 1 Site 3 Site 4 Site 5 Data produced at site 1 to be replicated to other sites

EDG GDMP Tutorial - n° 9 Using GDMP 2  Start with subscription n gdmp_host_subscribe –remotehost -remoteport Site 2 Site 1 Site 3 Site 4 Site 5 gdmp_host_subscribe Subscriber list

EDG GDMP Tutorial - n° 10 Using GDMP 3  Publish new files…can combine with filtering n gdmp_publish_catalogue n gdmp_filter_catalog-export Site 2 Site 1 Site 3 Site 4 Site 5 Subscriber list gdmp_publish_catalogue Export catalog Import catalog Import catalog Import catalog

EDG GDMP Tutorial - n° 11 Site 2 Site 1 Site 3 Site 4 Site 5 Subscriber list Export catalog Import catalog Import catalog Import catalog gdmp_get_catalogue Import catalog Using GDMP 4  Poll for change in catalog (pull model)…can combine with filtering…also used for error recovery. n gdmp_get_catalogue –host n gdmp_filter_catalog -import

EDG GDMP Tutorial - n° 12 Site 2 Site 1 Site 3 Site 4 Site 5 Subscriber list Export catalog Import catalog Import catalog Import catalog Import catalog gdmp_replicate_file_get Using GDMP 5  Transfer files…can use the progress meter n gdmp_replicate_file_get n get_progress_meter…produces a progress.log. n replica.log has all files already transferred.