A possible SRB to iRODS Migration (a path from here to there)‏ Adil Hasan RAL.

Slides:



Advertisements
Similar presentations
How We Manage SaaS Infrastructure Knowledge Track
Advertisements

Peter Berrisford RAL – Data Management Group SRB Services.
The Platform as a Service Model for Networking Eric Keller, Jennifer Rexford Princeton University INM/WREN 2010.
Unveiling ProjectWise V8 XM Edition. ProjectWise V8 XM Edition An integrated system of collaboration servers that enable your AEC project teams, your.
Data Grid: Storage Resource Broker Mike Smorul. SRB Overview Developed at San Diego Supercomputing Center. Provides the abstraction mechanisms needed.
1 OBJECTIVES To generate a web-based system enables to assemble model configurations. to submit these configurations on different.
1 Opentest Architecture Table of Content –The Design Basic Components High-Level Test Architecture Test Flow –Services provided by each Layer Test Mgt.
2004 Cross-Platform Automated Regression Test Framework Ramkumar Ramalingam, Rispna Jain IBM Software Labs, India.
Distributed Systems 1 Topics  What is a Distributed System?  Why Distributed Systems?  Examples of Distributed Systems  Distributed System Requirements.
1 Security on OpenStack 11/7/2013 Brian Chong – Global Technology Strategist.
OpenVMS System Management A different perspective by Andy Park TrueBit b.v.
Grid and CDB Janusz Martyniak, Imperial College London MICE CM37 Analysis, Software and Reconstruction.
Objektorienteret Middleware Presentation 2: Distributed Systems – A brush up, and relations to Middleware, Heterogeneity & Transparency.
Approaches to EJB Replication. Overview J2EE architecture –EJB, components, services Replication –Clustering, container, application Conclusions –Advantages.
1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.
Overview Of Microsoft New Technology ENTER. Processing....
Robust Tools for Archiving and Preserving Digital Data Joseph JaJa, Mike Smorul, and Mike McGann Institute for Advanced Computer Studies Department of.
How Clients and Servers Work Together. Objectives Learn about the interaction of clients and servers Explore the features and functions of Web servers.
Hands-On Microsoft Windows Server 2008 Chapter 8 Managing Windows Server 2008 Network Services.
Enterprise Reporting with Reporting Services SQL Server 2005 Donald Farmer Group Program Manager Microsoft Corporation.
ADVANCED MICROSOFT ACTIVE DIRECTORY CONCEPTS
By Jeerarat Boonyanit. As you can see I have chosen Cpanel for my server management tool. cPanel is a Linux based web hosting control panel that provides.
Hands-On Microsoft Windows Server 2008 Chapter 1 Introduction to Windows Server 2008.
ATIF MEHMOOD MALIK KASHIF SIDDIQUE Improving dependability of Cloud Computing with Fault Tolerance and High Availability.
QCDgrid Technology James Perry, George Beckett, Lorna Smith EPCC, The University Of Edinburgh.
Module 14: Configuring Print Resources and Printing Pools.
Technology Overview. Agenda What’s New and Better in Windows Server 2003? Why Upgrade to Windows Server 2003 ?  From Windows NT 4.0  From Windows 2000.
Copyright © 2006 by The McGraw-Hill Companies, Inc. All rights reserved. McGraw-Hill Technology Education Copyright © 2006 by The McGraw-Hill Companies,
E X P E R I E N C E Y O U R A M E R I C A Natural Resource Monitoring Database Development Using SQL Server A Comparison Between Visual Basic/ADO.Net and.
Cluster Reliability Project ISIS Vanderbilt University.
Rule-Based Data Management Systems Reagan W. Moore Wayne Schroeder Mike Wan Arcot Rajasekar {moore, schroede, mwan, {moore, schroede, mwan,
Production Data Grids SRB - iRODS Storage Resource Broker Reagan W. Moore
CERN IT Department CH-1211 Geneva 23 Switzerland t Storageware Flavia Donno CERN WLCG Collaboration Workshop CERN, November 2008.
Introduction to the Adapter Server Rob Mace June, 2008.
BaBar Data Distribution using the Storage Resource Broker Adil Hasan, Wilko Kroeger (SLAC Computing Services), Dominique Boutigny (LAPP), Cristina Bulfon.
Team 708 – Hardwired Fusion Created by Nam Tran 2014.
Software Confederations and the Maintenance of Global Software Systems Jaroslav Král, Michal Žemlička Charles University, Prague
1 Week #10Business Continuity Backing Up Data Configuring Shadow Copies Providing Server and Service Availability.
Tool Integration with Data and Computation Grid GWE - “Grid Wizard Enterprise”
Rule-Based Preservation Systems Reagan W. Moore Wayne Schroeder Mike Wan Arcot Rajasekar Richard Marciano {moore, schroede, mwan, sekar,
Chapter 10 Chapter 10: Managing the Distributed File System, Disk Quotas, and Software Installation.
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
Transaction-based Grid Data Replication Using OGSA-DAI Presented by Yin Chen February 2007.
ICALEPCS’ GenevaACS in ALMA1 Allen Farris National Radio Astronomy Observatory Lead, ALMA Control System.
Experiment Management System CSE 423 Aaron Kloc Jordan Harstad Robert Sorensen Robert Trevino Nicolas Tjioe Status Report Presentation Industry Mentor:
1 e-Science AHM st Aug – 3 rd Sept 2004 Nottingham Distributed Storage management using SRB on UK National Grid Service Manandhar A, Haines K,
Introduction to The Storage Resource.
A Technical Overview Bill Branan DuraCloud Technical Lead.
Grid Technology CERN IT Department CH-1211 Geneva 23 Switzerland t DBCF GT Upcoming Features and Roadmap Ricardo Rocha ( on behalf of the.
Tool Integration with Data and Computation Grid “Grid Wizard 2”
CASTOR project status CASTOR project status CERNIT-PDP/DM October 1999.
National Archives and Records Administration1 Integrated Rules Ordered Data System (“IRODS”) Technology Research: Digital Preservation Technology in a.
Institute for the Protection and Security of the Citizen HAZAS – Hazard Assessment ECCAIRS Technical Course Provided by the Joint Research Centre - Ispra.
Interactions & Automations
Interstage BPM v11.2 1Copyright © 2010 FUJITSU LIMITED INTERSTAGE BPM ARCHITECTURE BPMS.
+ Auto-Testing Code for Teachers & Beginning Programmers Dr. Ronald K. Smith Graceland University.
Amazon Web Services. Amazon Web Services (AWS) - robust, scalable and affordable infrastructure for cloud computing. This session is about:
Digital Library Storage using iRODS Data Grids Mark Hedges, Tobias Blanke Centre for e-Research, King’s College London Arts and Humanities Data Service.
MQ Series Cross Platform Dominant Messaging sw – 70% of market
Jean-Philippe Baud, IT-GD, CERN November 2007
Appium Studio Appium testing made easy at any scale.
Security on OpenStack 11/7/2013
Created with Logic Apps on Azure for Health Care Innovation and Interoperability, Corolar Cloud Is a Smarter and Faster Health Care Integration Engine.
Containers as a Service with Docker to Extend an Open Platform
Meemim's Microsoft Azure-Hosted Knowledge Management Platform Simplifies the Sharing of Information with Colleagues, Clients or the Public MICROSOFT AZURE.
Shared Services with Spotfire
Accelerate Your Self-Service Data Analytics
Firewalls Types of Firewalls Inspection Methods Firewall Architecture
Media365 Portal by Ctrl365 is Powered by Azure and Enables Easy and Seamless Dissemination of Video for Enhanced B2C and B2B Communication MICROSOFT AZURE.
MQ Series Cross Platform Dominant Messaging sw – 70% of market
Presentation transcript:

A possible SRB to iRODS Migration (a path from here to there)‏ Adil Hasan RAL

Why iRODS? SRB used by large number of projects.  Provides data management system virtualizes data.  X509 authentication. Misses functionality that's commonly required like:  checksum data regularly,  extract metadata store in different database etc.

Why iRODS? Missing functionality has to be scripted outside of SRB.  Difficulty in interacting with SRB. SRB has a lot of functionality, even more than most people use, not easy to exclude unwanted functionality. SRB starting to become difficult to maintain.

What else is there? Domain specific Data Management Systems (like ESA DMS). Difficult to generalize, require extra s/w. Distributed file-system (nfs, webdav). Data not easily queriable, requires interface to MSS. Build your own, problem is you have to maintain it.  Better to be part of a bigger group contributing to an DMS.

iRODS Rule-oriented system.  Each data management task regarded as a rule.  Each rule is executed by one or more micro- services.  Rules can be queued, run asynchronously. Configurable system.  Can compile-in only micro-services required. Ability to enhance storage resources.

Migration Issues But we have data in SRB how do we present data in iRODS? Migration plan important. Also helps to define what we want from the system.  Criteria system should meet, services it should provide etc.

Migration Issues Important to have group in place that can:  Pool together requirements and functionality the system must provide.  Be used to organize effort on producing a production-quality iRODS system.  Share knowledge on iRODS (a self-help group). Group should primarily be EU-focussed (time- zones). But, have interest from Australians (maybe EU++).

Migration Issues Group called DataGridsEXchange (dgex): (coords J-Y Nief, a.hasan).

iRODS Requirements Based on our systems want:  Ability to handle small files.  Interface to our MSS.  Single-Sign-On capabilities (X509, Kerberos).  Fine-grained access-control (to be refined).  Auditing (who accessed, when etc).  Replication (automatic replication to resources subject to certain criteria).

iRODS Requirements Synchronize data (ie update iRODS copy or replicas). Checksum data (md5, system V etc). Bulk operations. Robust file-transfer (handle flaky networks) maybe with checkpointing. GUI front-ends and web-based front end (drag and drop on all platforms desirable).

iRODS Requirements Ability to queue and run rules at regular intervals (can pause queue etc). APIs important in Java, Python, Perl, C. Audit info of rules (which version when run etc, incl versioning of micro-services). Easy to install, update. Monitoring of system health. Load balancing.

iRODS Requirements File-system-like interface (webdav, nfs) for legacy apps. Understandable uniform logging. Interoperation with SRB and other systems.

Migration Plan Get people to join dgex (the more the merrier). Identify critical (ie must have) common components of the system. Test iRODS:  See how iRODS scales with number of files.  See how iRODS scales with concurrent access.  See how iRODS handles error situations.  Compare with SRB.

Migration Plan Based on tests either:  provide feedback to SDSC as to what needs to be improved (with suggestions).  provide SDSC with modified code. Requires SDSC (or someone) hosts Gforge system to manage projects.

Migration Plan Work on missing, critical components. Test, test, test. Work on interoperation of SRB with iRODS.  Able to access SRB data through iRODS. Work on migration tools extracting data from SRB inserting into iRODS.

Migration Plan Creating a production iRODS system will require close collaboration from all interested parties. Communication through dgex will be important.

Take Care of Your Data Aiee! Save me! All my data is being eaten by these bug-eyed monsters! And now they want to Sashimi my brains! We Don't Want This We Do Want This I'm here to make protect all the data in this building by making sure all the rules are obeyed and incinerating those who don't obey!