1. Introduction to iRODS Christine Staiger SURFsara 10 th Oct 2014

2. iRODS Concepts The Basics ① Installation ② Tools for administration ③ Adding data and metadata ④ iRODS rules Contents

3. iRODS Concepts 3

4. What is iRODS? 4

5. What does iRODS provide? Storage virtualization of different disk and tape storage systems A logical namespace across storage locations A rule engine to automate data management according to defined policies Various client interfaces 5

6. iRODS Components 6

7. In the background: iRODS Resources (Storage) Resource is a Software or Hardware system that stores data 3 Resource classes: 7 High Latency High Latency Low Latency Low Latency Archive Resource Archive Resource Cache Resource Cache Resource Compound Resource Compound Resource POSIX like IO calls

8. Resource Groups: Distributing files 8 Low Latency Low Latency Cache Resource Cache Resource POSIX like IO calls Resource Group Resource Group Low Latency Low Latency Cache Resource Cache Resource Low Latency Low Latency Cache Resource Cache Resource @Server A@Server B@Server C

9. Resource Groups: Data replication 9 Low Latency Low Latency Cache Resource Cache Resource POSIX like IO calls Resource Group Resource Group Low Latency Low Latency Cache Resource Cache Resource Low Latency Low Latency Cache Resource Cache Resource @Server A@Server B@Server C Replicate

10. The users view: iRODS Data Collections 10 Collection 0 Data Object Data Object Data Object Data Object A Collection 1 Data Object Data Object Data Object X …/Collection0/ …/Collection0/DataObject A-B …/Collection0/Collection1/ …/Collection0/Collection1/DataObject X-Z Logical Name Space (iCAT) independent from physical resources Similar to Unix Directory and Files ichmod to set user/group ACLs Core.re: msiSetDefaultResc(Resource).irodsEnv: irodsDefResource=Resource Similar to Unix Directory and Files ichmod to set user/group ACLs Core.re: msiSetDefaultResc(Resource).irodsEnv: irodsDefResource=Resource

11. Hands on The Basics 11

12. Demo Data Grids Virtual machines: ssh user@IP IP adressUserPassword 145.100.57.159irods1irods104 145.100.57.162irods2irods204 145.100.57.164irods3irods304 145.100.58.150irods4irods404 145.100.58.155irods6irods604 145.100.58.157irods7irods704 145.100.58.158irods8irods804 145.100.58.160irods9irods904 145.100.58.161irods10irods1004

13. Installation and configuration iRODS3: Install script./irodssetup iRODS4: (rpm, dmg) package and source code ~/iRODS/config/irods.config # Database configuration $DATABASE_TYPE = 'postgres'; … $DATABASE_HOST = 'localhost'; $DATABASE_PORT = '5432'; $DATABASE_ADMIN_PASSWORD = 'irods2'; $DATABASE_ADMIN_NAME = ’alice'; # iRODS configuration IRODS_HOME = '/home/alice/iRODS'; $IRODS_PORT = '1247’; … $IRODS_ADMIN_PASSWORD = ’alice'; $IRODS_ICAT_HOST = '';

14. .irodsEnv file # iRODS personal configuration file. # # This file was automatically created during iRODS installation. # Created Mon Jun 18 13:13:27 2012 # # iRODS server host name: irodsHost ’surfsara-cloud' # iRODS server port number: irodsPort 1247 # Default storage resource name: irodsDefResource 'demoResc' # Home directory in iRODS: irodsHome '/alicesZone/home/alice' # Current directory in iRODS: irodsCwd '/alicesZone/home/alice' # Account name: irodsUserName ’alice' # Zone: irodsZone ‘alicesZone' The.irodsEnv determines the zone and the user for the icommands-client. You can only use one such file at a time.

15. iRODS i-commands Documentation: https://wiki.irods.org/index.php/icommands (we don’t provide a exhaustive list of i-commands)

16. iRODS control Execute./irodsctl in the folder iRODS Restarting, starting and stopping the iRODS and iCAT server with = restart, start, stop Restarting, starting and stopping only the iRODS server with = irestart, istart, istop Initialise an account and log in with iinit 16

17. ilsresc ilsresc lists iRODS resources and resource-groups Usage: ilsresc [-lvVhA] [Name] alice@sara-cloud:~$ ilsresc demoResc alice@sara-cloud:~$ ilsresc –l demoResc resource name: demoResc resc id: 10010 zone: alicesZone type: unix file system class: archive location: surfsara-cloud vault: /home/alice/iRODS/Vault …

18. User and Data administration Change to admin mode: iadmin Quit admin mode: q iadmin mkresc, iadmin rmresc Create new resource: mkdir Data (in your home, not in the iRODS environment) iadmin mkresc “unix file system” archive you.rIP.add.ress “/home/ /Data” Make/delete a user: mkuser, rmuser, moduser (iadmin) mkuser user1 rodsuser (iadmin) moduser user1 password ***** Parameters and their values: iadmin lt, iadmin lt user_type

19. Put and replicate data Add the file /home/login/put1.txt to resource iput -v put1.txt ils -l or ils -L (gives also physical location) iput -h Create a collection and add data imkdir / /archive iput put2.txt / /archive/put2.txt Check physical location of the data ils -L / /archive Replicate data across resources: irepl -R put1.txt Check with ils -L Trim number of copies: itrim -S -N 1 put1.txt

20. iRODS Metadata Commands: imeta, iquest, idbo Add metadata: imeta add -d put1.txt “Date” “08.10.2013” imeta add -d put1.txt “AnotherDate” “04.04.2014” imeta ls -d put1.txt Remove metadata: imeta rm -d put1.txt “Date” “08.10.2013” Querying the iCAT: iquest "SELECT DATA_NAME,DATA_CHECKSUM WHERE DATA_RESC_NAME like ’%Resc%’” DATA_NAME = put1.txt DATA_CHECKSUM = c16d7d3488677a3348b12eb82795d28c

21. iRODS Microservices and rules Micro-services are small, well-defined procedures/functions that perform a simple task Examples: “msiCollCreate”, “msiDataObjRepl”, “msiDataObjChksum”, … Convert any C procedure (createCollection) to a microservice by creating an interface routine (msiCreateCollection) Micro-services can be combined in rules to enforce policies Rules can be executed on action, with time-delay or periodically

22. iRODS Rule Engine 22

23. Replicate to another resource replicate{ msiWriteRodsLog("starting replication", *status); msiWriteRodsLog("source = *source", *status); msiDataObjRepl(*source, *destRescName, *CC); } INPUT *source=”/ /home/ /put1.txt”, *destRescName=“testResc” OUTPUT ruleExecOut  Save as replicate.r  Execute irule -vF replicate.r  Check with ils -L

24. Replicate to another collection Useful for backup of a data collection Changing the storage medium iput put2.txt replicate{ msiWriteRodsLog("starting replication", *status); msiWriteRodsLog("source = *source", *status); msiDataObjRsync(*source, "IRODS_TO_COLLECTION", *destRescName, *destination, *rsyncStatus); } INPUT *source="/ /home/ /put2.txt", *destination=”/ /coll1", *destRescName=“ " OUTPUT ruleExecOut ils -L /irodsXZone/coll1/home/irodsX

25. Thank you !

26. Appendix

27. Trouble shooting ERROR: putUtil: put error for /alicesZone/home/bob#bobsZone/put1.txt, status = -9000 status = -9000 SYS EXCEED CONNECT CNT Host configuration problem on the server. Solution: Edit iRODS/server/config/irodsHost localhost you.rIP.XX.XX surfsara-cloud localhost 127.0.0.1./irodsctl restart