1 iRODS: A Rule Oriented Data ManagementSystem SRB Space

2 Beyond the Storage Resource Broker SRB is a data management system for large-scale data Logical name space -- Independence from Physical Pin Downs Integrated Data and Metadata Management Uniform Access Interfaces Caters to multiple tasks and paradigms Data grid Federations for distributed and replicated data handling Cooperating Autonomous Virtual Organizations (VO) Persistent Archives for long-term preservation Building light, dim and dark archives Digital Libraries for semantically searchable data sharing multiple domains with collection-level functionalities Server-side Operations for performing data intensive operations Data sub setting, data fusion, administrative management Used in large-scale systems in production

3 What Next? SRB is quite complex – with many functions and operations > 90 commands with many options several 100 unique ops The intelligence is hard-coded extensions/modifications require extreme care but, the modules are fairly robust and reusable SRB is a one-size fits all architecture everyone gets the same code base Users want more functionality increased customizability want a small foot print as necessary Easy for them to modify independence from developers functionality to fit policies and not the other way around!!

4 What Do the Users Want? Innovative Access Control Sometimes by groups, sometimes by users & sometimes by roles Based on their login type – how they got authenticated Third party authorization - outside authority agent Dynamically changeable access control Access Control Lists, Denial lists, over-rides,… Ticket-based short-term and controlled access Data Placement Strategies Completely user controlled – user preference policies Completely Administration controlled – site policies Group-based policies Over-rides, exceptions Based on Data characteristic or Collection characteristic Policies for staging, caching, archiving, purging, synchronization,… Ingestion Policies Check for authenticity – anonymization, Pre and post process Replication policies, metadata extraction policies, permission policies,… And others …

5 Rule Oriented Data Management Adaptive Middleware Architecture Customizable and Flexible – User Configurable Administratively Simpler – Admin Configurable Build upon the experience of SRB Data Grid Rule-oriented Programming Well-defined set of functionalities --- Micro services Define Rules which chain micro-services Work-flow of micro services Define Rule Application Condition Define Recoverability for failure management Administrators can set site policies Users can encode their preferences Groups can set their process requirements Control actions at collection-level, format level, user level, resource level, ….

6 Rules and Constraints Rule-based Lower-level Functions are composed of micro-services Higher-level Functions are composed of rules of lower-level micro-services Rules are interpreted using a rule engine Customizability Problems with rule composition Integrity checks to make sure rules do not break higher-level functionailties Declarative programming Rules define semantics Operational programming Rule invocation provides procedural interpretation Rules can be used as “checks and balances” to make sure that collections are self-consistent Example: Rule makes two copies of each files Constraint checking: can be used to see if the collection is consistent with this rule

7 Rule-Oriented Data Systems Framework Resources Client InterfaceAdmin Interface Metadata Modifier Module Config Modifier Module Rule Modifier Module Consistency Check Module Confs Rule Base Meta Data Base Engine Rule Curren t State Rule Invoker Micro Service Modules Resource-based Services Micro Service Modules Metadata-based Services Service Manager Consistency Check Module Consistency Check Module

8 Rules Flow Application Client Call Server Call Select Firstt/Next Rule Find Appropriate Rules Condition Check Execute Next MicroService/Action Success Execute Recovery MicroService/Action Yes No Success: No More MS/A True False Failure: No More Rules

9 ingestObject(*F) createFile(*F), registerFile(*F). ingestObject(*F) $userDept == sdsc OR $userDept == sio createFile(*F), registerFile(*F), computeChkSum(*F),!, findBackUpRsrc(*F, *R), replicateFile(*F, *R), computeCheckSum(*F, *R), compareCheckSum(*F). ingestObject(*F) $dataType == FITS Image createFile(*F), registerFile(*F), extractFITSMetadata(*F). Sample Rules

10 Format of a Rule Action :- Condition | MS 1, …, MS n | RMS 1, …, RMS n Action to be performed Condition checked to see if rule is applicable If applicable micro services {1,…n} are executed If any micro service fails, recovery micro service(s) executed to maintain transactional capability createFile(*F) removeFile(*F) ingestMetadata(*F,*M) rollback Caveats: More than one rule can define an action R/MS i can be actions Micro services can pass parameters

11 AMA & ROP A New Paradigm in Middleware Development Higher level Services composed of Micro-services Customizable at multiple levels Glass Box Architecture Can explain what happens Semantics can be checked Run-time Version Control Combines multiple paradigms Workflow systems, active databases, rule-based execution, transaction systems, data grids and remote execution of services Flexible Management Administrative ease Triggers for handling low/high water marks Periodic Job execution – backup, archive, usage control,…

12 Components of Rule System Actions Name Space of Actions Client Call Maps to Actions Micro Services Well-defined Server-side Procedures and Functions Rule Definitions for Actions Workflow of what to do Composed of of Actions and Micro Services Invoked to execute an Action Rule Base Set of Rules Each User Community can choose their own rule base Data Components Blackboard Architecture Used by Micro Services,Actions and Rules

13 Data Components of Rule System Persistent Data Attributes: # Has an external name space Mapping to internal database attributes Persists across sessions Session Data Attributes: $ Has an external name space Mapping to internal data structures Used by micro-services/actions inside a session Side Effects Set: % Changes affected outside the system File created, File Copied, Sent, … Well-defined name space of activities

14 Micro Services Compiled Functions Short and Well-defined functionality Should have a clear semantics Works on $,#,% Examples: Metadata Extraction for DICOM Access Control Permission Changed to User Replicate a file from Source to Destination

15 Semantics Micro Service Semantics Input /Output Variables (in terms of $) Input: what is needed Output: what gets changed Persistent Changes (in terms of #) Updates to Databases Activities Performed (in terms of %) External Activities Performed

16 Semantics Rule Semantics Based on component micro services Action Semantics Based on corresponding rules Only one rule semantics apply

17 ingestInCollection(S) :- /* store & backup */ chkCond1(S), ingest(S), register(S) findBackUpRsrc(S.Coll, R), replicate(S,R). ingestInCollection(S) :- /*store & check */ chkCond2(S),computeClntChkSum(S,C1), ingest(S), register(S), computeSerChkSum(S,C2), checkAndRegisterChkSum(C1,C2,S). ingestInCollection(S) :- /store, chk, backup & chk */ chkCond3(S),computeClntChkSum(S,C1), ingest(S), register(S), computeSerChkSum(S,C2), checkAndRegisterChkSum(C1,C2,S), findBackUpRsrc(S.Coll, R), replicate(S,R) computeSerChkSum(S,C3), checkAndRegisterChkSum(C2,C3,S). ingestInCollection(S) :- /*store,check, backup & extract metadata */ chkCond4(S),computeClntChkSum(S,C1), ingest(S), register(S), computeSerChkSum(S,C2), checkAndRegisterChkSum(C1,C2,S), findBackUpRsrc(S.Coll, R), [replicate(S,R) || extractRegisterMetadata(S)]. ingestInCollection(S) :- /* just store */ ingest(S), register(S). Sample Rules chkCond1(S) :- user(S) == chkCond1(S) :- coll(S) like ‘*/scec.sdsc/img/*’. chkCond2(S) :- user(S) == chkCond3(S) :- user(S) == chkCond4(S) :- user(S) == datatype(S) == ‘DICOM’. [OprList] implies delay for later or send to a CronJobManager Opr||Opr implies do them in parallel Opr, Opr implies do them serially

18 Middleware Software providing complex distributed applications/services Client-server Peer-to-peer Web servers, Content Managers, Databases, Application Servers,… Client access through common protocols RPC, Message-oriented, Object Request Broker, WSDL or service-oriented Middleware provide a specific set of services

19 Middleware Normal Middleware are black boxes Expose a set of interfaces/service definitions No customization System Developer has complete control A Service will have very configurability option - even in open source middlewares Applications are developed on top of middleware

20 Adaptive Middleware Architecture Similar to normal middleware Provides a set of services Has a well-defined access protocol AMA not a Black Box Admin/User Customizable Service Tweak services to achieve alternate goals Can explain at a high-level what is happening One can compare two AMA services to see how they differ Useful for verification and analysis

21 Adaptive Middleware Architecture External View – Logical Name Space Persistent Memory – Database Transient Memory – Variables External Side-effects Interaction to outside world Ex. File is created, is sent Services, Methods, Actions Rules, Workflow Internal View – Programmatic View Changes in DB Tables, internal variables/structure Procedures, Methods and Functions Drivers, Protocols Users, Resources, Data Objects – methods affecting them Mapping External to Internal Capturing Semantics of Services and Rules Validation, Analysis, Introspection