1. RDA 9th Plenary
Breakout 3, 5 April :00-17:30
Joint meeting: IG Brokering, IG Data Fabric “Advancing Data Fabric with Brokering services”
Wrap-up and next steps (Stefano Nativi, Bridget Almas, Peter Wittenburg, Jay Pearlman, Larry Lannom, ….)

2. Why are we here?
Data Fabric Core Components, such as the PIT and DTR, provide essential pieces of infrastructure for managing and sharing Research Data
But in order to be useful, they must be connected to provide functionality in research workflows
Brokers have traditionally served this role in SOA architectures, providing facade interfaces, mediation and abstraction

3. GAP Composition Abstractions and Implementations
PID
Abstract Composition
Data Brokers
Executable Workflow(s)
Objet Types
Processing Brokers
PID and Object typing help fill the gap
Brokering services (e.g. Data services brokering and Processing services brokering) help fill the gap.

4. From Human-Controlled Process to Type-Triggered Automatic Processing
End-to-end architecture
Community Services architecture N M
N + M N M
N x M
Data Server
Client applications
Client applications
Harmonization task
Mediation task
Intermediation services
Community (Third-party)
Human Controlled Processing (HCP)
Type-Triggered Automatic Processing (T-TAP)
Web-as-a-Platform
Mediation task
Mediation task
Data Servers 4

5. Client-Broker-Server architecture
Community Services architecture
Data Server
Client applications
Community services (Third-party)
Web-as-a-Platform
Intermediation services
Broker(s)

6. Useful (architectural and Software) engineering Patterns
Separation of concerns (SoC)
separate architecture into distinct components, such that each component addresses a separate concern
Information Expert
Assign a responsibility to the component that has the information needed to fulfill it
Low coupling
Implement low dependency between the components

7. Brokering Services
Provide the intermediation functionalities to pass from Abstraction to Implementations and vice versa
Apply Service-Oriented Architecture approach
Utilize Web-as-a-Platform (Web 2.0 patterns)
Intermediation-as-a-Service
Third-party services: not managed by Client/Server organizations

8. Brokering Services Benefits Challenges
Multi-purpose, Reusable, Participate in Composition
Sustainable through ability to evolve
Flexible, Configurable, Extensible
Challenges
Trust, Governance
New cultural and business model
Requires specialist knowledge
Types
Discovery and Access
Processing and Transformation

9. Future: Data and workflow integration
Goal: Make the life easier for scientists who are no experts in programming and handling data
Portals: integrate data and compute workflows
Data
Workflow
Result
Preview:
Set 1 ✔
Set 2 ✔
Set 3
Workflow diagram: Diana tools – Taverna
Heatmap: wikipedia …
Data: <PID> 9

10. Future: Data and workflow integration
Label data with PID
Label (parts of) workflows with PIDs
#Load data
mRNA = W1(D1)
miRNA = W1(D2)
#Analysis
res1 = W2(mRNA, miRNA)
res = W3(res1)
#PID for result
file = writeToFile(res)
create_PID(file)
#Plot for preview
Plot(res)
PID D1
PID D2
PID D3
PID W1
PID W2
Portal builds the script and executes it based on PIDs for data and PIDs for code snippets.
PID W3
PID W4 10

11. Matching in federation environment
Repository
metadata of DO
bit sequences of DO
PID Resolver MD bs
PID record 1
DTR 5
PID
Rights DB
Type Record 3 2 12 4
rights record 6 13 10
Agent 9 Y 8 7 11
Controller WF 14
Matcher 14 14
Broker
Profiles
Broker
Broker

12. Scalability, efficiency, flexibility, sustainability, close to the Client needs
Clients
Application
Community services & Interface
new breed of organizations providing intermediation services to “abstract” current “implementations” and enable composition
Composition
Intermediation
Supply system

13. PID Centric Data Management and Access
Brokering & Mediation
services

14. Biodiversity Use Case

15. Climate data processing use case (I)
Challenge:
Worsening ratio of bandwith vs. volume
Server-side processing – cultural change in scientific work
Solution:
Favour support for most common processing tasks over complexity/variety
Service innovation rather than technical innovation
Understand users, train influencers, reach high quality

16. Climate data processing use case (II)
Motivation:
multiple initiatives to build data processing solutions – some exist, some to come
variety of data sources: CMIP6, Copernicus, ...
some standardization across them, but detail differences
brokering as middleware service – not for end- users, but software agents

17. WPS (final, preprocessing)
Climate data processing use case (III)
Agent
(user need)
WPS (final, preprocessing)
Data sources
Broker
Broker component with a specific role:
Middleware service catalogs will come; defines broker role
Needed: Clear definition for broker component, respecting multiple roles
common interface for broker tasking and output format (to be fed into controller)

18. Broker acting as Controller

19. Discussion

20. Thank you !