Physics data processing with SOA 4/10/2019 V. Gyurjyan. CLAS12 software meeting

V. Gyurjyan. CLAS12 software meeting Architecture Formal description Purpose, functions, external visible properties, interfaces Internal components and their relationships Scalability and maintenance Evolution 4/10/2019 V. Gyurjyan. CLAS12 software meeting

V. Gyurjyan. CLAS12 software meeting 4/10/2019 V. Gyurjyan. CLAS12 software meeting

V. Gyurjyan. CLAS12 software meeting What is SOA? Style of building reliable systems that deliver functionality as services Loose coupling between interacting services 4/10/2019 V. Gyurjyan. CLAS12 software meeting

V. Gyurjyan. CLAS12 software meeting Service Atomic unit of an SOA. Encapsulates logic, data, or process. Autonomous Location Transparency It is defined by the messages it can accept and the responses it can give. Messages can be one-way, synchronous or asynchronous. They can be integrated to provide higher-level services It can be completely self-contained, or it may depend on the availability of other services or other resources. 4/10/2019 V. Gyurjyan. CLAS12 software meeting

V. Gyurjyan. CLAS12 software meeting Service interaction Service Registry Discover Advertise Service Consumer Service Provider Interact 4/10/2019 V. Gyurjyan. CLAS12 software meeting

V. Gyurjyan. CLAS12 software meeting Loose Coupling Flexibility Scalability Replaceability Fault tolerance 4/10/2019 V. Gyurjyan. CLAS12 software meeting

V. Gyurjyan. CLAS12 software meeting Statelessness Service Registry Discover Discover Advertise Service Provider Service Consumer 1 Data storage Service Provider 2 3 Service Provider 4/10/2019 V. Gyurjyan. CLAS12 software meeting

SOA is for distributed computing SOA design can be used for a single system - individual processes as services - communication through well defined interfaces, using local channels or high-speed interconnect. 4/10/2019 V. Gyurjyan. CLAS12 software meeting

V. Gyurjyan. CLAS12 software meeting SO vs. OO OO advocates tight coupling, i.e. users have static knowledge of the object. Service has an internal thread of control. SOA promotes the design of the system in which it is not necessary that requests and responses are handled by the same set of communicating entities. 4/10/2019 V. Gyurjyan. CLAS12 software meeting

Do we need distributed physics data processing? HEP and NEP go GRID GRID – distributed computing in which resources often spread across different physical locations and administrative domains. Two classifications for GRID: - compute - data 4/10/2019 V. Gyurjyan. CLAS12 software meeting

V. Gyurjyan. CLAS12 software meeting SO GRID Grid Standards OGSI Extension to WSDL WS-Resource WS-ResourceLifetime WS-ResourceProperties WS-RenewableReferences WS-ServiceGroup WS-BaseFaults SOA Standards WSDL UDDI BPEL WS-Profile WS-Security WS-Choreography 4/10/2019 V. Gyurjyan. CLAS12 software meeting

CLAS12 mini-GRID infrastructure CNU Grid infrastructure ODU JLAB UVA 4/10/2019 V. Gyurjyan. CLAS12 software meeting

CLAs Reconstruction and Analyses framework CLAS12 software framework and middleware CLAS12 physics data processing services 4/10/2019 V. Gyurjyan. CLAS12 software meeting

Framework design choices Separation between: a) data and algorithm b) transient and persistent data 4/10/2019 V. Gyurjyan. CLAS12 software meeting

V. Gyurjyan. CLAS12 software meeting Algorithm services Algorithm services will process one type of data and generate data objects of a different type. Algorithm Service Data Data Data 4/10/2019 V. Gyurjyan. CLAS12 software meeting

Persistent and Transient Data Algorithm service should not use directly data objects in the persistent storage. Transient data storage as a means of communication between physics algorithm services. Data service translates data from persistent storage into transient storage. 4/10/2019 V. Gyurjyan. CLAS12 software meeting

V. Gyurjyan. CLAS12 software meeting Why not one data object? Two different optimization criteria for applications using persistent and transient data. Being independent from the persistent storage technology. 4/10/2019 V. Gyurjyan. CLAS12 software meeting

V. Gyurjyan. CLAS12 software meeting Data service Persistent Data Transient Data Data Data Service Adapter 4/10/2019 V. Gyurjyan. CLAS12 software meeting

Data Object categories Event data Detector data Statistical data 4/10/2019 V. Gyurjyan. CLAS12 software meeting

V. Gyurjyan. CLAS12 software meeting Framework Services One Supervisor service for each application. Algorithm services (Tracking, Calibration, SegmentFinder, VertexReconstruction, TruthAssociation,etc.) Data services (Geometry, PersistentEvtData, PersistentDetectorData, TransientEvtData, etc.) JobOptions Messaging etc. 4/10/2019 V. Gyurjyan. CLAS12 software meeting

Hypothetical tracking Supervisor start retrieve SpacePointFormation EvtData TransientEvtData record start CoarseTrackFinder DetectorData retrieve Raw Data SeadMaker StatData Space Points VertexFinder record ClusterAnalyzer Track candidates TransientDetData retrieve start AmbiguitySolver TransientStatData record Resolved Tracks start retrieve TrackFitter Final Tracks TrackScoring Transient Storage Tracking State machine 4/10/2019 V. Gyurjyan. CLAS12 software meeting

ClaRA implementation details Platform 4/10/2019 V. Gyurjyan. CLAS12 software meeting

ClaRA Service Categories Normative services administrative service Crate and manage all other services Environment administration Supervisor service - Contains state machine of the particular data processing Data processing service NS SS S 4/10/2019 V. Gyurjyan. CLAS12 software meeting

Communication between services All service communications are through message transfer. Message format is ACL (Agent Communication Language) defined by FIPA (Foundation for Intelligent Physical Agents). Each message is one of several predefined communication types. Message structure: - Sender - Receiver - Content (Data Object or Ontology Object) - Language - Ontology - Protocol Not understood Inform-content Refuse-reason Types Agree Failure-reason 4/10/2019 V. Gyurjyan. CLAS12 software meeting

Service Creation NC NR NA ClaRA platform Data processing description Administrative domain and container ClaRA platform NC NR Data processing description NA Service Service container Service: Java or c/c++ through JNI 4/10/2019 V. Gyurjyan. CLAS12 software meeting

Legacy software/hardware virtualization Service abstracts software/hardware component Service invoke actions which monitor or change the component state. S S S ClaRA Platform IPC IPC IPC Fortran Program Farm Node Storage 4/10/2019 V. Gyurjyan. CLAS12 software meeting

V. Gyurjyan. CLAS12 software meeting Summary SOA is widely seen as the basis for developing flexible, highly scalable software systems, that can span management and ownership domains, regardless of the hardware and software platforms deployed in each. SOA encourages modularity and encapsulation, and promotes system hierarchical structure. Normative services of the framework are programmed and tested. Legacy software/hardware virtualization as a service has bin developed and tested. System distribution and scalability has bin tested. 4/10/2019 V. Gyurjyan. CLAS12 software meeting

V. Gyurjyan. CLAS12 software meeting Future plans Develop PersistentEvtDataService, capable of handling EVIO data format (based on JEVIO). Define persistent detector data IO, and develop PersistentDetectorDataService. Define persistent statistics data format, and develop PersistentStatisticsDataService. Define transient data format and interface, and develop TransientDataServices. Develop ClaRA specific ontology concepts, used in service description and communications. 4/10/2019 V. Gyurjyan. CLAS12 software meeting