The Publisher-Subscriber Interface Timm Morten Steinbeck, KIP, University Heidelberg Timm Morten Steinbeck Technical Computer Science Kirchhoff Institute for Physics University Heidelberg

The Problem Timm Morten Steinbeck, KIP, University Heidelberg  Create an interface to transport event data between several analysis steps needed for the Alice Level 3 Trigger.  Communication is presumed to be primarily local.  Transport of actual data should be kept to a minimum - only descriptors should be sent between processes.  One data source should support multiple destinations.  Two kinds of data destinations: Data processing and monitoring.

The Approach Timm Morten Steinbeck, KIP, University Heidelberg  Two kinds of processes: Publisher - Subscriber.  Subscribers announce interest in events to publisher (Subscription).  Publisher informs subscribers of new events when they arrive.  Subscribers inform the publisher when they are done working on the event.  Event is freed when all subscribers are finished with it. Subscriber Publisher Subscriber Publisher Subscriber Publisher Subscribe Event Done New Event

Publisher-Subscriber Principle Timm Morten Steinbeck, KIP, University Heidelberg Subscriber Publisher Subscriber Publisher Subscriber Publisher Subscribe Event Done New Event

Publisher-Subscriber Communication Timm Morten Steinbeck, KIP, University Heidelberg  Communication between publisher & subscriber via named pipes.  Event data is kept in shared memory area.  Only event descriptors are sent via named pipes.  Descriptors describe location of data in shared memory.  Event descriptor can describe multiple datablocks. Publisher Shared Memory Event M Data Block n Data Block 0 New Event Event M Descriptor Block n Block 0 Subscriber

Event Descriptors Timm Morten Steinbeck, KIP, University Heidelberg Event descriptors contain  The ID of the event  The number of data blocks described For each data block:  The size of the block  The shared memory ID  The starting offset in the shared memory  The type of data  The ID of the originating node Shared Memory Event M Data Block 0 Event M Data Block n Event ID: M Block count: n Block n  Shm ID  Offset  Size  Datatype  Producer ID Block 0  Shm ID  Offset  Size  Datatype  Producer ID

Persistent and Transient Subscribers Timm Morten Steinbeck, KIP, University Heidelberg Two kinds of subscribers supported: Persistent and Transient  Persistent subscribers get all events.  Publisher frees events only when all persistent subscribers are finished.  Transient subscribers can specify event selection criteria (Event number modulo, trigger words).  Transient subscribers can have events canceled by the publisher. Subscriber Publisher Subscriber Publisher Cancel Event New Event Publisher Subscriber Publisher Subscriber Publisher Event Done New Event Free EventFree Event Publisher Free EventFree Event

Analysis Components Timm Morten Steinbeck, KIP, University Heidelberg Analysis processes made up of three parts:  Subscriber as data input for new event data.  Processing code that works on the data in shared memory and possibly writes some new output data into shared memory.  Publisher to make the resulting output data available. Subscriber Publisher Analysis Code Shared Memory New Event Event Input Data Event Output Data New Event Read data Write data

Data Transport Components Timm Morten Steinbeck, KIP, University Heidelberg A number of components exist to transport event data and to build an analysis chain:  Event Scatterer distributes events in round-robin fashion for load balancing.  Event Gatherer merges blocks from multiple event data streams into one event.  Event Collector merges multiple event data streams into one stream.  Bridge transports event data over network between computers.

Event Scatterer Timm Morten Steinbeck, KIP, University Heidelberg  One subscriber input.  Multiple publisher outputs.  Incoming events are distributed to output publishers in round-robin fashion. Subscriber Publisher New Event New Event Subscriber Publisher New Event New Event

Event Gatherer Timm Morten Steinbeck, KIP, University Heidelberg  Multiple subscribers attached to different publishers.  One output publisher.  Data blocks from incoming events are merged into one outgoing event. Publisher Merging Code Subs Shared Memory Event M Block 1 Event M Block 0 Event M Block 0 Event M Block 1 Event M Block 0 Block 1

Event Collector Timm Morten Steinbeck, KIP, University Heidelberg  Multiple subscribers attached to different publishers.  One output publisher.  Each incoming event is published unchanged by output publisher. Publisher Subs New Event Publisher Subs New Event

Bridge Timm Morten Steinbeck, KIP, University Heidelberg Bridge consists of two programs:  SubscriberBridgeHead  Contains subscriber.  Gets input data from publisher.  Sends data over network.  PublisherBridgeHead  Reads data from network.  Publishes data again. Node B Data consumer Publisher Subscriber Bridge Network Code New Event Node A Data producer Publisher Subscriber Bridge Network Code New Event Network data transport

Bridge Networking Timm Morten Steinbeck, KIP, University Heidelberg  Bridge networking code is based on abstract communication classes.  Different communication classes for (short) message transfers and (big) datablock transfers.  Prototype implementation currently exists for SCI.  Further implementations possible for TCP/IP, Scheduled Transfer Protocol STP, raw (Gigabit) Ethernet, other (future) system area networks.

TCP/IP Bridge Timm Morten Steinbeck, KIP, University Heidelberg Second approach exists for TCP/IP bridging:  Publisher-Subscriber interface ported to TCP (from named pipes + shared memory) by Paul Starzetz.  Pipe-TCP Bridge application can convert between pipe-shm and TCP publisher-subscriber code. Node B Data consumer Pipe Publ Pipe Subs Bridge New Event Node A Data producer Pipe Publ Pipe Subs Bridge New Event TCP New Event TCP Publ TCP Subs

Analysis Chain Framework Timm Morten Steinbeck, KIP, University Heidelberg  Building of arbitrary analysis chains possible.  Chain setup determined at program start.  No recompilation of source code needed.  Chain functionality can be tested on one node.  Performance tests or production runs on multiple nodes.

First Benchmarks Timm Morten Steinbeck, KIP, University Heidelberg Benchmarks of basic publisher-subscriber communication via named pipes:  Platform: 733 MHz dual Pentium III PC.  Minimal event data size (1 block, 4 byte).  One publisher, no subscriber: 95  s/msg.  One publisher, one subscriber, no data processing * : 1,3 ms/msg. * But full descriptor processing.

Status Timm Morten Steinbeck, KIP, University Heidelberg  Implementation of most needed components finished.  Now testing and debugging code.  Further performance tests.  Also integration tests with analysis code in progress.