1 Distributed Monitoring of Peer-to-Peer Systems By Serge Abiteboul, Bogdan Marinoiu Docflow meeting, Bordeaux

2 2 Outline  The Monitoring Problem & Approach  A language for specifying monitoring tasks: P2PML  P2PMonitor System  ActiveXML Stream Algebra  Architecture of P2PMonitor  Monitoring Plan Generation & Query Rewriting  Focus on Filtering  Reusing running tasks  Work in progress

3 3 The Monitoring Problem P2P systems are:  a popular support for content sharing communities, distributed applications  highly dynamic (intense communications, content changing rapidly, peers come/leave) and difficult to observe Observation is important !  error management & diagnosis  statistics gathering & optimization issues : the « busiest » peer in a network  business applications : billing & quality of service  Web surveillance

4 4 Is it possible to observe & analyse a P2P system ?  Difficult (if not impossible) in a centralized way  Yes, in a distributed manner

5 5 XML strea m Approach  Detect events at the (monitored) peer level Data changes, Web service calls -> alerters  Each event is represented as an XML document  XML Stream  (distributed) XML stream processing system  XML Streams are published

6 6 Outline  The Monitoring Problem & Approach  A language for specifying monitoring tasks: P2PML  P2PMonitor System  ActiveXML Stream Algebra  Architecture of P2PMonitor  Monitoring Plan Generation & Query Rewriting  Focus on Filtering  Reusing running tasks  Work in progress

7 7 P2PML statement structure XQuery FLWR flavour  For – maps streams to XML variables  Let – assigns new XML variables  Where – imposes conditions on events (filtering and join criteria)  Return – generates reports / restructures XML  By – specifies publication means :  in channels for inside system publication  s, Web pages, RSS feeds for outside system publication

8 8 P2PML statement example for $c on local: outCOM let $timeCall := $c.call.time and $duration := $c.response.time - $timeCall where $c.call.method = “GetTemp” and $duration > 10 and $c.call.site = " return {$timeCall} {$duration} by channel “QoS:Alerts”

9 9 Outline  The Monitoring Problem & Approach  A language for specifying monitoring tasks: P2PML  P2PMonitor System  ActiveXML Stream Algebra  Monitoring Plans  Architecture of P2PMonitor  Focus on Filtering  Reusing running tasks  Work in progress

10 ActiveXML Stream Algebra is the support for monitoring plan representation and the basis for the its optimization :  Distribute the work among the peers  Try to place computation close to data if possible  Try to reduce redundancy

11 Scenario P2PMLQueries XML streams

12 Monitoring Plans

13 Architecture of P2PMonitor(1)  Subscription Manager  Alerters (WS Alerter, Database Alerter, RSS Alerter)  Stream Processors  Without « storage »: Filter, Restructure, Union  With « storage »: Join, Group-By, Duplicate Removal  Publishers  , WebPage, RSS  Channel Publisher : a user or another peer may subscribe to it

14 Architecture of P2P Monitor(2)

15 Outline  The Monitoring Problem & Approach  A language for specifying monitoring tasks: P2PML  P2PMonitor System  ActiveXML Stream Algebra  Monitoring Plans  Architecture of P2PMonitor  Focus on Filtering  Reusing running tasks  Work in progress

16 Focus on Filtering Filtering is a crucial operator in stream processing ! E.g., Many users might be interested in events coming from the same source / alerter : bottleneck hazard Our approach to the problem : two-stage filtering Reasons:  Attributes of XML document’s root reflect the most important properties of an event  The event’s details can be given intentionnally (ActiveXML style)

17 Two-stage filtering A subscription is viewed as a conjunction of simple conditions (e.g., « attribute » = « value ») and « more difficult » XPath queries 1st data structure regroups the (ordered) simple conditions of all the subscriptions by commonalities (Atomic Event Set structure) 2nd data structure regroups XPath queries of all the subscriptions (path – based indexing YFilter style – using NFA) On a XML document: 1st stage: read the root, evaluate AES, detect the « difficult » XPath queries that remain to be evaluated 2nd stage (if needed): adapt the second structure and evaluate the set of XPath queries on the body of the XML document (if necessary execute Web service calls). The output is the set of the subscriptions « hit » by the XML document

18 Outline  The Monitoring Problem & Approach  A language for specifying monitoring tasks: P2PML  P2PMonitor System  ActiveXML Stream Algebra  Monitoring Plans  Architecture of P2PMonitor  Focus on Filtering  Reusing streams / running tasks  Work in progress

19 Reusing running tasks  Optimization by trying to avoid redundancy  Before building new operators (and streams), try to discover useful ones  Stream representation in XML:  Stream Definition Database – description of available streams  Distributed, not centralized (avoid bottlenecks)  Implemented using KadoP – index and repository system over a DHT

20 Stream replication and equivalence(1)  Streams can be replicated between peers  With two similar operators on two replicas of the same stream, we obtain two equivalent streams  Replication can be represented in the Stream Definition Database  Stream Equivalence is difficult to detect

21 Algorithm for discovering useful streams  It uses XPath queries on the Stream Definition Database: E.g. for identifying the output stream of alerter inCOM : 1, P 1 )  It goes bottom-up on the query tree E.g., Join P (σ F 1 ), 2 ) (S 5, P 1 )

22 Outline  The Monitoring Problem & Approach  A language for specifying monitoring tasks: P2PML  P2PMonitor System  ActiveXML Stream Algebra  Monitoring Plans  Architecture of P2PMonitor  Focus on Filtering  Reusing streams / running tasks  Work in progress

23 Work in progress (1)  Link with Incremental View Maintenance Defining a monitoring task by a tree-pattern query on an active document with streams - powerful way of expressing complex monitoring tasks (difficult to express directly in P2PML)

24 Work in progress (2)  Introducing explictly the « time » in P2PML -possible impact on P2PMonitor performance (reactivity) and resource consumption (needed storage) E.g. for $e1 on P1:inCOM, $e2 on P2:outCOM where $e1.timeEvent > $e2.timeEvent +25 …  Queries on traces obtained by P2PMonitor – diagnosis, detecting patterns of evolution for the monitored system E.g. Trace = I 1,I 2 …I n - instances of a document For each new order detected in instance I k, there is a payment present in one of the following instances

25 Thank you very much!