of Rostock University DuDE: A D istributed Computing System u sing a D ecentralized P2P E nvironment The 4th International Workshop on Architectures, Services and Applications for the Next Generation Internet (WASA-NGI-IV) Bonn, Germany, October 4th, 2011 J. Skodzik, P. Danielis, V. Altmann, J. Rohrbeck, D. Timmermann University of Rostock, Germany Institute of Applied Microelectronics and Computer Engineering T. Bahls, D. Duchow Nokia Siemens Networks Broadband Access Division Greifswald, Germany

Outline Introduction & Motivation DuDE in General The DuDE Algorithm in Detail Test Scenario and Evaluation Summary and Future Work 2

Increasing number of Internet users and traffic data Internet Service Providers (ISPs) want to ensure: Quality of Service (QoS) The detection of bottlenecks The detection of attacks How to ensure these issues?  Statistics generated from existing log data Situation today 3 Does an AN have enough resources? Does it provide sufficient statistics at all?

60% Introduction & Motivation 4 CPU MEM Resources utilization

Simple support of new statistics types Simultaneous computation of multiple statistics Processing of increasing log data volumes Processor utilization RAM utilization Drops Number of packets Short term statistics (STS) for single ANs Supported Introduction & Motivation 5 Not supported Long term statistics (LTS) Creation of global statistics

One AN does not have enough hardware ressources  Usage of multiple ANs to compute statistics Efficient resource sharing with high resilience and scalability  Utilization of P2P technology DuDE: Exploitation of already available resources  No extra costs for additional equipment Introduction & Motivation 6

DuDE in General Logical P2P ring ID 7 Node1 Node2 Node4 Node3

DuDE in General 8 Log data (some hundreds of KBs) 8 Data chunk (ca. 100 KBs)

DuDE in General Objective: High log data availability = % Simple replication wastes memory ressources  Reed-Solomon Codes Split log data of each AN into m data chunks 9

DuDE in General Objective: High log data availability = % Simple replication wastes memory ressources  Reed-Solomon Codes Split log data of each AN into m data chunks Encoding: Add k interleaved coding chunks  n=m+k chunks 10

DuDE in General Objective: High log data availability = % Simple replication wastes memory ressources  Reed-Solomon Codes Split log data of each AN into m data chunks Encoding: Add k interleaved coding chunks  n=m+k chunks Decoding: Restore log data from any m of n chunks 11

DuDE in General 12 Log data (some hundreds of KBs) Data chunk (ca. 100 KBs) How to apply our application to P2P?

Task Job = collection of STS and/or LTS tasks Task = part of job, e.g., request for „CPU“ statistics Jobscheduler (JS): Reception and monitoring of job Taskwatcher (TW): Reception and processing of task DuDE in General 13 Admin. Job Which steps are necessary to compute statistics?

60% 50% 30% 10% The DuDE Algorithm in Detail Stage 1: Resource collection 14 Admin. … Job … Task … Log data … Global statistics

The DuDE Algorithm in Detail 1. Resource collection 15 Admin. Stage 2: Jobscheduler determination … Job … Task … Log data … Global statistics

The DuDE Algorithm in Detail 16 Admin. 2. Jobscheduler determination Stage 3: Resource re-collection 1. Resource collection … Job … Task … Log data … Global statistics

The DuDE Algorithm in Detail 17 Admin. 3. Resource re-collection Stage 4: Task assignment 2. Jobscheduler determination 1. Resource collection Request for Processor utilization STS Request for RAM utilization LTS Request for Drops LTS … Job … Task … Log data … Global statistics

The DuDE Algorithm in Detail … Job … Task … Log data … Global statistics 18 Admin. 3. Resource re-collection Stage 4: Task assignment 2. Jobscheduler determination 1. Resource collection How to find all log data for global statistics computation?

Node1 Node2 Node3 Node5 no The DuDE Algorithm in Detail Request for global statistics  All data needed ID Taskwatcher iSucc?NN >=AID yes no yes Node1 Node2 Node3 Node4 Node5 Algorithm is done Global Peer Data Discovery Algorithm - Threshold value A = 2 Algorithm is done 5yes0no Node5 61no Node6 7yes2 Node7 19

The DuDE Algorithm in Detail 20 Admin. 4. Task assignment Stage 5: Log data collection 3. Resource re-collection 2. Jobscheduler determination 1. Resource collection … Job … Task … Log data … Global statistics

The DuDE Algorithm in Detail 21 Admin. 5. Log data collection 4. Task assignment 3. Resource re-collection 2. Jobscheduler determination 1. Resource collection Stage 6: Statistics computation Processor utilization stat. RAM utilization stat. Drops stat. … Job … Task … Log data … Global statistics

The DuDE Algorithm in Detail 22 Admin. 5. Log data collection 4. Task assignment 3. Resource re-collection 2. Jobscheduler determination 1. Resource collection Stage 7: Send results and display them 6. Statistics computation Admin. … Job … Task … Log data … Global statistics

The DuDE Algorithm in Detail 23 Admin. 5. Restore log data 4. Assign tasks 3. Resource recollection 2. Determine job scheduler 1. Resource collection Stage 7: Send results and display them 6. Compute statistics Admin. … Job … Task … Log data … Global statistics

Test Scenario and Evaluation 24 PC Configuration Pentium 4 (1.5 GHz) 512 MB RAM  Equivalent to AN HW

Test Scenario and Evaluation Parameters: Number of tasks inside job Number of log data sets in the P2P network Computational load for statistics computation Measurements: Time for finishing a job Memory utilization 25

Test Scenario and Evaluation 26 Linear Increase of Needed Time  Time is Constant

Test Scenario and Evaluation 27 Linear Increase of Needed Time  Time is Constant

Test Scenario and Evaluation Linear Increase of Memory Utilization  Constant Memory Utilization 28

Test Scenario and Evaluation Memory utilization increases more at the single AN than at the taskwatcher 29

Test Scenario and Evaluation Memory utilization is constant and independent of the computational load 30

Summary and Future work P2P-based system for distributed computing of statistics STS and LTS Statistics for a single AN and the whole network Global Peer Data Discovery Algorithm Successfully developed prototype (demo session) Investigation of further use cases 31

Thanks for your attention! Questions? 32