Distributed Systems Laboratory www.cs.technion.ac.il/Labs/dsl 9/22/2018 MSR HPC visit

Lab People - Faculty Prof. Ran El-Yaniv (Learning, Data Mining) Prof. Roy Friedman (Distributed Systems, Ad hoc Networks) Prof. Erez Petrank (Memory Management) Dr. Avi Mendelson (Computer Architecture) Prof. Assaf Schuster, HEAD (Large-Scale Data Processing, Distributed Systems) 9/22/2018 MSR HPC visit

Lab People Engineers: Eran Issler, Max Kovgan, David Carmeli, Valentin Kravtchov, Artiom Sharov About 40 graduate research students (best of breed!) Dozens of undergraduate and graduate students working on projects each semester Hundreds of undergraduate students in systems courses 9/22/2018 MSR HPC visit

Sponsors and Partners 9/22/2018 MSR HPC visit

Scope Applications Middleware, Virtualization Hardware 9/22/2018 Large Scale Distributed Data Mining Grid/P2P/Sensor Data Mining Genetic Linkage Analysis Applications Distributed Scalable Model Checking Anonymous and Private distributed Data Mining Machine Learning Sensor Networks Internet Mining Light-weight group communication Fast interconnects for HPC and data processing Condor – Grid Computing – Research, development, deployment Software Distributed Shared Memory System Services for Ad-hoc networks Middleware, Virtualization Locality in large-scale computations Data Privacy in Distributed Databases Multilevel caching in storage systems Scalable Data Race Detection Highly Available Distributed Java Computer Architecture: Fine Grain Parallelization Hardware 9/22/2018 MSR HPC visit

The Resource Hierarchy GLOW - UW Madison Boinc @HOME 9/22/2018 MSR HPC visit

EGEE 9/22/2018 MSR HPC visit

DSL users Dr. Avi Mendelson – Trace cache Prof. Ran El Yaniv – Machine Learning Prof. Roy Friedman – Group Communication Prof. Assaf Schuster – Large scale and grid Prof. Eli Biham – Cryptography Prof. Dan Geiger – Genetic Linkage Analysis Prof. Orna Grumberg – Scalable Model Checking Prof. Uri Weiser – Computer Architecture Prof. Ron Pinter – Caching Architectures Prof. Ronny Kimmel – 3D Image processing Prof. Reuven Cohen – Communication Networks Prof. Danny Raz – Active Distributed Services Prof. Idit Keidar – Distributed Systems Prof. Mooly Sagiv – Compiler Analysis Prof. Shaul Markovitch – Machine Learning Prof. Yoram Rosen – High Energy Physics …. 9/22/2018 MSR HPC visit

Contents - Tools Multiview – Distributed Shared Memory Data race detection Model checking-based DRD Grid Monitoring System Decorative HA for grids 9/22/2018 MSR HPC visit

Contents – Large-Scale Distributed Systems Peer-to-Peer Data Mining DataMiningGrid project QosCosGrid project Distributed runtime for multithreaded Java Distributed Model Checking

Multiview – Technologies for Distributed Shared Memory [OSDI’99] 9/22/2018 MSR HPC visit

See Multiview in a separate presentation 9/22/2018 MSR HPC visit

Data Race Detection for C++ Programs [PPOPP’03] 9/22/2018 MSR HPC visit

See MultiRace in a separate presentation 9/22/2018 MSR HPC visit

Model Checking-Based Data Race Detection [PPOPP’05] 9/22/2018 MSR HPC visit

Difficulties in model checking dataraces Infinite state space Huge number of interleavings Huge transition systems Size problem 9/22/2018 MSR HPC visit

Basic idea 9/22/2018 MSR HPC visit

hybrid solution Combine Lockset & Model Checking Provide witnesses for dataraces Rare dataraces Dataraces in large programs Model Checking Provide witnesses for rare DR + Lockset scale for large programs 9/22/2018 MSR HPC visit

Multi-threaded program Idea and Prototype Multi-threaded program List of Warnings Violations of locking principle Lockset Access suspicious of racing Find a1 Extend 1 Wolf Model checker 1 snapshot witness 2 9/22/2018 MSR HPC visit

Benchmark programs Lines Description Program 706 Tsp 708 Our_tsp 3751 traveling salesman from ETH Tsp 708 Enhanced traveling salesman Our_tsp 3751 Multithreaded raytracer from specjvm98 mtrt 29948 Web Crawler Kernel from ETH Hedc 362 Parallel sort SortArray 129 Finds prime numbers in a given interval PrimeFinder 150 Elevator simulator Elevsim 166 Shared DB simulator DQueries 9/22/2018 MSR HPC visit

Experimental results 4 threads 3 threads 2 threads Program our_tsp Memory (MB) Time (sec) Time (sec) Mem Out 353 35069 our_tsp 396 1334.93 123 569.3 SortArray 168 4547.1 143 2645.5 116 888.7 PrimeFinder 48 147.9 33 67.92 28 33.02 ElevSim 136 585.97 89 201.8 60 140.1 DQueries 17 9 12 7.33 11 2.66 Hedc Out Mem 377 35243 tsp 9/22/2018 MSR HPC visit

Mining for Misconfigured Machines in a Grid System [KDD’06] Tested with success on a production environment. 9/22/2018 MSR HPC visit

Grid Batch Systems Many potential causes of failures and misbehaviors Many organizations or administration sites. 10000s machines Heterogeneous machines Non dedicated Different installation and configuration Many potential causes of failures and misbehaviors Software bugs, hardware, network , configuration Current solutions Manual diagnosis Ruled based expert system. Data mining Limited, if any, prior knowledge Submission Resource broker Execution 9/22/2018 MSR HPC visit

Data Acquisition Data collector Data collector Data miner Data miner Non-intrusive Distributed Database Preprocessing Data miner Distributed Data miner Data collector 9/22/2018 MSR HPC visit

Distributed Outlier Detection 9/22/2018 MSR HPC visit

Distributed Outlier Detection 9/22/2018 MSR HPC visit

Distributed Outlier Detection 9/22/2018 MSR HPC visit

Distributed Outlier Detection 9/22/2018 MSR HPC visit

Distributed Implementation SG3 S1 SG2 S2 S3 2 1 1 1 SG 9/22/2018 MSR HPC visit

Distributed Implementation SG3 S1 SG2 S2 S3 2 3 1 1 SG 9/22/2018 MSR HPC visit

Distributed Implementation SG3 SG1 S1 SG2 S2 S3 3 1 SG 9/22/2018 MSR HPC visit

Evaluation on DSL Hardware 3 of the top 4 suspected machines are actually misconfigured. bh10: unknown reason. i4: loaded by network service. bh13: active HyperThreading. i3: root file system was nearly full. 9/22/2018 MSR HPC visit

Future Work Fault identification, analysis, classification, prediction. Better resource allocation; better system utilization Feedback to user on submitted jobs description Optimizing transparent operation Collaboration with INTEL NetBatch team 9/22/2018 MSR HPC visit

HA for large scale grids [HPDC’06] Production System – Condor distribution 9/22/2018 MSR HPC visit

The Challenges WAN backups Lightweight protocols Autonomous partitions Failure detection is not perfect - no bounded delay Network anomalies - links are asymmetric, not transitive IP fail-over techniques inapplicable Lightweight protocols Traditional Group Communication algs do not scale well Autonomous partitions Transient failures Legacy applications without HA Grid developers do not want to deal with HA Random, uniformly chosen, partial membership Provides random representative in every netwsork part. 9/22/2018 MSR HPC visit

The Goal The goal is to turn HA into a commodity Decoration “HA out of the box” No need to change or adapt your existing service HA is provided as a Grid service itself Solution: Decoration Transparent addition of HA to already existing and deployed services No changes to the decorated service 9/22/2018 MSR HPC visit

Application: HA for Condor Central Manager Job queue machine Job queue machine Central Manager Collector Negotiator Execution machine Execution machine Job queue machine Job queue machine Execution machine 9/22/2018 MSR HPC visit

Solution Architecture 9/22/2018 MSR HPC visit

Solution Highlights HAInvocator - High Availability for Negotiator Leader election Automatic failure detection Transparent failover to backup “Split brain” reconciliation after network partitions HAReplicator - Persistency of Negotiator state State replication between active and backups Proxy for multicasting client’s messages to Collector Loose coupling between replication and HA 9/22/2018 MSR HPC visit

Status Passed **testing** in 2005 Not a single code line of Condor changed Except for several bug fixes  Inside Condor distribution effective Version 6.8 Some important clients Some success stories On-going collaboration with the Condor team 9/22/2018 MSR HPC visit