WING: A Consistent Computing Platform Yiwei Ci 22/02/2012

Goals Give processes suitable places for computing. Take processes to the place suitable. Transparency – OS abstraction (SSI). – Hardware abstraction supports heterogeneous system Seamless – Dynamically switch workplace.

Challenges Security – Prevent malicious software? Availability – The availability of each computer is high today. How about the whole distributed system? Scalability – Provide just-in-time scalability?

Application scenarios Build your data center. – Exchange between data migration and computation migration. Build your virtual cluster. – PC + PC – PC + Mobi – Mobi + Mobi Build your test bed. – Easy deployment

Related works XtreemOS – Grid operating system supporting virtual organizations. – Latest version: 3.0 (2012/02/10) OpenSSI – Provide single process space, single I/O space, single IPC space, and single root for the cluster. – Latest version: (2010/02/18) OpenMosix – A Linux kernel extension for single-system image clustering – Latest version: (closed)

Related works OpenSSIOpenMosixKerrighed Single Administrative DomainYesNoYes Cluster Membership GuaranteesYesNo High-Availability (HA) ClusteringYesNo Single Process Management SpaceYesNoYes Process MigrationFullPartial Process Load BalancingYes Process Migration is HAYesNo Migrate Processes Using SemaphoresYes Migrate Processes Using Shared MemoryYesNoYes Single Thread MigrationOnly thread groupNo Process Checkpointing3rd party Yes Single IPC NamespaceYesNoYes

Related works OpenSSIOpenMosixKerrighed Distributed Shared MemoryMigratesNoYes Single PTY NamespaceYesNo Single-Site File NamingYesNo ? Coherent Cache / File AccessYes ? ? Migrate Active FilesystemExperimentalNo HA Cluster Filesystem (CFS)YesNo HA Single Cluster Name/AddressYesNo HA Network Load Balancer (LB)YesNo LB Auto Detects TCP/UDP IP ServicesYes (UDP 1.9.1)N/A Transparent Socket MigrationNot yetNoYes Automatic Service FailoverYesNo Diskless Nodes via Network BootYes (PXE/EtherBoot)NoYes

Outline Design Implementation Results Conclusions

Design Virtualization (1)

Design Virtualization (2)

Design Virtualization (3)

Design Virtualization (4)

Design Virtualization (5)

Design Basic Architecture

Design Metaphor Object Operation System callPathFile operation …/class_key_entry op id flags Metaphor: everything is a file Subject object subject

Design Resource – Computation resource Process – Data resource File Memory – Common resource IPC: msg, sem, shm …

Design Area Basically, an operating system has two areas for different processes. – Local area: local computing. – Global area: distributed computing. /mnt/repo The resources in local area are invisible to the processes in global area. The resources in global area are consistent.

Design Isolation – Process Isolation Processes in WING are “invisible” to the world outside. – Data Isolation Data in WING is “invisible” to the world outside. The private data of one user is “invisible” to other users. – Private files. – The private data inside a file.

Implementation Components WTRANS: kernel translator WLN: linker WRES: common resource WCR: checkpoint kernel module CFS: checkpoint file system WMAN: process manager LDFS: lightweight distributed file system IFS: image file system

Implementation Checkpoint Record the status of a process (transparent to the programmer) – Registers – Opened files – Signals – Credentials – Memory – IPC

Implementation Checkpoint – Incremental saving – Dynamic recovery – Dedicated file system

Implementation Distributed IPC message 、 semaphore 、 shared memory – System V IPC does not support distributed communication. – It is complex to dump and to restore the IPC status in kernel.

Implementation Distributed IPC – Coexist of conventional System V IPC and distributed IPC (the same interface). – Ensure the consistency of IPC resources in the distributed environment. – Stateless in kernel (for process migration) Re-implement IPC in user-space. provide a pseudo file system to store the status of the IPC resource (RAM based). – High availability

Implementation Distributed IPC – Event Driven Event-Flow

Implementation Msg: Requirements: 1) msgtyp = 0 ： the first message on the queue is received. 2) msgtyp > 0 ： the first message of type msgtyp is received. 3) msgtyp < 0 ： the first message of the lowest type that is less than or equal to the absolute value of msgtyp is received. Sem Requirements: 1) RPC has a time-out mechanism (for semtimedop). 2) RPC has an undo mechanism (for exit_sem).

Implementation Shared memory Problems: – Find (key, value) pair – Frequently update of (key, value) Consistency model: – Sequential consistency Features: – Multi-owner – Versioning – Write invalidation

Implementation Shared memory Handle shm_fault

Implementation Shared memory Handle shm_fault

Implementation Image File System (IFS) Data can be shared, but the data for each user needs to be protected. – Each user can have a different view of a file. – The processes of the same user have the same view of a file.

Implementation Image File System

Results Environment: – VM (x2): 512MB RAM, 2 processors, NAT – OS: Based on kernel – Host: 2048MB RAM, Intel Core 2 Duo CPU T6570 Experiments: – Msg – Sem

Results Msg Leader: 1.Use msg( key 0 ) to collect start requests from members; 2.If all start requests are received then 3. Use msg( key i ) (i > 0) to send start i ; 4. Use msg( key N+1 ) to collect stop requests; 5. if all stop requests are received then 6. return success; 7.return fail; Member i: 1.Use msg( key 0 ) to send start request to leader; 2.If start i is successfully received by msg( key i ) then 3. create_process(msg_snd); 4. create_process(msg_rcv); 5. if msg_snd and msg_rcv are finished then 6. Use msg( key N+1 ) to send stop request to leader; Process msg_snd 1.for n = 1 to ROUNDS do 2. Use msg( key 0 ) to receive req; 3. Build mtext (|mtext| = req.msgsz); 4. i := req.src; 5. Use msg( key i ) to send mtext; Process msg_rcv at member i 1.for n = 1 to ROUNDS do 2. req.msgsz := rand() % MSG_SIZE + 1; 3. req.src := i; 4. Use msg( key 0 ) to send req; 5. Use msg( key i ) to receive mtext; 1 leader, 1 member, ROUNDS = 1000, MSG_SIZE = 128, time = sec

Results sem 1 leader, 1 member, ROUNDS = 1000, time = sec Leader: 1.Create sem( key 0 ) (sem( key 0 ) has N items); 2.Assign each item of sem( key 0 ) to 0; 3.If all items of sem( key 0 ) are not 0 then 4. for i = 1 to N do 5. remove sem( key i ); Member i: 1.Create sem( key i ); 2.If all sem( key j ) (j ≠ i) are created then 3. for n = 1 to ROUNDS do 4. k := rand() % N + 1; 5. down(sem( key k )); 6. up(sem( key k )); 7. endfor 8. sem( key 0 ).item i := 1;

Conclusions By operating system virtualization, WING provides processes in different nodes with consistent views of the distributed resources. There are no additional libraries required. Conventional multi-task applications can be used for distributed computing.

Future work Components – WMAN: process manager – LDFS: lightweight distributed file system Tools: – Profiler – Test suits Stability Security

Thanks

Q&A