1. The Development of Process Migration and Dynamic Load Balancing in Cluster Middleware
Developer: Thadpong Pongthawornkamol
Advisor: Asst. Prof. Dr. Putchong Uthayopas
Department of Computer Engineering
Faculty of Engineering
Kasetsart University

2. Outline
Goal
Migration Library
Load Balancing Module
Performance Measurement
Problems
Future Work

3. .... Goal Develop Process Migration feature in cluster middleware
Use Process Migration system to build Dynamic Load Balancing service in cluster middleware
Busy
Node
....
process
Middleware

4. Migration Library (libmig)
a C dynamic library developed for process migration purpose
Based on linux operating system
Preemptive process migration
User-space implemented
no kernel modification
User-transparent
no code recompiled or relinked
operate at run time

5. Libmig : mechanism machine machine Process Process signal libmig
checkpoint
restore
libmig
libmig
.sck
file

6. Libmig : example [b4205091@compute2 b4205091]$ migsh ./endless 0
Hello # from process 29842
Hello # from process 29842
Hello # from process 29842
Hello # from process 29842
Hello # from process 29842
b ]$ kill -s SIGUSR
b ]$ ls .migrate/
compute2.cpe.ku.ac.th sck
b ]$
b ]$ rsh compute3
Last login: Mon Feb 24 10:38:46 from compute10
b ]$ migsh ./endless 1 ./.migrate/compute2.cpe.ku.ac.th sck
Hello # from process 3457
Hello # from process 3457
Hello # from process 3457
Hello # from process 3457

7. Libmig : function interception
Libmig intercepts user’s program by using “LD_PRELOAD” shell variable.
“LD_PRELOAD” is used to specify dynamic library that will be loaded before running program
Set “LD_PRELOAD=libmig.so” command will automatically load migration library into user’s program

8. Libmig : function interception (cont.)
User
program
Hey, Mr. OS
Where’s __libc_start_main() ?
libmig.so
__libc_start_main() is in me!!
libc.so.6
Ouch!!!
Migration library inject itself into user’s program via __libc_start_main().

9. Libmig : manage stack Checkpoint stack Restore stack
Find top of stack address
Use the last local variable address in top of stack function.
Read memory from top of stack address to address 0xC
Write stack into checkpoint file using fwrite().
Restore stack
Read stack from checkpoint file using fread().

10. Libmig : manage registers
Checkpoint registers
Save the stack context/environment, including register, by setjmp() function , which will write all register information into sigjmp_buf type variable
Write sigjmp_buf variable into checkpoint file
Restore registers
Read sigjmp_buf variable from checkpoint file
Restore stack using longjmp() command

11. Libmig : manage memory Checkpoint memory Restore memory
Receive memory information from memory map file ( /proc/{process ID}/maps )
Read all memory region except stack (the last line of map file)
Restore memory
Create new memory region using mmap
Map new memory region to “/dev/null” device

12. Libmig : manage file table
Checkpoint file descriptor table
Get file descriptor table from /proc/{process ID}/fd directory
Read file name
Read file offset using fseek() command
Restore file descriptor table
Read file name and offset from checkpoint file
Open file and use fseek() to set file descriptor to appropriate offset

13. Load Balancing module
a dynamic load balancing component built in KSIX cluster middleware
Work with migration library
KSIX provide process migration service via API (kx_rspawnIO() , kx_migrate()) or shell command (ksixexec , ksixmigrate)

14. LB thread A thread created by KXD for load balancing purpose
Wake up every constant time to check node’s load and migrate process to other node if necessary
Node
KSIX Daemon
KXD
LB thread
process

15. LB thread : policy
wake up every constant time
compare local load to global load average
If local load is more than load average, LB will act as a requester, find apropriate node and send migration request to that node
If local load is less than load average, LB will act as an acceptor, choose a request it receives and migrate process from request node to its node.

16. Performance measurement
submit povray processes into a node in cluster
Measure task completion time (min : sec)
#nodes
#processes 1 2 3 4 5
10:02
8:09
7:34
7:07 10
15:00
10:17
10:07
8:55 20
26:10
15:50
11:35
11:00

17. problem Migration library Load Balancing module
Process information access and control problem due to user-space implementation
Require several techniques to solve
Load Balancing module
Merge into KSIX : stability and dependency problem
Future design suggest module separated from KSIX

18. Future work Migration library LB module
Migrate socket, multithread process
LB module
Separate from KSIX
KXLB (another daemon)
Policy adjustment via API or config file
Use KSIX service layer API to migrate process
Develop advanced policy and measure performance

19. The End
Any Question ?

20. LB module flowchart Wake up Compare local load to global load average
Is local load more than
global load average ?
Send request to
apropriate node
Choose request
From busy nodes
Migrate remote
process to
local node
Sleep for
Constant time no
yes