1. Shared Memory Programming on SCI-Connected Clusters Lab Session SCI Summer School 2001 Trinitiy College Dublin Joachim Worringen Lehrstuhl für Betriebssysteme

2. SCI Summer School 2001 – DublinLehrstuhl für Betriebsysteme Software Setup Frontend scripts are: smicc to compile SMI applications smirun for launch SMI applications Pathes should be working: verify with type smicc Copy examples to your home directory: cp -r ~worringen2k/lab. cd lab/SMI Open a shell window, from there: xemacs &

3. SCI Summer School 2001 – DublinLehrstuhl für Betriebsysteme SMI Experiments First steps towards SMI: helloworld Compiling & starting SMI applications Shared Memory Programming: Establishing different layouts Access characteristics Passing pointers Synchronization

4. SCI Summer School 2001 – DublinLehrstuhl für Betriebsysteme helloworld cd helloworld Compiling the programm: smicc helloworld.c –o helloworld Starting the programm: smirun –np 2 helloworld Starting the programm & see what happens: smirun –np 2 –v helloworld (-verbose)  no „machine file“ found! Create a machine file!

5. SCI Summer School 2001 – DublinLehrstuhl für Betriebsysteme machine file List of nodes/hosts on which SMI programs can be run – looks like: pc628 pc629 Create a matching machinefile for your 2 nodes! Locations where smirun looks for it: -machinefile filename./machines $(HOME)/.machines Copy your machine file to ~/.machines If not found: All processes on the local host Special option: –nodes,node0,...,nodeN-1

6. SCI Summer School 2001 – DublinLehrstuhl für Betriebsysteme More smirun Options Verbose startup: -v Output to seperate terminal windows: -xterm ( xhosts & DISPLAY need to be set correctly) Pipe output through a pager: -pager (can be configured via env. variables) Redirect input / output: -stderr filename, -stdout filename, -stdin filename Separate file generated for each process

7. SCI Summer School 2001 – DublinLehrstuhl für Betriebsysteme Access Speed cd regions; make Creates different memory layouts: each process sweeps over complete region  measures access time for each process Region types to be used: SMI_SHM_UNDIVIDED SMI_SHM_BLOCKED SMI_SHM_CUSTOMIZED load test_regions.c Variations: Change write to read access Perform accesses with different strides (gaps between accesses)

8. SCI Summer School 2001 – DublinLehrstuhl für Betriebsysteme Passing Pointers Addresses within shared memory regions: SISCI: different start address for each process  passing of pointers not possible! SMI: identical start address for each process possible Example: pass_pointer.c: Passing the address of a string make, than run pass_pointer – result ? Test with SMI_SHM_NONFIXED flag - result ?  Adaption required: Pass offset relative to region address, not the absolute address!

9. SCI Summer School 2001 – DublinLehrstuhl für Betriebsysteme Synchronization Techniques (I) Locks: protection of objects (data structures) sequentialization of code segments Polling Barriers: Collective synchronization (all processes need to participate) Polling

10. SCI Summer School 2001 – DublinLehrstuhl für Betriebsysteme Synchronization Techniques (II) Progress counters: Individual synchronization possible Waiting for certain state of a process Polling Signals: Individual synchronization possible Not polling  Callbacks possible

11. SCI Summer School 2001 – DublinLehrstuhl für Betriebsysteme Example: Linked List A linked list is created in shared memory Process 0 inserts 50 entries Process 1 prints all entries 50 entries (1... 50) should appear on screen in order cd sync; make Run shmem_list with 2 processes – result ? Introduce proper synchronization using - Locks ( SMI_Mutex_init, SMI_Mutex_lock, SMI_Mutex_unlock ) - Barriers ( SMI_Barrier ()) - Progress counters

12. SCI Summer School 2001 – DublinLehrstuhl für Betriebsysteme Copying memory Different types of memory copying techniques achieve best results in different situations SMI_Memcpy() takes care of chosing the right technique, BUT: being smart costs time hints by the user help to save time Utilizing DMA for asynchronous operation: SMI_Imemcpy() starts/enqueues operation SMI_Memwait() waits for completion very low CPU load

13. SCI Summer School 2001 – DublinLehrstuhl für Betriebsysteme Membench cd membench; make results in membench.out Options: unidirectional write: no options bidirectional write: -b results in membench.out.x (for each process) read instead of write: -r DMA instead of PIO: -a use top to compare CPU load with/without -a

14. SCI Summer School 2001 – DublinLehrstuhl für Betriebsysteme Sort Sorting an array of integers: Each process sorts a part of the global array Sorted parts of the array are merged: Intermitting merge direction: from upper / from lower rank For each two processes merging: Lower process merges lower halfs of two sorted parts Upper process merges uppers halfs Merging is repeated numproc times  All numbers move up/down to their final locations

15. SCI Summer School 2001 – DublinLehrstuhl für Betriebsysteme Merging via Shared Memory Different ways of merging possible: Direct access to other processes‘ array Exchange data via memcpy() operations

16. SCI Summer School 2001 – DublinLehrstuhl für Betriebsysteme SHM Regions: Creation Speed Common situation: each process exports a region; then imports all other regions Simple approach: For N processes, create N SMI_SHM_UNDIVIDED regions load measure_undivided.c make measure_undivided Advanced approach: create a single SMI_SHM_FRAGMENTED region load measure_fragmented.c make measure_fragmented Compare duration of creation!