1. TORQUE Kerry Chang CCLS December 13, 2010

2. O UTLINE Torque How does it work? Architecture MADA Demo Results Problems Future Improvements

3. T ORQUE – W HAT IS IT ? Open source project by Cluster Resources Inc. Cluster resource manager Manages batch jobs A series of programs to be executed without manual intervention Manages distributed compute nodes Distributed servers on which to execute batch jobs

4. T ORQUE A RCHITECTURE

5. T ORQUE S CHEDULER Currently using standard built-in schedule (FIFO) MOAB – more advanced scheduler

6. W HAT HAVE I DONE ? Used MADA as an application of TORQUE Treated the application as a blackbox Text parallelization on input Created a series of scripts for text manipulation and job submission to Torque queue Linear improvement in processing time by using Torque

7. MADA System for Morphological Analysis and Disambiguation for Arabic Input file is separated line by line

8. MADA A RCHITECTURE

9. H OW DO THE SCRIPTS WORK ? 1) First split the text file evenly across the number of specified jobs to be submitted 2) Create a script for each newly split text file e.g. If you wanted to run 5 jobs, split the text into 5 files and create a script to run each of the 5 files. 3) Submit each script to Torque 4) Concatenate the output of each script

10. D EMO Demonstration of Torque and MADA 3 Output Files file.bw file.bw.mada file.bw.mada.tok

11. R ESULTS 30 lines

12. R ESULTS 300 Lines

13. R ESULTS 3,000 Lines

14. R ESULTS 30,000 Lines

15. R ESULTS Network – Local Temp comparison (seconds) NetworkLocal TempImprovement 1141086 2011938 1,4771,40077 14,54413,973571 145,105131,64913,456

16. P ROBLEMS How do we know when MADA has finished and we can concatenate the results? Where do we run MADA and have the results output to? Submission to compute node hangs Use smarter scheduler Supply machines dedicated to running Torque jobs

17. F UTURE I MPROVEMENTS Pipeline many jobs to Torque Work from local temp folders instead of on the network Split and rebuild certain output files by looking at provided testing.madaconfig file MADA TOKAN Preprocessor

18. Q UESTIONS