1. Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz

2. Introduction Welcome to my workshop, where we will explore scripted job submission with Grisu Jython Why script job submission? Introducing Grisu Jython Practical Exercises – Trivial HelloWorld – Submitting multiple jobs using lists – Using BatchJobObject to reduce overheads

3. Why script job submission? “I’ve got a bad feeling about this...” - various, the Star Wars franchise

4. BeSTGRID Jobs The Grisu clients are used to submit jobs to compute resources via the BeSTGRID Jobs service. Usually with the Grisu Template Client: Convenient GUI for job submission File management Templates for common applications Detailed job settings can be exposed

5. Limitations of Grisu Template Cilent The Grisu Template Client is not suitable for all kinds of jobs: Batches of many jobs Automated job submission Jobs that require local processing to prepare job items Jobs that require complex submission parameters

6. Batches of Many Jobs The BeSTGRID Jobs service can handle hundreds, or even thousands of job submissions, but doing this with the Grisu Template Client is mind numbingly repetitive and requires inhuman patience. SOLUTION: Write a script or application that iterates through all the jobs, submits them, and then retrieves the output.

7. Automated Job Submission The Grisu Template Client is a human interface and can not submit jobs without human interaction. Not useful for submitting jobs from another application or service SOLUTION: Write a script or object that can submit a job from within an application, or can be called as a scheduled task.

8. Jobs Requiring Pre-processing A job may require a number of steps to be performed before a job is even entered into the Grisu Template Client for submission. This may include chaining multiple jobs together. SOLUTION: Write a script that performs pre- processing steps as part of the job submission process

9. Complex Job Parameters A job may require specific parameters, such as: Specific submission sites Specific applications and versions Specific selection and partitioning of resources Which may be difficult to repeat reliably using a human interface SOLUTION: Write a script that sets the parameters exactly every time

10. All of the Above Complex job submission may exhibit any and all of the features discussed previously, and the human interface provided by the Grisu Template Client becomes cumbersome and limits job submission. These can all be solved by creating a script or application to make job submission automated, repeatable and reliable.

11. How? There are a number of solutions from the Grisu Project: Grisu Framework: for creating Java applications using the Grisu libraries GriCli: a command line interface for Grisu Grisu Jython: Python-like scripting using Grisu http://github.com/grisu

12. Introducing Grisu Jython “Everything should be made as simple as possible, but not simpler.” Albert Einstein

13. Grisu Jython Grisu Jython allows scripting using Jython, a Python-like dynamically typed language, allowing complex job submission scripts using tools that many researchers are already familiar with, without the overheads of Java, or not easily done with GriCli. Python, and Jython, can be used very simply, are well established in the research community, and are known for ease of use for new users.

14. Why Jython? Jython is Python implemented in Java, and though this seems cumbersome, it provides some advantages over a pure Python implementation: Java is ubiquitous, and the JVM is available on almost all platforms Jython allows access to native Java objects and methods to a Python-like language Jython can be bundled into.jar files for distribution, hence does not need installing

15. Using Grisu Jython 1.Download the grisu-jython.jar file from: http://code.ceres.auckland.ac.nz/downloads 2.Write your script in your favourite editor or IDE 3.Execute your script in the form of: java -jar grisu-jython.jar myscript.py

16. Monitoring Jobs The Grisu Template Client still has a use in Grisu Jython job submission scripts. It provides a convenient interface for monitoring and managing submitted jobs, and as a file manager between the local workstation and remote sites. Download the Grisu Template Client from: http://code.ceres.auckland.ac.nz/downloads Run the Grisu Template Client with: java -jar grisu-template-client.jar

17. Practical Exercises “Work! Work!” - Orc Peons, WarCraft

18. Practical Exercises Next comes the practical component of the workshop, where we will explore three use cases of Grisu Jython. 1.A trivial HelloWorld job 2.Submission of multiple HelloWorld jobs using the Python List objects 3.Using the BatchJobObject to parallelise job submission and minimise upload and download time and volume

19. Do you have a Grid identity? Let’s sort that out now.

20. Get the examples The example code is available on my personal GitHub repository: http://github.com/aethylred/grisu-scripted-submission This should soon be merged with the Grisu Examples repository: http://github.com/grisu/examples This can be retrieved via eGit within Eclipse

21. Trivial HelloWorld Submit a job that echoes “Hello World” to the standard output. This example demonstrates the absolute minimum of code required to submit a job. Good for repurposing to run shell commands for testing or demonstrations. Requires specifying the job submission site

22. Processing jobs in a list Using one script to run a HelloWorld script as a job multiple times. Attaching input files to jobs Using loops to create multiple jobs in a list Iterating through a list of jobs to: – Define and submit jobs – Wait for jobs to finish – Retrieve job outputs – Kill jobs and clean up

23. BaconCount The Bacon Counter uses random text generated by the Bacon Ipsum site: http://baconipsum.com http://baconipsum.com The Bacon Counter is actually a simple dictionary matching script that counts the incidents of the words provided in the dictionary.txt file in each file in the input directory.

24. Using BatchJobObject The BatchJobObject has additional features and methods that can upload files and submit jobs in parallel. Submission location Blacklists and Whitelists De-duplicate file uploads by uploading files per batch, rather than per job Simplify staging files, submitting jobs, and killing job within the batch file BUT not perfect, job output has to be retrieved individually using the BatchJobObject as list

25. Submitting Bacon The submitbacon.py script : Uploads and submits 5 jobs at a time Uses the BatchJobObject to upload common files ( countbacon.py and dictonary.txt ) only once Create jobs per file in a directory

26. Support for Grisu Grisu JavaDocs on GitHub: http://grisu.github.com/grisu/javadoc Known issues at the Grisu Project site on GitHub http://github.com/grisu/grisu-jython/issues http://github.com/grisu/grisu/issues New issues should be submitted to NeSI support with “[Grisu]” in the subject support@nesi.org.nz

27. Finished I hope you enjoyed the workshop. Aaron Hicks hicksa@landcareresearch.co.nz 06 353 4954 PS. Yes, I wrote this presentation on the flight down