Www.mimos.my© 2010 MIMOS Berhad. All Rights Reserved. Nazarudin Wijee Mohd Sidek Salleh Grid Computing Lab MIMOS Berhad Blender Job Submission in P-GRADE.

Slides:



Advertisements
Similar presentations
MIMOS Berhad. All Rights Reserved. Nazarudin Wijee Mohd Sidek Salleh Grid Computing Lab MIMOS Berhad P-GRADE Performance.
Advertisements

Cluster Computing at IQSS Alex Storer, Research Technology Consultant.
MIMOS Berhad. All Rights Reserved. Nazarudin Wijee Mohd Sidek Salleh Grid Computing Lab MIMOS Berhad Key Size Analysis of Brute Force.
Job Submission Using PBSPro and Globus Job Commands.
MIMOS Berhad. All Rights Reserved. Nazarudin Wijee Mohd Sidek Salleh Grid Computing Lab MIMOS Berhad 7 Stages Heating System Amber Job.
Southgreen HPC system Concepts Cluster : compute farm i.e. a collection of compute servers that can be shared and accessed through a single “portal”
IT MANAGEMENT OF FME, 21 ST JULY  THE HPC FACILITY  USING PUTTY AND WINSCP TO ACCESS THE SERVER  SENDING FILES TO THE SERVER  RUNNING JOBS 
ISG We build general capability Job Submission on the Olympus Cluster J. DePasse; S. Brown, PhD; T. Maiden Pittsburgh Supercomputing Center Public Health.
Submit Host Setup (user) Tc.data file Pool.config file Properties file Vdl-gen file Input file Exitcode checking script.
Introduction to Linux and Shell Scripting Jacob Chan.
Understanding the Basics of Computational Informatics Summer School, Hungary, Szeged Methos L. Müller.
ISG We build general capability Purpose After this tutorial, you should: Be comfortable submitting work to the batch queuing system of olympus and be familiar.
CS 141 Labs are mandatory. Attendance will be taken in each lab. Make account on moodle. Projects will be submitted via moodle.
Shell Programming, or Scripting Shirley Moore CPS 5401 Fall August 29,
Introduction to UNIX/Linux Exercises Dan Stanzione.
Sponsored by the National Science Foundation Configuration Management For Experimenters: Ansible Hands-On Sarah Edwards, GPO.
1 Operating Systems Lecture 3 Shell Scripts. 2 Brief review of unix1.txt n Glob Construct (metacharacters) and other special characters F ?, *, [] F Ex.
ISG We build general capability Introduction to Olympus Shawn T. Brown, PhD ISG MISSION 2.0 Lead Director of Public Health Applications Pittsburgh Supercomputing.
1. Introduction  The JavaScript Grid Portal is trying to find a way to access Grid through Web browser, while using Web 2.0 technologies  The portal.
1 Shell Programming – Extra Slides. 2 Counting the number of lines in a file #!/bin/sh #countLines1 filename=$1#Should check if arguments are given count=0.
Linux Shell Programming Tutorial 3 ENGR 3950U / CSCI 3020U Operating Systems Instructor: Dr. Kamran Sartipi.
VIPBG LINUX CLUSTER By Helen Wang March 29th, 2013.
Bigben Pittsburgh Supercomputing Center J. Ray Scott
Airavata Usecases from SEAGrid Applications and Workflows Sudhakar Pamidighantam 4 Oct 2015.
Additional UNIX Commands. 222 Lecture Overview  Multiple commands and job control  More useful UNIX utilities.
Introduction to Linux OS (IV) AUBG ICoSCIS Team Prof. Volin Karagiozov March, 09 – 10, 2013 SWU, Blagoevgrad.
Rochester Institute of Technology Job Submission Andrew Pangborn & Myles Maxfield 10/19/2015Service Oriented Cyberinfrastructure Lab,
Shell Scripting AFNOG IX Rabat, Morocco May 2008.
July 28' 2011INDIA-CMS_meeting_BARC1 Tier-3 TIFR Makrand Siddhabhatti DHEP, TIFR Mumbai July 291INDIA-CMS_meeting_BARC.
Java Commodity Grid (Java CogKit) Java CogKits allow developers to use commodity technologies such as Java or Python in programming the Grid based on Globus.
CS252: Systems Programming Ninghui Li Slides by Prof. Gustavo Rodriguez-Rivera Topic 7: Unix Tools and Shell Scripts.
MIMOS Berhad. All Rights Reserved. Nazarudin Wijee Mohd Sidek Salleh Grid Computing Lab MIMOS Berhad P-GRADE Portal Heuristic Evaluation.
Linux Commands C151 Multi-User Operating Systems.
Cluster Computing Applications for Bioinformatics Thurs., Sept. 20, 2007 process management shell scripting Sun Grid Engine running parallel programs.
1 Unix/Linux commands and shell programming-Part 2 (Dr. Mohamed El Bachir Menai)
FESR Consorzio COMETA - Progetto PI2S2 WMS - Scripting techniques Fabio Scibilia INFN – Catania, Italy Tutorial per utenti e sviluppo.
ISG We build general capability Introduction to Olympus Shawn T. Brown, PhD ISG MISSION 2.0 Lead Director of Public Health Applications Pittsburgh Supercomputing.
Introduction to HPC Workshop October Introduction Rob Lane & The HPC Support Team Research Computing Services CUIT.
06/08/10 P-GRADE Portal and MIMOS P-GRADE portal developments in the framework of the MIMOS-SZTAKI joint project Mohd Sidek Salleh MIMOS Berhad Zoltán.
Portable Batch System – Definition and 3 Primary Roles Definition: PBS is a distributed workload management system. It handles the management and monitoring.
Compunet Corporation Introduction to Unix (CA263) Round and Round By Tariq Ibn Aziz Dammam Community College.
1 Support for Parameter Study applications in the P-GRADE Portal Cevat Şener Dept. Of Computer Engineering, METU.
Advanced topics Cluster Training Center for Simulation and Modeling September 4, 2015.
2 nd Intro to Shell Scripting. Dates for Last Week of Class Homework 7 – Due Tuesday 5/1 by midnight Labs 7 & 8 or late ones – 8 is extra credit – Due.
Experiment No. 13 Presented by, Mr. Satish Pise. Write a shell script which checks disk space and store the value to the variable and display it. #!/bin/sh.
Wouter Verkerke, NIKHEF 1 Using ‘stoomboot’ for NIKHEF-ATLAS batch computing What is ‘stoomboot’ – Hardware –16 machines, each 2x quad-core Pentium = 128.
Part I: n Q1: n $ file * | grep “ascii text” | wc -l n Q2: n $ a.out | tee logfile n Q3: $touch file.`date +’%H%M%S’` n.
Scaling bio-analyses from computational clusters to grids George Byelas University Medical Centre Groningen, the Netherlands IWSG-2013, Zürich, Switzerland,
+ Introduction to Unix Joey Azofeifa Dowell Lab Short Read Class Day 2 (Slides inspired by David Knox)
Setting up visualization. Make output folder for visualization files Log into vieques $ ssh
Centre for Parallel Computing Tamas Kiss Centre for Parallel Computing A Distributed Rendering Service Tamas Kiss Centre for Parallel Computing Research.
Advanced Computing Facility Introduction
Hackinars in Bioinformatics
GRID COMPUTING.
PARADOX Cluster job management
Unix Scripts and PBS on BioU
WMS - Tecniche di scripting
Prepared by: Eng. Maryam Adel Abdel-Hady
Welcome to Indiana University Clusters
How to use the HPCC to do stuff
IW2D migration to HTCondor
Shell Script Assignment 1.
Practice #0: Introduction
MIK 2.1 DBNS - introduction to WS-PGRADE, 2013
Basic UNIX OLC Training.
College of Engineering
Compiling and Job Submission
UNIX Reference Sheets CSE 2031 Fall 2010.
Working in The IITJ HPC System
Presentation transcript:

MIMOS Berhad. All Rights Reserved. Nazarudin Wijee Mohd Sidek Salleh Grid Computing Lab MIMOS Berhad Blender Job Submission in P-GRADE Portal

Blender and Python Information Blender Version: 2.49b Python Version: MIMOS Berhad. All Rights Reserved.

Blender Workflow MIMOS Berhad. All Rights Reserved. #PBS -N Blender #PBS -J 0-10:1 #PBS -r y #PBS -j oe #PBS -m n #PBS -o /dev/null export PYTHONHOME=/opt/Python export LD_LIBRARY_PATH_OLD=$LD_LIBRARY_PATH export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PYTHONHOME/lib PROJECT_DIR=job_execution_directory EXE=/opt/blender-2.49b-linux-glibc236-py25-x86_64/blender INPUT_FILE=$PROJECT_DIR/input_file.blend OUTPUT_FILE=$PROJECT_DIR/output-# LOG_FILE=$PROJECT_DIR/$PBS_ARRAY_INDEX.log cd $PBS_O_WORKDIR $EXE -b $INPUT_FILE -x 1 -o //output-### -F JPEG -s $PBS_ARRAY_INDEX.0 -e $PBS_ARRAY_INDEX.0 -a 2>&1 | tee $LOG_FILE export LD_LIBRARY_PATH=$LD_LIBRARY_PATH_OLD What will Submit job do? 1.It will submit (qsub) pbs-script to the local resource manager (PBS Pro) 2.Save the submission PBS Job ID in a file and pass it to the Monitor job.

For MAYA Rendering MIMOS Berhad. All Rights Reserved. #!/bin/sh #PBS -l arch=linux #PBS -N e-Render #PBS -J 0-10:1 #PBS -r y #PBS -j oe #PBS -m n #PBS -o /dev/null TMPDIR="/usr/tmp" if [ ! "$PBS_ARRAY_INDEX"]; then PBS_ARRAY_INDEX=1 fi; umask 000 if test -d "job_execution_directory/renderData" then echo -e "Directroy already exists" else mkdir job_execution_directory/renderData fi mkdir job_execution_directory/renderData/images mkdir job_execution_directory/renderData/logs Render -n 1 -r sw -of jpg -rd job_execution_directory/renderData/images/ -s $PBS_ARRAY_INDEX.0 -e $PBS_ARRAY_INDEX.0 job_execution_directory/input_file.mb 2>&1 | tee job_execution_directory/renderData/logs/$PBS_ARRAY_INDEX.log

Blender Workflow MIMOS Berhad. All Rights Reserved.. executor.info echo "PBS_JOBID = $PBS_JOBID" echo "MONITOR_JOB_DIR = `pwd`" echo # begin monitoring FINISH_STATUS="0" until [[ $FINISH_STATUS -eq "1" ]] do WC=`tracejob $PBS_JOBID | grep 'dequeuing from' | wc -l` if [[ $WC -eq 1 ]]; then FINISH_STATUS="1" else FINISH_STATUS="0" fi sleep 120 done What will Monitor job do? 1.It will receive PBS Job ID from Submit job 2.During runtime, it will goes to cluster head node and do PBS Pro tracejob to check it the given PBS Job ID already finished.

Blender Workflow MIMOS Berhad. All Rights Reserved.. executor.info zip -r $COLLECT_JOB_DIR/project-directory.zip $PROJECT_DIR echo "Done" > collect.status What will Collect and CleanUp job do? 1.Collect will compress all the output files and in one zip file 2.CleanUp will delete the Job Execution Directory which contain the output files and also delete all the log files.. executor.info echo "Removing project directory..." rm -rf $PROJECT_DIR echo "Removing csm output pot and log files..." $EXEC_DIR/run/cleanup.sh

THANK YOU MIMOS Berhad. All Rights Reserved.