1. Computational chemistry packages (efficient usage issues?) Jemmy Hu SHARCNET HPC Consultant Summer School June 3, 2016 /work/jemmyhu/ss2016/chemistry/

2. Chemistry Software managed by SHARCNET Electronic Structure TheoryMolecular Dynamics Gaussian (09, 03) -commercialAMBER -commercial ADF/Band -commercialGromacs NWChemCPMD Quantum ExpressoNAMD ORCA Gamess/Firefly Abinit MolproVASP TurbomoleCHARMM User/group owned software (commercial)

3. Nature of the software SoftwareParallelPurpose GaussianThreaded (OpenMP) http://www.gaussian.com/g_prod/g09.htm ADFMPI https://www.scm.com/product/adf/ NWChemMPI http://www.nwchem- sw.org/index.php/Main_Page Quantum ExpressoMPI http://www.quantum-espresso.org/ VASPMPI https://www.vasp.at/index.php/about- vasp/59-about-vasp ORCAMPI https://orcaforum.cec.mpg.de/

4. Open Software used by 2 or more groups – add to SHARCNET software stack with software usage instruction/example webpage online https://www.sharcnet.ca/my/software Software used by one group – user installs in his/her own space with help from SHARCNET if necessary SHARCNET owned commercial software – free Self owned software – user’s response for license issues Software are installed as module package, setup in your bash configuration file, i.e.,.bash_profile file under /home/userid/ module load gaussian/g09_D.01 General policy for software

5. module load packages Login to a cluster, e.g., ssh userID@saw.sharcnet.ca Open the shell configuration file, e.g., vi.bash_profile Add the module load lines at the end of the file module load gaussian/g09_D.01 module load adf/2014.05 module load nwchem/6.5 module load espresso/5.1.1 module load orca/3.0 Save the file Exit Re-login Goto your /work directory: cd /work/userID Copy the files cp –rf /work/jemmyhu/ss2016/chemistry.

7. System Info Parallel Computer Memory Architectures Hybrid Distributed-Shared Memory (Cluster solution)

8. Size of Threaded Jobs on specific system https://www.sharcnet.ca/my/systems SystemNodesCPU/Node (threads) Memory GB (Max for full cpus usage) orcaOpteron16 or 2430 sawXeon815 brownXeon816 Job uses less than the max threads and memory on a node will be preferable

9. Gaussian 09 issues Latest version: g09-D.01 Input sample File location Memory Disk Restart Utilities https://www.sharcnet.ca/my/software/show/21

11. #p b3lyp/6-311++g(2df,p) opt freq > 0 1 c o 1 co co=1.1281 (leave an blank line at the end) G09 input file: simplest case File name: CO-b3lyp-1.com

12. Real jobs should be submitted to the threaded queue using sqsub command First check the files under your /work and /scratch directory open two windows, one for /work, one to view files in /scratch For testing purpose today, we run jobs from command line on the dev. node on saw (saw-dev1, …, saw-dev6) ssh saw-dev4 run g09 CO-b3lyp-1.com >& CO-b3lyp-1.log & Output file CO-b3lyp-1.log Runtime files (5 of them,.rwf is big) /scratch/userID Job uses defaults [1 cpu, minimum memory (256MB?)] Only.log output file will be generated in the run directory Runtime files will be removed automatically once finished successfully, otherwise they will stay. Run job, file structure

13. Specify checkpoint file: name.chk %chk=CO-b3lyp-1chk #p b3lyp/6-311++g(2df,p) opt freq > 0 1 c o 1 co co=1.1281 Input file: CO-b3lyp-1chk.com Run on dev: g09 CO-b3lyp-1chk.com >& CO-b3lyp-1chk.log & Files: CO-b3lyp-1chk.log, CO-b3lyp-1chk.chk

14. G09 input file: CO-b3lyp-4.com, use 4 cpus %nosave %mem=500MB %chk=CO-b3lyp-4 %nproc=4 #p b3lyp/6-311++g(2df,p) opt freq > 0 1 c o 1 co co=1.1281

15. Run Job in the queue %mem=500MB (--mpp=2x%mem, memory per job, for all cpus) %nproc=4 (-n 4) Submit command is sqsub -q threaded -n 4 -r 1.0d –mpp=2g -o CO-6311-4.out g09 CO-6311-4.com What’s happening afterwards? -sqjobs to see job status, jobID -If runs, where are the files?.log and.chk files are generated in the run directory runtime files,.rwf, etc will be in /scratch/yourID/jobID/ -Failed? remove the runtime files, rm –rf /scratch/yourID/jobID. correct input file, re-run

16. %rwf=CO-b3lyp-4rwf %nosave %mem=500MB %chk=CO-b3lyp-4rwf %nproc=4 #p b3lyp/6-311++g(2df,p) opt freq > 0 1 c o 1 co co=1.1281 The named.rwf file will be generated in the current directory Input file with named.rwf: CO-b3lyp-4rwf.com

17. Checkpointing 1..chk can be used to restart an Optimization job, or used for property (freq, etc) calculation afterwards. 2. In G09,.rwf can be used to restart a frequency calculation.

18. Restart (see examples) Restart opt job from.chk Restart freq from.rwf %rwf=/scratch/jemmyhu/Cu_test.rwf %NoSave %chk=Cu_test.chk %mem=500mb %nproc=4 #p restart (one blank line) %NoSave %chk=Cu_test.chk %mem=500mb %nproc=4 #B3LYP/DGDZVP Opt Geom=AllCheck Guess=Read (one blank line)

19. Gaussian Utilities Utilities formchk cubegen freqchk …… use the same version direct run from command line NBO

20. Review Input file, memory (%mem) /cpu (%nproc) sqsub (-n, --mpp), sqjobs, sqkill Use job info from scheduler File location Failed jobs File cleanup

21. ADF ADF is frequently used for studying transition metal complexes and molecules with heavy atoms. ADF offers unique capabilities to predict molecular properties of organic electronics materials.transition metal complexesheavy atomsorganic electronics ADF, BAND, DFTB https://www.scm.com/

23. Use ADF on SHARCNET Available on major clusters, viz-stations Versions: 2013.01, 2014.05 Setup, add module load line to.bash_profile file, i.e., module load adf/2014.05 Examples /opt/sharcnet/adf/2014.05/examples

24. [jemmyhu@saw-login1:~/tests/test_ADF/saw/test_adf] module show adf/2014.05 ------------------------------------------------------------------- /opt/sharcnet/modules/adf/2014.05: module-whatis Provides ADF 2014.05 binary for x86_64 Centos 6 systems conflict adf setenv ADFROOT /opt/sharcnet/adf setenv ADFHOME /opt/sharcnet/adf/2014.05 setenv ADFBIN /opt/sharcnet/adf/2014.05/bin setenv ADFRESOURCES /opt/sharcnet/adf/2014.05/atomicdata setenv SCMLICENSE /opt/sharcnet/adf/2014.05/license setenv SCM_TMPDIR /scratch/jemmyhu setenv SCM_USETMPDIR yes prepend-path PATH /opt/sharcnet/adf/2014.05/bin ------------------------------------------------------------------- Default settings

25. Two input options name.inp (one step a time) use a runscript ‘ADF’ or ‘BAND’, to bypass the runtime files to /scratch/username/jobid/ https://www.sharcnet.ca/help/index.php/ADF/BAND sqsub -q mpi --nompirun -n 8 --mpp=2g -r 1.0d -o GO_H2O-adf.log ADF GO_H2O-adf.inp sqsub -q mpi --nompirun -n 8 --ppn=4 --mpp=2g -r 1.0d -o GO_H2O-adf.log ADF GO_H2O-adf.inp sqsub -q mpi --nompirun -n 8 --mpp=2g -r 1.0d -o name_band.log BAND name_band.inp

26. /home/jemmyhu/work/ss2016/chemistry/examples/adf GO_H2O-adf.inp

27. sqsub -q mpi --nompirun -n 8 --ppn=4 --mpp=2g -r 1.0d -o NMR_B3LYP.log./NMR_B3LYP.run Input for multi-steps: name.run runtime files are under /scratch/username/

32. Restart TAPE13 It serves as a checkpoint file. TAPE13 can be used to restart the calculation at a point not too far before the fatal condition occurred TAPE13 is upgraded during the calculation but discarded upon normal termination. To keep TAPE13, -use the SAVE key TAPE21 Upon normal (i.e. program-controlled) termination of a calculation, the TAPE21 result file can be used for restart purposes When a crash occurs, however, chances are that TAPE21 has not correctly been closed and that its data structure is inconsistent

33. ADF GUI GUI software to run on viz-workstations vdi-centos6.user.sharcnet.ca viz-uwo10.sharcnet.ca viz-uwo11.sharcnet.ca Turn on X11 locally, then login to he workstation adfview

35. https://www.sharcnet.ca/help/index.php/NWCHEM

37. [jemmyhu@saw333:~/work/ss2016/chemistry/examples/nwchem] pwd /home/jemmyhu/work/ss2016/chemistry/examples/nwchem [jemmyhu@saw333:~/work/ss2016/chemistry/examples/nwchem] ls h2o_hu.nw sqsub -q mpi -n 4 -r 3.0d --mpp=4g -o h2o_hu.log nwchem h2o_hu.nw RUN nwchem on SHARCNET

39. https://www.sharcnet.ca/help/index.php/ESPRESSO

41. https://www.sharcnet.ca/help/index.php/ORCA

43. https://www.sharcnet.ca/help/index.php/VASP Research group owns the license Installation guideline

44. What else? Question? Discussion?