1. Chemistry Packages at CHPC
Anita M. Orendt
Center for High Performance Computing
Fall 2012

2. Purpose of Presentation
Identify the computational chemistry software and related tools currently available at CHPC
Present brief overview of these packages
Present how to access packages on CHPC
Next presentation – Gaussian09 & GaussView
18 October 2012

3. Brief Overview CHPC Resources
Computational Clusters – sanddunearch, updraft and ember
Home directory – NFS mounted on all clusters
/uufs/chpc.utah.edu/common/home/<uNID>
generally not backed up (there are exceptions)
Scratch systems
/scratch/serial – all clusters – 15 TB NFS
/scratch/general – updraft only – 3.5TB NFS
/scratch/ibrix/chpc_gen – updraft and ember – 55TB
/scratch/local on compute nodes (60GB sda, 200GB updraft, 400GB ember)
Applications
/uufs/arches/sys/pkg
/uufs/chpc.utah.edu/sys/pkg
/uufs/cluster.arches/sys/pkg where cluster = sanddunearch, updraft or ember

4. 256 nodes/2048 cores Infiniband and GigE 85 nodes general usage
4/15/2017
Sanddunearch
156 nodes/624 cores
Infiniband and GigE
Updraft
256 nodes/2048 cores Infiniband and GigE
85 nodes general usage
Owner nodes
Switch
Ember
about 382 nodes/4584 cores
Infiniband and GigE
67 nodes general usage
Turret Arch
Telluride
12 GPU nodes (6 CHPC)
Meteorology
/scratch/ibrix/chpc_gen
ember, updraft
IBRIX
Administrative
Nodes
scratch systems
serial – all clusters
general – updraft
NFS
NFS
Home Directories 4

5. Getting Started at CHPC
Account application – an online process
Username is your unid with password administrated by campus
Interactive nodes
two CHPC owned nodes per cluster (cluster.chpc.utah.edu) with round-robin access to divide load
CHPC environment scripts – in account when created
Getting started guide
Problem reporting system
or to

6. Interactive Node Usage
Interactive nodes for prepping/testing of input files, analyzing results, compilations, debugging, data transfer, etc
no running of jobs
15 min MAX cpu
no jobs of any time length that negatively impact ability of other users to get work done (e.g., heavy cpu, memory usage and/or i/o)

7. Batch System
All use of compute nodes go through a batch system using Torque (PBS) and Maui (Moab) for scheduling
#PBS -S /bin/csh
#PBS – A account
#PBS –l walltime=24:00:00,nodes=2:ppn=8
qsub –I to get interactive use of a compute node (-X if X11 forwarding needed)
Walltime limits
72 hours on sanddunearch
24 hours on updraft
72 hours on ember (long qos by request)

8. Allocation Procedure Allocation process now within jira
Allocations on a calendar quarter basis
Next requests due about Dec 1 for Jan-Mar 2013
Can request for up to 4 quarters at a time
Allocations for different clusters are kept separate
Sanddunearch is no longer being allocated – all freecycle
One allocation per research group; accessible by all members of group

9. Out of Allocation Options
If your group did not request any allocation – can get a quick allocation (limited to 5000 units) on one of the clusters
Can run in freecycle mode on sanddunearch; once job starts it will run until done or walltime requested is reached
Can run on smithp-guest on smithp nodes ember or freecycle on CHPC nodes on either ember or updraft (jobs are preemptable)

10. Security Policies (1) No clear text passwords - use ssh and scp
Do not share your account under any circumstances
Don’t leave your terminal unattended while logged into your account
Do not introduce classified or sensitive work onto CHPC systems
Use a good password and protect it – see gate.acs.utah.edu for tips on good passwords

11. Security Policies (2)
Do not try to break passwords, tamper with files, look into anyone else’s directory, etc. – your privileges do not extend beyond your own directory
Do not distribute or copy privileged data or software
Report suspicions to CHPC
Please see for more details

12. Access to Interactive Nodes
From Windows machine:
Need ssh client
PuTTY:
Xshell 4:
For Xwindowing – need tool to display graphical interfaces such as Gaussview, ECCE, AutoDockTools
Xming (use Mesa version) -

13. Default login scripts
CHPC maintains default login scripts that will set up necessary environment for batch commands and many of the programs to work
Located in your home directory as .tcshrc or .bashrc
Can comment out setups for packages not used
Default ones provided have chemistry package setups commented out – need to remove # at start of line
Can customize by creating .aliases file that is sourced at end of the CHPC script – do not customize .tcshrc/.bashrc file as you will need to occasionally get a new version

14. Location of Installations
Have a three layer system for installations:
/uufs/chpc.utah.edu/sys/pkg
Accessible on all clusters and CHPC supported linux desktops
/uufs/arches/sys/pkg
Accessible on all clusters, but not desktops
/uufs/cluster.arches/sys/pkg
Single cluster access
For builds of packages customized to some feature (usually for the MPI) of the given cluster
10/21/ Slide 13

15. Software - General Information
Available on CHPC web pages/wiki
Can get to from → Software documentation → More (for complete list)
Available for most packages
Has information on licensing restrictions, example batch scripts, where to get more information on a specific package
Also has useful information on running of jobs (scaling, issues, etc) for some applications

16. Types of Chemistry Packages
Molecular Mechanics/Dynamics
Electronic Structure Calculations
Plane Wave Pseudopotential based packages – allows for periodic boundary conditions
Support Packages
Integrated Modeling Platforms

17. Molecular Mechanics/Dynamics
Amber
Gromacs
NAMD
LAMMPS
Charmm (licensed by research group)
Note – Gaussian and NWChem also have MM capabilities

18. Quantum (Mainly) Packages
Gaussian03/09
NWChem
GAMESS
Molpro – serial only
Dalton – serial only
Orca
Psi
SIESTA – licensed by individual research group but can add others to license at no charge
CFOUR

19. Plane Wave Pseudopotential (good for periodic systems)
Quantum Espresso
CPMD
VASP – licensed by group
BigDFT (wavelet basis set)

20. Integrated Molecular Platforms
Schrodinger – owned by individual groups

21. Support Packages Gaussview Molden VMD NBOView Babel (Openbabel) Dock
AutoDock (and Autodock Vina)
Cambridge Structural Database
ECCE
Special case – installed on carbon.chpc.utah.edu
Talk to me first if you want to use

22. GaussView Molecular builder and viewer for Gaussian input/output files
CHPC provides campus license for linux version
For Gaussian03 – standard is version 4
For Gaussian09 – standard is version 5
Chemistry Department has campus license for GV5 for windows; can buy into license
Access with gv & – provided you have uncommented the Gaussian setup from the standard .tcshrc
DO NOT submit jobs from within GaussView – instead create and save input file and use batch system

23. NBOView Package to view NBO orbitals Works only with G03 (not G09)
For information see
How to use at CHPC:
Add to your path: /uufs/chpc.utah.edu/sys/pkg/nboview/nboview1.1
nboview

24. Molden
Another program for viewing molecular/electronic structures; version 4.7
Works with Gamess, Gaussian, Molpro
Supports plots of electronic density, MOs, etc
More information at
How to use at CHPC:
Make sure your path includes /uufs/chpc.utah.edu/sys/pkg/molden/std
molden &

25. VMD Visualization, mainly for MM/MD latest version (1.9.1) installed
Reads a number of different file formats
Information at
Can install on own desktop (windows/mac/linux versions available)
/uufs/chpc.utah.edu/sys/pkg/vmd/vmd-std/bin/vmd (can uncomment setup in .tcshrc then vmd)
Can use for 3D viewing on CHPC’s vis wall

26. ECCE Extensible Computational Chemistry Environment
Package developed at EMSL at PNNL
See ecce.pnl.gov for info
Set of modules to manage computational chemistry computer jobs (Gaussian, NWChem) from start (including building system) to finish (analyzing output) all from your desktop system
Installed on carbon.chpc.utah.edu
User accounts need setup – see me if interested as carbon is not always kept running

27. OpenBabel
Tool to interconvert structure files between a number of formats used in molecular modeling
See openbabel.org for more information
To run:
source /uufs/chpc.utah.edu/sys/openbabel/etc/babel.csh (or uncomment in your .tcshrc)
babel -i < input-type > < infile > -o < output-type > < outfile >
babel –H to see format for usage, options, and input/output file-types

28. Dock/AutoDock
Programs to look at binding of a small molecule within the active site of a receptor, usually a macromolecule
Dock
version 6.5 installed
get info at:
source /uufs/chpc.utah.edu/sys/pkg/dock/etc/dock.csh
dock6.mpi to start (needs arguments)
Autodock
version 4.2
info available at
source /uufs/chpc.utah.edu/sys/pkg/autodock/etc/autodock.csh
autodock4 (with proper arguments) or autogrid4
autodocktools, a GUI interface, installed – start with adt &
Autodock Vina – multicore performance and enhanced accuracy – start with vina &

29. Cambridge Structural Database
Moved from library to CHPC summer 2006 – additions to the database are made every 3-4 months; package updated annually
for information
Need CHPC account to use
From PC need Xterm/Xwindowing software (Putty/XMing work well) to start session on any of the interactive nodes
source (or uncomment line in .tchsrc) /uufs/chpc.utah.edu/sys/pkg/CSD/std/cambridge/etc/csd.csh
cq & <- to start conquest (search engine)
mercury & <- to start crystal structure viewer
The first time you use it on a given computer you will be asked to confirm licensing
need to provide site/license codes (840/7E92A7)

30. Amber
Molecular mechanics/dynamics package Current version is Amber 12 (Amber 11 still active)
Basic information on getting started:
For more information see
The Amber Mail Reflector Archive and mailing list is very useful!
Different parallel builds available for each cluster to make use of the different interconnects
Have GPU version – initial tests show impressive performance gains

31. -- All have builds optimized for individual clusters --
Other MM/MD packages
-- All have builds optimized for individual clusters --
Gromacs -
version installed with builds of both single and double precision
LAMMPS -
NAMD:
version 2.6 on sanddunearch (2.7 current)
NWChem also has MM/MD capabilities and Gaussian09 also has limited MM capabilities

32. CHARMM
Package is licensed per research group and for a specific version
If interested – get license and we can do installation such that only your group has access
See for details

33. Gaussian03/09 Commercial electronic structure package
for information and User’s Guide
Last installed revision of G03 is E.01
/uufs/chpc.utah.edu/sys/pkg/gaussian03/std
Has been updated to include latest NBO5
Version C.01 of G09 installed
/uufs/chpc.utah.edu/sys/pkg/gaussian09/AMD64
/uufs/chpc.utah.edu/sys/pkg/gaussian09/EM64T
For information on accessing the CHPC installation

34. NWChem Package developed at PNNL to work on massively parallel systems
Goal: Computational chemistry solutions that are scalable with respect to both chemical system size and MPP hardware size
Has quantum mechanics, molecular mechanics/dynamics, and quantum molecular dynamics, plane waves for periodic systems
Version 6.1 (with Python) – still available in chpc.utah.edu
/uufs/chpc.utah.edu/sys/pkg/nwchem/nwchem-6.1/bin/LINUX64
/uufs/ember.arches/sys/pkg/nwchem/nwchem-6.1/bin/LINUX64 (not tested)
/uufs/updraft.arches/sys/pkg/nwchem/nwchem-6.1/bin/LINUX64 (not tested)
To run:
source appropriate etc/nwchem.csh
must have $HOME/pdir directory if running parallel
More information and example batch script at

35. GAMESS General Atomic and Molecular Electronic Structure System
Another option for most ab initio quantum calculations
On 22 February 2006 version – can be updated if needed
/uufs/chpc.utah.edu/sys/pkg/GAMESS/gamess
for information on usage and capabilities
Can run both parallel or serial
For information on accessing the CHPC installation see

36. Other Quantum Codes
-- Mainly codes that are specialized in terms of functionality --
Dalton – only serial build
Focus on property calculations at HF, DFT, MCSCF and CC levels of theory
version 2.0 installed; Dalton2011 recently released
Molpro – only serial build
Emphasis on highly accurate calculation methods
Version installed; new available
Psi -
Orca -
CFOUR -

37. Plane Wave Codes
Plane wave codes for the study of systems under periodic boundary conditions (PBC)
NWChem has some functionality
VASP –
Ab initio QM/MD
Licensed per research group (have 1 group with an old version)
Quantum Espresso -
Understands crystal space groups
Has GIPAW module to do NMR calculations
J-ICE and XCrysDen to view
CPMD -
BigDFT -

38. Schrodinger Suite
Commercial package geared for chemical modeling and simulation in the pharmaceutical field, specifically drug discovery
Interface – Maestro (free for academia)
Calculation code includes Jaguar (Quantum), Macromodel (MM), Qsite (QM/MM)
Interfaces with Desmond for MD (installed)
Tools for structure based drug design
Docking, ligand design, binding affinities, screening libraries
Year to Year licensing
Token based, so limited in number of concurrent uses
Purchased by two groups in Medicinal Chemistry
If interested in using – come talk to me

39. Finally…..
Let us know if there is some other package that does something that our current packages do not; we can look into the possibility of getting it.
Factors: cost, hardware/OS requirements, licensing issues, usage needs
Any questions – contact me
Phone:
Office: 422 INSCC