Chemistry Packages at CHPC Anita M. Orendt Center for High Performance Computing anita.orendt@utah.edu Fall 2012

Purpose of Presentation Identify the computational chemistry software and related tools currently available at CHPC Present brief overview of these packages Present how to access packages on CHPC Next presentation – Gaussian09 & GaussView 18 October 2012 http://www.chpc.utah.edu

Brief Overview CHPC Resources Computational Clusters – sanddunearch, updraft and ember Home directory – NFS mounted on all clusters /uufs/chpc.utah.edu/common/home/<uNID> generally not backed up (there are exceptions) Scratch systems /scratch/serial – all clusters – 15 TB NFS /scratch/general – updraft only – 3.5TB NFS /scratch/ibrix/chpc_gen – updraft and ember – 55TB /scratch/local on compute nodes (60GB sda, 200GB updraft, 400GB ember) Applications /uufs/arches/sys/pkg /uufs/chpc.utah.edu/sys/pkg /uufs/cluster.arches/sys/pkg where cluster = sanddunearch, updraft or ember http://www.chpc.utah.edu

256 nodes/2048 cores Infiniband and GigE 85 nodes general usage http://www.chpc.utah.edu 4/15/2017 Sanddunearch 156 nodes/624 cores Infiniband and GigE Updraft 256 nodes/2048 cores Infiniband and GigE 85 nodes general usage Owner nodes Switch Ember about 382 nodes/4584 cores Infiniband and GigE 67 nodes general usage Turret Arch Telluride 12 GPU nodes (6 CHPC) Meteorology /scratch/ibrix/chpc_gen ember, updraft IBRIX Administrative Nodes scratch systems serial – all clusters general – updraft NFS NFS Home Directories http://www.chpc.utah.edu 4

Getting Started at CHPC Account application – an online process https://www.chpc.utah.edu/apps/profile/account_request.php Username is your unid with password administrated by campus Interactive nodes two CHPC owned nodes per cluster (cluster.chpc.utah.edu) with round-robin access to divide load CHPC environment scripts – in account when created www.chpc.utah.edu/docs/manuals/getting_started/codes/chpc.tcshrc www.chpc.utah.edu/docs/manuals/getting_started/codes/chpc.bashrc Getting started guide www.chpc.utah.edu/docs/manuals/getting_started Problem reporting system http://jira.chpc.utah.edu or email to issues@chpc.utah.edu http://www.chpc.utah.edu

Interactive Node Usage Interactive nodes for prepping/testing of input files, analyzing results, compilations, debugging, data transfer, etc no running of jobs 15 min MAX cpu no jobs of any time length that negatively impact ability of other users to get work done (e.g., heavy cpu, memory usage and/or i/o) https://wiki.chpc.utah.edu/display/policy/2.1.1+Cluster+Interactive+Node+Policy http://www.chpc.utah.edu

Batch System All use of compute nodes go through a batch system using Torque (PBS) and Maui (Moab) for scheduling #PBS -S /bin/csh #PBS – A account #PBS –l walltime=24:00:00,nodes=2:ppn=8 qsub –I to get interactive use of a compute node (-X if X11 forwarding needed) Walltime limits 72 hours on sanddunearch 24 hours on updraft 72 hours on ember (long qos by request) http://www.chpc.utah.edu

Allocation Procedure Allocation process now within jira https://jira.chpc.utah.edu/secure/CreateIssue.jspa?pid=10130&issuetype=12 Allocations on a calendar quarter basis Next requests due about Dec 1 for Jan-Mar 2013 Can request for up to 4 quarters at a time Allocations for different clusters are kept separate Sanddunearch is no longer being allocated – all freecycle One allocation per research group; accessible by all members of group http://www.chpc.utah.edu

Out of Allocation Options If your group did not request any allocation – can get a quick allocation (limited to 5000 units) on one of the clusters https://jira.chpc.utah.edu/secure/CreateIssue.jspa?pid=10130&issuetype=13 Can run in freecycle mode on sanddunearch; once job starts it will run until done or walltime requested is reached Can run on smithp-guest on smithp nodes ember or freecycle on CHPC nodes on either ember or updraft (jobs are preemptable) http://www.chpc.utah.edu

Security Policies (1) No clear text passwords - use ssh and scp Do not share your account under any circumstances Don’t leave your terminal unattended while logged into your account Do not introduce classified or sensitive work onto CHPC systems Use a good password and protect it – see gate.acs.utah.edu for tips on good passwords http://www.chpc.utah.edu

Security Policies (2) Do not try to break passwords, tamper with files, look into anyone else’s directory, etc. – your privileges do not extend beyond your own directory Do not distribute or copy privileged data or software Report suspicions to CHPC (security@chpc.utah.edu) Please see http://www.chpc.utah.edu/docs/policies/security.html for more details http://www.chpc.utah.edu

Access to Interactive Nodes From Windows machine: Need ssh client PuTTY: http://www.chiark.greenend.org.uk/~sgtatham/putty/ Xshell 4: http://www.netsarang.com/products/xsh_overview.html For Xwindowing – need tool to display graphical interfaces such as Gaussview, ECCE, AutoDockTools Xming (use Mesa version) - http://www.straightrunning.com/XmingNotes http://www.chpc.utah.edu

Default login scripts CHPC maintains default login scripts that will set up necessary environment for batch commands and many of the programs to work http://www.chpc.utah.edu/docs/manuals/getting_started/code/chpc.tcshrc http://www.chpc.utah.edu/docs/manuals/getting_started/code/chpc.bashrc Located in your home directory as .tcshrc or .bashrc Can comment out setups for packages not used Default ones provided have chemistry package setups commented out – need to remove # at start of line Can customize by creating .aliases file that is sourced at end of the CHPC script – do not customize .tcshrc/.bashrc file as you will need to occasionally get a new version http://www.chpc.utah.edu

Location of Installations Have a three layer system for installations: /uufs/chpc.utah.edu/sys/pkg Accessible on all clusters and CHPC supported linux desktops /uufs/arches/sys/pkg Accessible on all clusters, but not desktops /uufs/cluster.arches/sys/pkg Single cluster access For builds of packages customized to some feature (usually for the MPI) of the given cluster 10/21/2010 http://www.chpc.utah.edu Slide 13

Software - General Information Available on CHPC web pages/wiki Can get to from www.chpc.utah.edu → Software documentation → More (for complete list) http://www.chpc.utah.edu/docs/manuals/software Available for most packages Has information on licensing restrictions, example batch scripts, where to get more information on a specific package Also has useful information on running of jobs (scaling, issues, etc) for some applications http://www.chpc.utah.edu

Types of Chemistry Packages Molecular Mechanics/Dynamics Electronic Structure Calculations Plane Wave Pseudopotential based packages – allows for periodic boundary conditions Support Packages Integrated Modeling Platforms http://www.chpc.utah.edu

Molecular Mechanics/Dynamics Amber Gromacs NAMD LAMMPS Charmm (licensed by research group) Note – Gaussian and NWChem also have MM capabilities http://www.chpc.utah.edu

Quantum (Mainly) Packages Gaussian03/09 NWChem GAMESS Molpro – serial only Dalton – serial only Orca Psi SIESTA – licensed by individual research group but can add others to license at no charge CFOUR http://www.chpc.utah.edu

Plane Wave Pseudopotential (good for periodic systems) Quantum Espresso CPMD VASP – licensed by group BigDFT (wavelet basis set) http://www.chpc.utah.edu

Integrated Molecular Platforms Schrodinger – owned by individual groups http://www.chpc.utah.edu

Support Packages Gaussview Molden VMD NBOView Babel (Openbabel) Dock AutoDock (and Autodock Vina) Cambridge Structural Database ECCE Special case – installed on carbon.chpc.utah.edu Talk to me first if you want to use http://www.chpc.utah.edu

GaussView Molecular builder and viewer for Gaussian input/output files CHPC provides campus license for linux version For Gaussian03 – standard is version 4 For Gaussian09 – standard is version 5 Chemistry Department has campus license for GV5 for windows; can buy into license Access with gv & – provided you have uncommented the Gaussian setup from the standard .tcshrc DO NOT submit jobs from within GaussView – instead create and save input file and use batch system http://www.chpc.utah.edu

NBOView Package to view NBO orbitals Works only with G03 (not G09) For information see http://www.chem.wisc.edu/~nbo5/v_manual.htm How to use at CHPC: Add to your path: /uufs/chpc.utah.edu/sys/pkg/nboview/nboview1.1 nboview http://www.chpc.utah.edu

Molden Another program for viewing molecular/electronic structures; version 4.7 Works with Gamess, Gaussian, Molpro Supports plots of electronic density, MOs, etc More information at http://www.cmbi.ru.nl/molden/molden.html How to use at CHPC: Make sure your path includes /uufs/chpc.utah.edu/sys/pkg/molden/std molden & http://www.chpc.utah.edu

VMD Visualization, mainly for MM/MD latest version (1.9.1) installed Reads a number of different file formats Information at http://www.ks.uiuc.edu/Research/vmd Can install on own desktop (windows/mac/linux versions available) /uufs/chpc.utah.edu/sys/pkg/vmd/vmd-std/bin/vmd (can uncomment setup in .tcshrc then vmd) Can use for 3D viewing on CHPC’s vis wall http://www.chpc.utah.edu

ECCE Extensible Computational Chemistry Environment Package developed at EMSL at PNNL See ecce.pnl.gov for info Set of modules to manage computational chemistry computer jobs (Gaussian, NWChem) from start (including building system) to finish (analyzing output) all from your desktop system Installed on carbon.chpc.utah.edu User accounts need setup – see me if interested as carbon is not always kept running http://www.chpc.utah.edu

OpenBabel Tool to interconvert structure files between a number of formats used in molecular modeling See openbabel.org for more information To run: source /uufs/chpc.utah.edu/sys/openbabel/etc/babel.csh (or uncomment in your .tcshrc) babel -i < input-type > < infile > -o < output-type > < outfile > babel –H to see format for usage, options, and input/output file-types http://www.chpc.utah.edu

Dock/AutoDock Programs to look at binding of a small molecule within the active site of a receptor, usually a macromolecule Dock version 6.5 installed get info at: http://dock.compbio.ucsf.edu source /uufs/chpc.utah.edu/sys/pkg/dock/etc/dock.csh dock6.mpi to start (needs arguments) Autodock version 4.2 info available at http://autodock.scripps.edu source /uufs/chpc.utah.edu/sys/pkg/autodock/etc/autodock.csh autodock4 (with proper arguments) or autogrid4 autodocktools, a GUI interface, installed – start with adt & Autodock Vina – multicore performance and enhanced accuracy – start with vina & http://www.chpc.utah.edu

Cambridge Structural Database Moved from library to CHPC summer 2006 – additions to the database are made every 3-4 months; package updated annually www.ccdc.cam.ac.uk for information Need CHPC account to use From PC need Xterm/Xwindowing software (Putty/XMing work well) to start session on any of the interactive nodes source (or uncomment line in .tchsrc) /uufs/chpc.utah.edu/sys/pkg/CSD/std/cambridge/etc/csd.csh cq & <- to start conquest (search engine) mercury & <- to start crystal structure viewer The first time you use it on a given computer you will be asked to confirm licensing need to provide site/license codes (840/7E92A7) http://www.chpc.utah.edu

Amber Molecular mechanics/dynamics package Current version is Amber 12 (Amber 11 still active) Basic information on getting started: http://www.chpc.utah.edu/docs/manuals/software/amber.html For more information see http://ambermd.org The Amber Mail Reflector Archive and mailing list is very useful! Different parallel builds available for each cluster to make use of the different interconnects Have GPU version – initial tests show impressive performance gains http://www.chpc.utah.edu

-- All have builds optimized for individual clusters -- Other MM/MD packages -- All have builds optimized for individual clusters -- Gromacs - http://www.gromacs.org/ version 4.5.4 installed with builds of both single and double precision LAMMPS - http://lammps.sandia.gov/ NAMD: http://www.ks.uiuc.edu/Research/namd/ version 2.6 on sanddunearch (2.7 current) NWChem also has MM/MD capabilities and Gaussian09 also has limited MM capabilities http://www.chpc.utah.edu

CHARMM Package is licensed per research group and for a specific version If interested – get license and we can do installation such that only your group has access See www.charmm.org for details http://www.chpc.utah.edu

Gaussian03/09 Commercial electronic structure package http://www.gaussian.com for information and User’s Guide Last installed revision of G03 is E.01 /uufs/chpc.utah.edu/sys/pkg/gaussian03/std Has been updated to include latest NBO5 Version C.01 of G09 installed /uufs/chpc.utah.edu/sys/pkg/gaussian09/AMD64 /uufs/chpc.utah.edu/sys/pkg/gaussian09/EM64T For information on accessing the CHPC installation http://www.chpc.utah.edu/docs/manuals/software/g03.html http://www.chpc.utah.edu

NWChem Package developed at PNNL to work on massively parallel systems http://www.nwchem-sw.org Goal: Computational chemistry solutions that are scalable with respect to both chemical system size and MPP hardware size Has quantum mechanics, molecular mechanics/dynamics, and quantum molecular dynamics, plane waves for periodic systems Version 6.1 (with Python) – 5.1.1 still available in chpc.utah.edu /uufs/chpc.utah.edu/sys/pkg/nwchem/nwchem-6.1/bin/LINUX64 /uufs/ember.arches/sys/pkg/nwchem/nwchem-6.1/bin/LINUX64 (not tested) /uufs/updraft.arches/sys/pkg/nwchem/nwchem-6.1/bin/LINUX64 (not tested) To run: source appropriate etc/nwchem.csh must have $HOME/pdir directory if running parallel More information and example batch script at http://www.chpc.utah.edu/docs/manuals/software/nwchem.html http://www.chpc.utah.edu

GAMESS General Atomic and Molecular Electronic Structure System Another option for most ab initio quantum calculations On 22 February 2006 version – can be updated if needed /uufs/chpc.utah.edu/sys/pkg/GAMESS/gamess http://www.msg.ameslab.gov/GAMESS for information on usage and capabilities Can run both parallel or serial For information on accessing the CHPC installation see http://www.chpc.utah.edu/docs/manuals/software/gamess.html http://www.chpc.utah.edu

Other Quantum Codes -- Mainly codes that are specialized in terms of functionality -- Dalton – only serial build Focus on property calculations at HF, DFT, MCSCF and CC levels of theory version 2.0 installed; Dalton2011 recently released http://dirac.chem.sdu.dk/daltonprogram.org/ Molpro – only serial build Emphasis on highly accurate calculation methods Version 2006.1 installed; new 2012.1 available http://www.molpro.net/ Psi - http://www.psicode.org/ Orca - http://www.thch.uni-bonn.de/tc/orca/ CFOUR - http://www.cfour.de/ http://www.chpc.utah.edu

Plane Wave Codes Plane wave codes for the study of systems under periodic boundary conditions (PBC) NWChem has some functionality VASP – http://cms.mpi.univie.ac.at/vasp/ Ab initio QM/MD Licensed per research group (have 1 group with an old version) Quantum Espresso - http://www.quantum-espresso.org/ Understands crystal space groups Has GIPAW module to do NMR calculations J-ICE and XCrysDen to view CPMD - http://www.cpmd.org/ BigDFT - http://inac.cea.fr/L_Sim/BigDFT/ http://www.chpc.utah.edu

Schrodinger Suite Commercial package geared for chemical modeling and simulation in the pharmaceutical field, specifically drug discovery http://www.schrodinger.com/ Interface – Maestro (free for academia) Calculation code includes Jaguar (Quantum), Macromodel (MM), Qsite (QM/MM) Interfaces with Desmond for MD (installed) Tools for structure based drug design Docking, ligand design, binding affinities, screening libraries Year to Year licensing Token based, so limited in number of concurrent uses Purchased by two groups in Medicinal Chemistry If interested in using – come talk to me http://www.chpc.utah.edu

Finally….. Let us know if there is some other package that does something that our current packages do not; we can look into the possibility of getting it. Factors: cost, hardware/OS requirements, licensing issues, usage needs Any questions – contact me anita.orendt@utah.edu Phone: 801-231-2762 Office: 422 INSCC http://www.chpc.utah.edu