1. http://aimpro.ncl.ac.ukMMG Skills Lecture Series Introductory concepts: Computer etiquette Jon Goss

2. MMG Skills Lecture Series 2 Outline Get organised Consistency vs efficiency Compute machine vs file server Hierarchical calculation strategy Restarting calculations Throughput vs turn-around time Interacting with a batch queue

3. MMG Skills Lecture Series 3 Get organised vast You are in an area of study that has the potential to produce vast amounts of data Why do we need to store it? crucial It is crucial that you adopt a comprehensive, structured filing system from the very start. index files Alpha-numerical file naming with index files – could be chronological. Potentially many files per directory. Hierarchical Hierarchical set of directories based on application Try to keep few files per directory. revise You’ll probably have to revise your filing system over time. Acid test: When asked to locate a file, can you do it within a few seconds? Minutes? Hours? Days?

4. MMG Skills Lecture Series 4 Consistency vs efficiency begin consistent When you begin researching an application, you may adopt a parameter set from some previous calculations to be consistent with them foundation The foundation work may already be there Bulk material: basis sets, lattice constants, convergence tests… may This may allow direct comparison between previous and current calculations Binding / reaction energies Marker method calculations for electrical levels Structures for better starting points reduceseliminate It reduces the probability of trivial errors – it does not eliminate them.

5. MMG Skills Lecture Series 5 Consistency vs efficiency not at any cost Consistency is good, but not at any cost: inefficient The old parameter set may be inefficient in terms of CPU time per calculation. better designed New parameters may be better designed (e.g. optimised basis sets) for this specific project In the long run it may be better to spend time at the start setting your parameters as this gives you a higher level of confidence in the accuracy of the results, and grounds you in the fundamentals of the calculations repeat You may well find that you have to repeat large numbers of calculations if you have to revise from a sub-optimal setup… Calculating the same basic values twice is unlikely to be exemplary efficiency

6. MMG Skills Lecture Series 6 Consistency vs efficiency What ever you do, it should be agreed with all involved in the project, and you should choose your strategy carefully at the start:Think!

7. MMG Skills Lecture Series 7 Compute machine vs file server This distinction is part of “organisation”, and is important in: duplication Avoiding duplication loss Avoiding loss of data access Allowing access for others involved in the project access Keeping access to important files for yourself calculations Some computers are there for calculations: Rhodes, Verity, Braid, Hector, HPCx, … store files Other machines are designed to store files: Trueman, Snufkin…

8. MMG Skills Lecture Series 8 File storage We need to answer three questions: Why do we need to store files? Which files do we need to keep? How can we minimise file-space usage?

9. MMG Skills Lecture Series 9 why? File storage: why? Why do we need to store files? Scientific ethics require us to be able to back up our claims! They are one of our chief resources for future research.

10. MMG Skills Lecture Series 10 File storage Which files do we need to keep? Always AIMPRO standard output, plus Bandstructures: bandst.out, bandt.plt EELS / OA: bandst.out, dieln output files Mulliken: bandt.out NEB: maybe res.neb AIMVIEW: bandst.out, dump files (careful with these) DDS: maybe derivs.txt DoS: bandst.out, dos.out, dos.dump Keep all parts of a run If you have to restart a relaxation, for example, keep all parts. What don’t we keep in our permanent filing: Restart dump files! fort.99, standard error files (e.g. aim2.3.01b.sh.e18371 ) specifically for this run Aimpro input files ( dat, pseudo-pots, hgh-pot, bandst.dat,…) unless they are modified specifically for this run, and cannot be re- created from the aimpro output.

11. MMG Skills Lecture Series 11 File storage How can we minimise file-space usage? Most files we generate are ASCII They can be reduced in size by compressing them (we’ll look again at this in another lecture): bzip2 (You can learn about this by typing man bzip2 on snufkin.) without any loss of data A typical aimpro output file may be reduced in size by 80% without any loss of data. ‘Bzipped’ files can be viewed, gres’d and even edited. quota You have a quota on most machines If you exceed it, you’ll be unable to do very much You may be prevented from logging into the machine! you may fill the disk space and affect all other users If there is no quota, you may fill the disk space and affect all other users.

12. MMG Skills Lecture Series 12 hierarchy of costs Calculation strategy: hierarchy of costs The majority of the AIMPRO computational time is taken in obtaining total energies: The self-consistent cycle. We focus on this part of the calculation for speed. Obtain ρ in Generate Ĥ Obtain ρ out ρ in =ρ out ? Done

13. MMG Skills Lecture Series 13 hierarchy of costs Calculation strategy: hierarchy of costs sequential Commonly, a goal may be reached in a sequential method, minimizing computational effort… science …we want to maximise the amount of science we can do so we need to be able to answer the following questions: How do the decisions we make in constructing a data file, and how we run the job affect the time & efficiency? Which factors are most important?

14. MMG Skills Lecture Series 14 hierarchy of costs Calculation strategy: hierarchy of costs Number of atoms Number of different species Self-consistency method Number of basis functions Number of exponents Maximum orbital angular momentum Initialisation charge density basis Number of k-points Location of k-points Spin state Plane-wave basis Number of processors The amount of memory available K-point parallelism Number of symmetry operations DIIS Real-space build The amount of vacuum (molecules and surfaces) Number of images in a NEB Number of k-points in a band structure Number of bands in an EELS run Number of energies in an EELS run Number of atoms included in the derivatives

15. MMG Skills Lecture Series 15 hierarchy of costs Calculation strategy: hierarchy of costs Number of atoms Number of different species Self-consistency method Number of basis functions Number of exponents Maximum orbital angular momentum Initialisation charge density basis Number of k-points Location of k-points (real/complex) Spin state (vacuum) Plane-wave basis (vacuum) Number of processors The amount of memory available K-point parallelism Number of symmetry operations (extensive parallelism) DIIS (extensive parallelism) Real-space build The amount of vacuum (molecules and surfaces) Number of images in a NEB Number of k-points in a band structure Number of bands in an EELS run Number of energies in an EELS run Number of atoms included in the derivatives

16. MMG Skills Lecture Series 16 hierarchy of costs Calculation strategy: hierarchy of costs SCF For an SCF step: The time for an energy scales as n α, α~2 → 4 where n is the dimension of the Hamiltonian. Going from real arithmetic to complex (at a general k-point), the time increases by a factor of ~½-1 order of magnitude. Sampling the Brillouin-zone mp2 3 generally includes 4 (complex) points for a cubic cell. Spin polarisation doubles the time taken The number of SCF steps for an energy may increase with spin-polarisation. Compare the time for an energy for a pppp, pdpp, ddpp and dddd basis. Compare the time for pppp, gamma point, spin averaged, and dddd, 4 complex k-points, spin polarised.

17. MMG Skills Lecture Series 17 hierarchy of costs Calculation strategy: hierarchy of costs The most common calculation we perform is a structural relaxation. accurate starting structure If we do not have an accurate starting structure, it is likely that the optimisation will require multiple optimisation iterations. Why is this likely to be the case? independent The number of structural iterations is approximately independent of how we run AIMRPO, provided the calculation is performed within a ‘reasonable’ set of parameters (i.e. not necessarily convergent ones). What does this tell us about the energy surface? more SCF steps A structural iteration generally takes more SCF steps when we’re far from the structural minimum. This is related to recycling charge densities from previous structures – why does this affect the number of SCF cycles?

18. MMG Skills Lecture Series 18 hierarchy of costs Calculation strategy: hierarchy of costs Example: Substitutional gold in silicon. We want to know the symmetry, donor and acceptor levels. We need to check the convergence with Cell size Basis Pseudo-potential – whether the 5d electrons are included in the valence or not How do we start? Sketch out a strategy to get all the data we need.

19. MMG Skills Lecture Series 19 hierarchy of costs Calculation strategy: hierarchy of costs Basis Design a hierarchical basis set sequence: C44G* → pdpp → dddd How do we know where we can start? And how far we need to go?

20. MMG Skills Lecture Series 20 hierarchy of costs Calculation strategy: hierarchy of costs Sampling viable Start with simplest viable sampling scheme: Gamma-point, mp2 3,… Use k-point parallelism The Hamiltonians for different k- points are diagonalised in parallel, rather than in serial parameter{use_kpar} If n k <n p, then n p must be an integer multiple of n k. If n k >n p, then this is not the case. viable How do we know what is “viable”? How does this improve efficiency? What are the potential pitfalls? k1k1 k2k2 k3k3 k4k4 k4k4 k4 k4k4 k4k4 k3k3 k3k3 k3k3 k3k3 k2k2 k2k2 k2k2 k2k2 k1k1 k1k1 k1k1 k1k1

21. MMG Skills Lecture Series 21 hierarchy of costs Calculation strategy: hierarchy of costs Spin polarisation For spin-polarized problems, first relax spin averaged – this makes the calculation run twice as fast. What does spin averaged / polarised mean? Can we always relax S=0? Is it always really helpful?

22. MMG Skills Lecture Series 22 hierarchy of costs Calculation strategy: hierarchy of costs Supercell size Start with a small unit cell and embed it in larger ones: 64 → 216 → 512 → 1000 atoms anchor point Use an anchor point symmetry Take care over symmetry What are the implications for timing?

23. MMG Skills Lecture Series 23 hierarchy of costs Calculation strategy: hierarchy of costs Starting structures continued… Use the structure obtained from one charge state to start others. Recycle similar systems: e.g. use a phosphorus structure to start an arsenic one. If you’ve already run in LDA and you now want it GGA, scale the structure according to the ratio of standardized lengths (typically lattice constants). What sort of errors might these short-cuts lead to, if any?

24. MMG Skills Lecture Series 24 hierarchy of costs Calculation strategy: hierarchy of costs Recycle the charge-density! When restarting an incomplete relaxation, or restarting at the end of the relaxation to get some analytical data (e.g. AIMVIEW dumps, band- structures, EELS spectra...) restart-dump Use a “restart-dump”! What does the restart dump contain? restart{make-dump} restart{load-dump} restart{load-dump,override-positions} How can the restart be used? How much time might this save?

25. MMG Skills Lecture Series 25 more generally Restarting calculations: more generally always Your default mode of operation is to always write to a restart dump using: restart{make-dump,file=dump.xxxxx} You might store them all in one place filespace{/scratch/njpg/DUMP-FILES} This is what I do! The xxxxx is a unique identifier associating the dump-file with the run. very large The restart dump files may be very large in terms of disk-usage! Restarts also exist for other calculations: NEB This will be discussed in detail elsewhere in this course Energy second derivatives with respect to position You should check the on-line documentation

26. MMG Skills Lecture Series 26 Sometimes we have to bite the bullet apple(?!). After all this…

27. MMG Skills Lecture Series 27 Throughput vs turn-around time parallel job When running a parallel job, there is a scaling penalty: will NOT Doubling the number of nodes will NOT half the time (in general). Why is this? WILL It WILL prevent other people from using the nodes. throughput turn-around You will constantly balance throughput (the number of jobs completed per day) with turn-around time (wall time from start to job completion). In general: throughput is maximised by adopting the minimum number of nodes for the job, given the batch queue time limits turn-around time is minimised by adopting larger numbers of nodes (subject to scaling). anti-social behaviour Minimising the turn-around time is anti-social behaviour. Big brother is watching you.

28. MMG Skills Lecture Series 28 batch queue Interacting with a batch queue system The batch queue systems differ from machine to machine. It is important that you familiarise yourself with: Memory per node CPUs/cores per node Time limits Scheduling priority Disk-usage Scaling (interconnect) Reliability How do these relate to the preceding discussion?

29. MMG Skills Lecture Series 29 Interacting with a batch queue system Every day, check all of your jobs Running Finished For those still running, check to see: Is all well? Can/should this run be terminated? When might this be? Maximise the available resources for all users For those finished: Restart them, if need be Incomplete relaxation Analysis File them carefully

30. MMG Skills Lecture Series 30 Concluding thoughts Be organised Be hardware aware Be efficient Be sociable THINK THINK about what you’re doing.