1. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 1 General Overview.. The CCP4 Package.

2. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 2 On What Systems Will It Work?  SGi: Irix (o32, n32 and 64bit)  Linux Systems (Redhat, SuSE…..)  Compaq OSF1 etc  Sun (Sun OS)  HP (All apart from Xwindows)  IBM (AIX)  Mac OSX  Microsoft Windows  +

3. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 3 Where It Can Be Downloaded  CCP4 ftp site:  ftp://ftp.ccp4.ac.uk ftp ftp.ccp4.ac.uk user name: ftp pw your@email.address.please cd pub/ccp4/current/ binary  CCP4 mirror sites:  Japan ftp://ftp.protein.osaka-u.ac.jp/mirror/ccp4  USA ftp://ftp.sdsc.edu/pub/sdsc/xtal/CCP4

4. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 4 How BIG Is the Download?  unpacked size =160Mb  tar.gz size = 36 Mb  tar.Z size = 56 Mb  Binary size = 25 – 107 Mb

5. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 5 Installation  Binaries or source code?  Unpacking  Directory structure  ccp4.setup  Configure  Source Code Make  Binary Make  Windows  Installing on multiple systems

6. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 6 Installation II  You do not need root privileges on UNIX systems  Just enough space!  You DO need Admin privileges on Windows NT/2K  This is used to set up environment variables etc.  What else do you need to install CCP4?  Source code installation:  Compilers C and FORTRAN.  Make (UNIX)  Gzip/Uncompress/tar (UNIX)  ccp4i requires tcl/tk/BLT  AstexViewer requires a Java VM.

7. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 7 Binaries or Source Code? BinarySource Install (inc ease) speed   Download speed  Developers (Customisation)  Range of systems/portability  Compiler problems/price  Ease of support 

8. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 8 Binaries or Source II? Supported Binaries? yes no Developer? Fast Net? Source Code and Compile yes no Reliable Compiler? yes no yes Binary + Source code no (Currently supported: irix, linux, OSF1, SunOS, Windows) Binaries or source?

9. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 9 Unpacking  Download the file into an area with enough space to download and unpack it.  To uncompress a.gz file use gunzip  To uncompress a.Z file use uncompress  A tar file is a collation of many files: To extract the files from a tar file use:  tar –xvf file.tar

10. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 10 Directory Structure ccp4_version.tar xtal/ccp4_version/ ccp4ietcincludelibsrc unsupported x-windows doc, examples, help, html, man, manual CCP4 interface, Interface documentation, roadmaps CCP4 utility scripts ccp4.setup, environ.def and default.def Library src and data Program Source Code bin Older programs that are being phased out Programs with graphical user interface that require x-windows

11. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 11 ccp4.setup  All CCP4 programs require ccp4 environment variables set-up to enable them to run.  ccp4.setup files found in include sub directory  ccp4.setup-dist and ccp4.setup-bash  It is essential that the following environment variables are correctly assigned:  CCP4_MASTER (where the ccp4_vers.tar was unpacked)  CCP4_SCR (CCP4 scratch directory),  CCP4I_TCLTK (where ccp4i finds bltwish)  Most other environment variables use these variables.  source ccp4.setup before configure and running CCP4 programs  Automatically source ccp4.setup in login files (.cshrc or.login) (UNIX variants only)

12. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 12 ccp4.setup II BINSORT_MEM Size of memory to use for sorting before suing disk CCP4I_HELPUsually $CCP4/ccp4i/help BINSORT_SCRDisk area to use for sortingCCP4I_TCLTKWhich bltwish to use CBINWhere ccp4 binaries will liveCCP4I_TOPUsually $CCP4/ccp4i CCP4Top directory of ccp4CHTML$CCP4/html CCP4_BIN$CBINCINCL$CCP4/include CCP4_LIBWhere ccp4 libraries will liveCLIBD$CCP4/lib/data CCP4_MASTERUsually $CCP4/../CLIBS$CCP4/lib/src CCP4_OPEN Overwrite files or not. UNKNOWN = overwrite CPROG $CCP4/src home of the ccp4 program code CCP4_SCR Where ccp4 programs will write temporary files. PUBLIC_FONT84 $CCP4_LIB/font84.dat a font data file. Environment variables There is no need to learn these (and there are more) unless things go wrong!

13. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 13 configure  Configuring allows system specific flags to be assigned  Configure gives you flexibility, and a quick and easy way to change all the important make files  Configure settings can be overridden by environment variables  configure + Makefile.in  Makefile + config.status  Once the system has been configured you can either make the source code or install the binary files.  A log of the configure settings are in config.status.  More than one configure is run by $CCP4/configure (UNIX variants only, when making from source code)

14. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 14 configure (II) configure [--flags] [operating system] Operating system (choice of 17!) --flags (choice of 15!) meaning irix, irix64SGi--with-x Compile the X programs linux /_compaq_compilers Linux--{*}dirsrc/bin/lib/tmp hpux Hewlett Packard --with-shared-libCompile shared libs osf1Compaq--only-libsOnly compile libs DarwinMac OSX--with-warningsOne for the developers! sunos Sun Microsystems

15. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 15 Source Code Make (installing from source code only)  Once you’ve downloaded the package, unpacked, set the environment variables and configured the package you are now ready to make the suite!  In $CCP4 type make.  You might want to keep a log with make > make.log  Make will firstly make the library components before the programs.  Once complete you will need to install with  make install.

16. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 16 Source Code Make II.  The make utility uses the settings in the Makefile which in turn was created from the Makefile.in by the configure script.  $CLIBS/fsplit – a small program used to break large library.f files into smaller files – this was needed on older systems.  Make in the x-windows directory.  The make in the x-windows directory will ignore errors in compiling programs and will carry on regardless.  Once complete you will need to install with make install.  This will install the programs into $CBIN  Moves the library files into $CCP4_LIB / $CLIB  Make also has numerous other options.

17. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 17 Source Code Make III make [option] OptionMeaning installInstalls the components into the correct area [program]Will make the program if its been updated clean, realclean, distclean Various degrees of severity of cleaning the directory structure. uninstallWhat it says on the box….. empty-targetsSets the size of e.g. $CPROG/[program] to zero

18. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 18 Source Code Make IV (reference only) Make option cleanRemoves *.o, *.~, etc realclean clean + any executable files in the directory distclean Realclean + removes Makefiles, config.status -> only distribution left uninstall Removes installed files created by the install step

19. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 19 Source Code Make V - Compilers  Under the majority of operating systems the native compilers should be used.  The GNU compilers gcc and g77 can used on many systems  Redhat 7.0 and 7.1 users should upgrade their gcc/g77  Use gcc 3.0.<4  Linux users can also use a variety of other compilers.  See recent ccp4bb conversations  (e.g Portland Group and Intel compilers).  Irix 6.5 users should not use f77 version 7.2.*  We currently use 7.3.1.2m

20. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 20 Binary Code Make  You should have:  $CCP4/BINARY.readme  $CCP4/BINARY.setup  $CCP4/bin/  You need to:  Get normal distribution and unpack it  Edit and source ccp4.setup  Run the script BINARY.setup  See also http://www.ccp4.ac.uk/dist/INSTALL.html  BINARY.about file in ‘binary’.tar.gz (UNIX variants only)

21. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 21 Windows  Tested on Windows 95/98/Me/NT(4.0)/2000  You will need Admin privileges  Windows NT and 2000  Use.exe auto install  Windows 95/98/Me variants  Use compressed.zip file  You will also need Tcl/tk and BLT for ccp4i (install before ccp4)  http://www.ccp4.ac.uk/dev/ccp4onnt.html  Cygwin is possible.

22. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 22 Installing on Multiple Systems  Multiple systems means e.g. Linux and Irix with shared/mounted disks  Install completely independent source tree  Use duptree (found in $CCP4)  duptree must be run before any files are altered in $CCP4  duptree creates a second identical source tree using links  Use --srcdir  Why?  Size matters.  You only need to bug fix one src.

23. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 23 Bug Fixing and Patching. http://www.ccp4.ac.uk/problems.html

24. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 24 More Information  CCP4 manual.  CCP4 website.  http://www.ccp4.ac.uk/dist/INSTALL.html  CCP4 problems pages.  ccp4@ccp4.ac.uk

25. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 25 Now What Have You Got?  The programs  How they communicate  CCP4 Data formats  How to use the programs  The main programs  What is new in 4.2

26. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 26 The Programs  Most programs written in standard(ish) fortran77  Over 150 programs mostly runnable from command line  Full listings on:  http://www.ccp4.ac.uk/dist/html/INDEX.html  http://www.ccp4.ac.uk/dist/html/FUNCTION.html

27. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 27 How They Communicate? Standard data formats that contain information and informative log outputs. The data formats are publicly known, documented and useable by anyone. Nobody is forced to use our data formats as we supply numerous conversion tools. Some CCP4 programs already output key information in XML files.

28. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 28 CCP4 Data Formats.  Formats used are: .mtz – reflection data .map.msk – map and mask formats .pdb – atom positions .cif – harvest files, refmac5 restraints  Binary formats were used for:  Size  Security  MTZ/map files are binary but portable as they contain a machine stamp  More information in the FILE FORMATS section on  http://www.ccp4.ac.uk/dist/html/INDEX.html

29. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 29 CCP4 Data Formats II – MTZ Reflection Files  Table of data  columns - I, F, sigma, phase, FOM, etc. etc.  rows - reflections  Columns grouped as datasets with  project nameGerE  dataset name se_inflection  cell dimensions 108.7 61.6 71.6 90.0 97.1 90.0  wavelength 0.981  In ccp4i, see : Reflection Data Utilities -> Edit MTZ Project&Dataset

30. MTZ file Title/history Spacegroup Crystal 1 Crystal name Project name Cell dimensions Crystal 2 Crystal name Project name Cell dimensions Dataset 1.1 Dataset name Wavelength Dataset 1.2 Dataset name Wavelength Column Heirarchical view of reflection data. CCP4 Data Formats III – MTZ Reflection Files II The future....

31. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 31 CCP4 Data Formats IV – MTZ Reflection Files III  Header information  Keyword – value pairs TITLE – title NCOL – no. of data columns CELL – global cell parameters SYMINF – symmetry info COL – column information RESO – resolution range NDIF – no. datasets DATASET – dataset info PROJECT – harvesting info BATCH – batch images TITLE make ccp4itut mtz inconspicuous Sfs from refined RNA_SANAT 8-7. NCOL 8 18077 0 CELL 64.8970 78.3230 38.7920 90.0000 90.0000 90.0000 SORT 1 2 3 0 0 SYMINF 4 4 P 19 P212121 PG222 COLUMN K H 0.0000 42.0000 COLUMN L H 0.0000 21.0000 COLUMN FP F 0.0000 1219.0000

32. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 32 CCP4 Data Formats V – MTZ Reflection Files IV  Viewing and manipulation  mtzdump – view ascii dump of mtz file  mtz2various – convert mtz file to MULTAN, SHELX, TNT, X-PLOR/CN* formats  mtzutils, cad, sftools – edit mtz files  scaleit, fhscal – column scaling  http://www.ccp4.ac.uk/dist/html/FUNCTION.html

33. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 33 CCP4 Data Formats VI – MAP Files - Densities mtz map Fourier phases  fft –switch between electron density and reflection data  mapdump – give information on map  View map with – O, PyMol, Mapslicer, AstexViewer  Format – header + symops + data  Binary - but machine independent.

34. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 34 CCP4 Data Formats VII – Atom Positions  PDB files:  Based on pdb version 2.1 draft  mmCIF  CCP4 uses a subset of the mmCIF dictionary  Used in;  Restraints library of Refmac5 and libcheck  mmdb library  Harvesting files

35. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 35 How To Run The Programs  ccp4i – the ccp4 interface  Command line options:  program [ ]  e.g. fft hklin native-Fs.mtz mapout 2Fo—Fc.map  Keyworded input:  Keyword  E.g. CELL a b c [    ] TITLE this job will work this time

36. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 36 How To Run The Programs II Common Keywords CELL a b c [    ] Cell dimensions in Å RESOLUTION [ ] Resolution limits END Terminates input and starts program SYMMETRY | | HISTORY Adds record to history stack of MTZ LABIN = … Associates the column labels that the program expects with MTZ cols LABOUT =.. Associates column labels used in output file with those in program.

37. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 37 The Main Programs  What do you get?  >150 programs encompassing the majority of protein crystallography – including:  Mosflm, Scala, Truncate, Mlphare, Rantan, Revise, Acorn, Oasis, SC, Amore, Molrep, Beast, DM, Fffear, Refmac5, Procheck, Sfcheck  See http://www.ccp4.ac.uk/dist/html/INDEX.html or http://www.ccp4.ac.uk/dist/html/FUNCTION.html

38. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 38 What Is New in 4.2  ACORN ab initio procedure for the determination of protein structure at atomic resolution (Yao Jia-Xing)  BEAST Brute-force molecular replacement with Ensemble Average Statistics, Maximum likelihood-based molecular replacement (Randy Read)  PROFESSS determination of NCS operators from heavy atoms (Kevin Cowtan)  ROTAMER list amino acids whose side chain torsion angles deviate from Richardson's Penultimate Rotamer Library (Dirk Kostrewa)  ASTEXVIEWER Java application for viewing proteins, ligands and electron density maps (Mike Hartshorn)  Announcement on: http://www.ccp4.ac.uk/news/msg00098.html

39. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 39 An Introduction to CCP4i The CCP4 Graphical User Interface Author: Peter Briggs CCP4 CCP4i Introduction

40. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 40 Offers user friendly interfaces to the programs Integrated help system Tools for file viewing & “project management” How to get started (Unix/Linux) Type ccp4i at the command prompt (Windows/NT) Select ccp4i via the Start menu Why Use CCP4i?

41. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 41 Tasks Modules Job DatabaseTools & Utilities On-line help Main CCP4i Window

42. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 42 Access the module list by clicking on the gold bar displaying the current module Modules

43. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 43 Tasks used in a particular part of the structure determination process Utility tasks for manipulating different types of data Alphabetical list of programs/tasks Modules II

44. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 44 Click on the appropriate button in the list to start a particular task This starts the interface for that task Tasks

45. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 45 Closed folders Advanced/infrequently used options Open folders Parameters that should be checked by the user before running Highlights indicate compulsory input File folder Set input and output file names Protocol folder Make the key decisions WORK FROM THE TOP DOWN Run task Save/restore parameters Always add a title to distinguish different runs of the same task Defaults - “If it’s not visible then it’s not important” Example Task Interface

46. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 46 Watch the progress of the job in the Job Database window: Running a Task

47. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 47 View logfile in Netscape (if it contains HTML tags) and/or text browser View input and output files (.mtz,.pdb, CCP4 maps) using appropriate viewer - also use the View Any File option) - or ccp4i -v at the prompt View graphs in logfile using loggraph - also use loggraph at the command prompt View Output From Job

48. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 48 Why bother? Benefits: - remind you what you did six months ago - keep track of multiple projects and associated data - facilitate “back-tracking” - make it easier to deposit your results/write your paper Project Management Using CCP4i

49. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 49 “All data files relating to one crystallographic project should be in a single project directory” One word alias... … for project directory containing data files Project Directories

50. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 50 Each project directory has an associated job database accessed through the central panel in the main window: This displays: - which tasks were run, and when - their status ( RUNNING, FINISHED, FAILED etc) - the title entered by the user The database also keeps a record of: - the parameters used to run the task - the input, output and log files associated with the task Job Database

51. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 51 View files from any job in the database Remove failed/unwanted jobs from the database and archive important data Rerun any job in the database (with the option of changing the parameters first) Use this to review parameters used in an earlier run Keep the database up-to-date Add runs of “external” programs Job Database Utilities

52. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 52 Notebook: record information for future reference Edit data: e.g. update title of run, change file locations Add external /non-interfaced program run to the database Edit Job Data

53. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 53 General help from main window Help with a particular option - Right hand mouse button click over that option Help for a particular task Brings up relevant documentation in browser On-Line Help

54. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 54 Message line help One-line help message appears in the bar at the top of the window Send e-mail to directly to CCP4 (please send as much information as possible!) More Help….

55. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 55 CCP4 Roadmaps - accessed via program index (under General) CCP4 Tutorial2000 - also from the program index Navigating Modules

56. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 56 On-line help: - help button on main window for general info - help button on task interfaces for help with individual tasks - right-hand mouse button click over part of the window for help on that option or feature. Roadmaps through the structure determination process: - $CCP4/ccp4i/help/roadmaps/index.html Tutorial material - $CCP4/examples/tutorial2000/html/index.html E-mail CCP4 - ccp4@ccp4.ac.uk CCP4 Help Summary

57. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 57 Key programs in the structure solution procedure write out harvest files - in CCP4 these are SCALA, TRUNCATE, MLPHARE, REFMAC and RESTRAIN At deposition time these files represent an accurate record of how the final model was obtained Each file records details of the method used and the results obtained (e.g. heavy atom sites used in phasing) Harvest files can be sent directly to the deposition site, avoiding much manual processing Data Harvesting

58. CCP4 Workshop ACA, Texas 2002 presentation by Alun Ashton, awa@dl.ac.uk 58 Harvesting operates “invisibly” Harvesting requires MTZ files contain Project and Dataset names: - add these when data is first imported into CCP4 ( Import Unscaled Data or Convert to MTZ & Standardise tasks in Data Reduction Module ), or - edit Project name or Dataset name using Edit MTZ Project&Dataset (in the Reflection Data Utilities module) Switch on the harvesting options: - in individual tasks e.g. SCALA, REFMAC, or - set the harvesting defaults in the CCP4i Preferences window (RHS of the main window) - you don’t need to think about it again until the end of the structure solution Data Harvesting in CCP4i