Katie Antypas User Services Group Lawrence Berkeley National Lab 17 February 2012 JGI Training Series

Until all users are migrated to NERSC we plan to hold weekly Friday sessions More on file and data management Open Office Hours Review of batch system policies Crius Rhea Thei a Kronos? Hyperion Oceanus Iapetus Themis Introduction to NIM

On NIM you can change your password, change your shell and set security questions Login to nim.nersc.gov Look under the actions menu to do the above tasks

File systems best practices Unfortunately disk is still expensive All of the JGI’s data can not be stored on disk within the current budget Archive and delete data you no longer need Disk usage will be controlled through quotas in some cases and purging in others

Only the “house” file system will be available on both JGI and NERSC systems initially JGI Space NERSC Space Compute cluster Some submit hosts Most web servers Netapps “projectb” house If your data needs access to both servers in JGI space and the compute cluster, it MUST go into “house” In other words – move data out of Netapps

But “house” is 90% full…… House 90% File systems above 90% are lower performing and at higher risk of failure We need your help deleting data from “house” and moving data from the netapps to “house”

NERSC has set up 2 fast “data transfer nodes” just for JGI users Login to dtn03.nersc.gov or dtn04.nersc.gov Type >df to see all the mounted file systems Back up data to HPSS (you authenticated at last week’s training don’t remember? Type hsi and then enter your NIM password) > cd /house/path/to/your/data > hsi put Or archive an entire directory > htar –cvf tarname.tar directory/

There are two areas of storage within the “project” layout of the “projectb” file system /projectb/ projectdirs/scratch/ PI/ RD/ fungal/ metagenome/ micro/ plant/ comparative/ user/ Group directories Not purged Subject to quota User directories cd $SCRATCH Purged, 12 weeks 1 TB, 500,000 inode quota Request a projectb directory for your group through the Jira ticket system Request a larger /scratch quota through the Jira ticket system ssh phoebe.nersc.gov

Use the fast data transfer nodes to move data between file systems Login to dtn03.nersc.gov or dtn04.nersc.gov Type >df to see all the mounted file systems You can move data to 3 file systems $HOME “project” “scratch” > mv /old/path/filename /new/path/filename

It is important for every group to come up with a data retention policy How long should we keep the raw data? Can the data be deleted or should it be archived? Can we set up an automated way to archive and delete data?

The JGI compute clusters have been consolidated into Crius with the following shares Crius Rhea Theia Kronos? Hyperion Oceanus Iapetus Themis

Users should submit jobs to the normal queue Jobs running longer than 12 hours or requesting large amounts of memory could see longer wait times

Useful commands