1. An Introduction to jhhkjhjhjhhjh
For New or Inexperienced Users
January 16, 2018
Good afternoon. My name is Virginia Trueheart. I’m a RESA at TACC and I help out with some of the training we provide to users as well as assisting them directly through the ticketing system. This presentation is designed to be a basic introduction to using Jetstream. Several presentations have been given before that focus on the more advanced options available on Jetstream like using the API but this is geared specifically towards users who are only just beginning to interact with HPC resources and may be more comfortable working through GUIs and other controlled environments. Jetstream is a great starting point for new students or researchers in fields that have lower exposure to HPC thanks to it's web interface and GUI options. (Ex. If asked: biology, anthropology, sociology, statistics, etc as opposed to chemistry, physics, astronomy)
Virginia Trueheart
11/8/201811/8/2018

2. Overview What is Jetstream? Architecture Accessing the System
Virtual Machines and Images
Using the Resources
As a quick run down, I’ll cover what Jetstream is and how it is set up as well as how to access the system. I’ll then go more in depth about how to select and use images and make the most of your allocation.
11/8/201811/8/2018

3. What is ? NSF funded production cloud facility
On-demand interactive computing and analysis
Configurable environments
User-friendly, widely accessible cloud environment
Library of preconfigured virtual machines
What is Jetstream? Jetstream is NSFs first production cloud facility. In short, this means you can access High Performance Computing via your web browser. It provides on demand computing resources to users that are in need of interactive sessions. Most traditional HPC computing is done via batch jobs which requires users to submit their work and then let it run unmonitored. Interactive computing allows users to make real time changes and more efficiently troubleshoot any issues. This is particularly valuable to new users who may still be learning how to use field specific software. Additionally, Jetstream provides fully configurable environments. Users have root access and can make any modifications they need and are not restricted by the administrative choices of a particular computing center. This system is designed to be user friendly in order to accommodate users of all experience levels and also make computing and analysis available anywhere. If you have an internet connection, you can run on Jetstream no special access is required. Finally, Jetstream hosts a large shared repository of images that can be deployed by any user. This makes it easy to get started and also provides templates for users to go on and build more complex images customized to their work.
11/8/201811/8/2018

4. Architecture Overview
Jetstream is a cloud system but as we know, the cloud is just other people’s computers. In this case, computers at Indiana Uni., TexAdvComCen, and the University of Arizona. A small subset of development nodes reside at UofA but it's rare that you would ever use them. The computational nodes are divided evenly between IU and TACC. These nodes are identical so it’s unlikely to matter which you choose to run on. That said, there are some images that are tied to a specific location in which case the drop downs will self select the appropriate option when you attempt to launch an instance of the image.
As you can see from the flow chart, each of the production clusters connect to Internet2 at 100 Gbps and the physically distributed system allows Jetstream to be highly available and resilient. This is a benefit of all cloud computing. By spreading out the location of the nodes, it’s unlikely that the resources would all become unavailable at the same time due to something like a power outage. If TACC is down you should still be able to run on the IU cluster and vice versa.
11/8/201811/8/2018

5. Accessing the System
Trial Access Allocation: access ECSS Staff Allocations: ASC & ASC URL: When you login choose XSEDE from the drop down and use your XSEDE credentials to login to the system. You may need to Allow the Jetstream Web App access to all Jetstream resources if you haven’t ever logged in before.
So now that you know a bit about Jetstream and would like to try it, how do you access it. To access Jetstream you must have an XSEDE account. You’ll also need to request an allocation on Jetstream from XSEDE which you should be able to get almost immediately with a Trial Access Allocation. Additionally, if you’re ECSS staff then you can be granted access to either of the Staff Allocations listed here. Once your allocation is approved, you’ll use your XSEDE credentials to log into the Jetstream site and after you have agreed to terms and allowed the Web App access, you will be placed on a dashboard that shows your allocation and the options available to you on the system.
11/8/201811/8/2018

6. That dashboard looks like this
That dashboard looks like this. This dashboard helps you keep track of your overall allocation usage, and how many instances you have running. This is also the easiest place to launch a new instance. Help resources will direct you to user guides and forums if you are experiencing issues or need more details about how something works, and Settings allows you do things like update your icons or store ssh keys. First though, we’ll review your allocation under the Projects tab.
11/8/201811/8/2018

7. Projects
Under the Projects tab you’ll be able to see cards with the various allocations you’re associated with. You can click on any of these cards and be taken to a page with more details about the allocation. Here, you can see what the details of my demos allocation card looks like. I've got one instance of Matlab running but no volumes, images or links stored. Anything you’ve saved or are running under an allocation will be listed here, so you can come back and access the items via this tab. Now that you know where to find your projects, lets talk about how I launched that instance.
11/8/201811/8/2018

8. Virtual Machines
These configurations are available when choosing a VM.
You should always request the smallest VM size that can accommodate your work.
Local storage is ephemeral and will be lost when your session dies.
For long term storage you will need an XSEDE-allocated Volume.
Before you launch an instance you’ll want to evaluate what kind of resources your jobs will need and weigh that against your available allocation. This chart displays all of the VM sizes available and can help you determine which size best meets your needs. It’s always a good idea to choose the smallest VM size at which your work can be completed. First, because it helps preserve your SUs and will help you make the most of your allocation, and second because it leaves more resources free for other users. While there is no queue on Jetstream like there is on a traditional computing system, there are still a limited number of resources and if they are all in use it is possible that it will take longer for your instance to launch. (still usually on the order of minutes not hours) Additionally, the more resources you request, the longer it may take for the instance to acquire them. Once it is launched, however, you have access to those nodes until you decide to release them. Again, remember you’re working on shared system so try not monopolize the resources.
In addition to the number of CPUs available, there is also a limited amount of local storage on each VM so it’s best to make sure you have selected a VM with enough space to handle what you’re attempting to do. Please note, however, that this storage only exists so long as the instance is running. Once your session is killed, all data saved on it will be lost. Be sure to back up your data externally, or request a storage allocation from XSEDE who will provide you with access to a Volume.
11/8/201811/8/2018

9. Images
In order to launch an instance, you will need to choose an image by going to the Images tab. There are a wide variety of images available and users add new public images regularly. If you know you need a specific piece of software like Matlab or Anaconda, then you can search for those terms and select an image that meets your needs from the tagged images. There are an immense number of tag options so feel free to dig around and see which ones apply to you and what other people find useful in their images (bioinformatics, specific operating systems, software, docker containers, etc)
11/8/201811/8/2018

10. Images cont.
When you select an image, you will be taken to a page that displays more details about the image. In this case, you can see that this image of Matlab is based on CentOS6 and requires an m1.medium VM as it’s minimum size. This means that you must select that image size or larger in order to run. You can also see at the bottom of the page that there are multiple versions of this instance available so if you know your code only works on a specific version you can select that at launch time. If you decide you want to use this image then you select Launch in the upper right hand corner.
11/8/201811/8/2018

11. Images cont.
After clicking Launch you will need to select options specific to the instance you want to run. In this case, I want to use v instead of an older version, I’m going to run on the nodes at IU and I want an m1.medium VM. This is also actually an example of an image that’s bound to a specific provider and IU is the only option available in the provider drop down. The same goes for the instance size. Since this requires an m1.medium or larger, the smaller options are not available in the drop down. Once you’ve configured the image to your liking, click launch instance at the bottom.
11/8/201811/8/2018

12. Images cont.
It can take a while for an instance to become active based on the state of the system and how many resources are being used. By default, the system should you once an instance becomes active so you can take care of other tasks and just wait for the notification. You can also see the status of the image under Instance Details Once the instance is active you can click on it and be brought to this summary page. From here, you can choose how you want to interact with it. For this example, I’m going to use the Web Desktop option at the bottom of the list. This will generate an emulation of a desktop that has the resources of the image available on it.
(if you lost your place you can go see your current images with Projects > select a project > instances > select desired instance)
11/8/201811/8/2018

13. Images cont.
So I've clicked on Open Web Desktop and as you can see it looks exactly like any other desktop, but has the added bonus of being preloaded with Matlab (extra added bonus, no license needed). You can now double click on matlab to open the application and run the software exactly as you would on your own machine. Depending on the size of the VM you selected, you should be able to run much larger computations than you would on your personal laptop since you’ll have access to more CPUs.
11/8/201811/8/2018

14. Modifying an Image
Now we’ll take a look at how to modify a base image to meet your personal requirements. This particular display, while similar to the last, is of a basic CentOS 6 Development image (no matlab). I’ve opened up the Web Desktop again, and then opened a terminal emulator which is under the Applications menu at the top under System Tools. Because this Image is basically a blank slate, it doesn’t have the samtools package that I want to use in post processing of some DNA sequences. Since I have root access when using a VM I can sudo to root and make use of yum to install samtools.
11/8/201811/8/2018

15. Modifying an Image cont.
So the yum install works and now Samtools is available everywhere on this image.
11/8/201811/8/2018

16. Saving an Image
Now that this image has samtools added to it, I want to save it so I can reuse it later without having to reinstall the software. In order to save an image, the instance must be active or stopped. Here you can see, I’ve stopped the instance and now I’m going to select Image under the Actions menu to save it.
11/8/201811/8/2018

17. Saving an Image cont.
When you select Image, you'll be given a popup that will take you through several steps. Here you can see the different stages laid out. Be sure to choose a unique name for your image, and document all of your modifications. This will help you when trying to sort through images later, and will help other users who might want to use your image if you have made it public. In this case, I’ve tagged the image with samtools and centos and made the image private since this is just a demo.
11/8/201811/8/2018

18. Your Image
Once you've documented the image and hit submit. your image will be sent to support for review and publication. You can review the status of your request under the images tab and once it has been published you’ll be able to see it under the My Images tab. Once it’s available, you’ll want to test the image and make sure it looks how you want it too before you kill your original instance. Any image that is Deleted or Shelved can’t be processed and can cause issues with the Image processing queue if you delete an Image after requesting it be saved. Eventually this process will be automated but for now, just leave the instance in a stopped state if you’re done and come back to disable later after you’ve determined the saved image is valid. The average wait time for an image to save is 1-2 hours so while your SUs are still being used, it should only take a couple of them at most. The rate of a stopped image is probably going to be lowered. Once everything is to your liking, you can kill the running image and just come back to the saved version whenever you need it.
11/8/201811/8/2018

19. Reminders The longer your instance runs the more SUs you consume
Be aware that you are on a shared resource
You have root access, but the original image stays intact
This is only the basic level of what you can do with Jetstream; there are many more options available in the Wiki
So that's the general run down on how to use Jetstream. A few reminders: The more resources you request and the longer you run the more SUs you consume and the faster you’ll run through your allocation. You’ll also be holding resources that other people may be waiting to use. If you need to run for extended periods of time, you certainly can but it’s best to keep limited.
While you can make any changes you want via root access, if you don’t save the image, everything will be lost once you kill the instance. Back up your work or you’ll have to start from scratch.
The wiki has detailed descriptions of even more functions for jetstream and also provides a link to the jetstream forum where you can confer with other users.
11/8/201811/8/2018

20. https://iujetstream.atlassian.net/wiki/spaces/JWT/overview
Q&A
11/8/201811/8/2018