1. June 21-25, 2004Lecture 1: Basic Skills1 Lecture 1 Basic Skills Presenter Name Presenter Institution Presenter email address Grid Summer Workshop June 21-25, 2004

2. Lecture 1: Basic Skills 2 Welcome! You are at the Grid Summer School  If you are looking for the underwater basket weaving classes, they are two block to the west. Sponsored by:  Center for Gravitational Wave Astronomy  GriPhyN  iVDGL  NMI GRIDS Center

3. June 21-25, 2004 Lecture 1: Basic Skills 3 Goals for this week Learn about Grids  What is a grid?  How do you… Submit jobs? Transfer data? Cope with failure?  Where are Grids going? Use grids  Hands-on exercises  Local and remote grids Format of the week  Morning Lecture Exercises  Lunch! Mmm…  Afternoon Lecture Exercises

4. June 21-25, 2004 Lecture 1: Basic Skills 4 Day 1 Basic Skills What is a Grid? Basics of Using a Grid

5. June 21-25, 2004 Lecture 1: Basic Skills 5 Day 2 Running a real application on a grid  What requirements do real applications have?  How should you build a real application? Taking care of an application  Condor-G  DAGMan  Staging your application

6. June 21-25, 2004 Lecture 1: Basic Skills 6 Day 3

7. June 21-25, 2004 Lecture 1: Basic Skills 7 Day 4

8. June 21-25, 2004 Lecture 1: Basic Skills 8 Day 5

9. June 21-25, 2004 Lecture 1: Basic Skills 9 Without further adieu...

10. June 21-25, 2004 Lecture 1: Basic Skills 10 Step 1: Basic Networking Skills We apologize in advance if you already know these concepts. We want to make sure that there is a certain basic level of understanding for all students.

11. June 21-25, 2004 Lecture 1: Basic Skills 11 IP Addresses All computers on the Internet use TCP/IP. All computers have at least one IP address.  32-bit number  Written as four numbers, like: 128.105.3.61 An IP Address identifies a network interface, not a computer.  A computer can have multiple IP addresses.

12. June 21-25, 2004 Lecture 1: Basic Skills 12 DNS DNS maps IP addresses to names, and vice-versa  www.amazon.com  207.171.163.30  Discover this with “host” or “nslookup” or “dig”  Try all three—how do they differ?

13. June 21-25, 2004 Lecture 1: Basic Skills 13 Whois Who owns or runs a domain? Amazon.com: Amazon.com, Inc. (HOS276-ORG) hostmaster@AMAZON.COM PO BOX 81226 SEATTLE, WA 98108-1300 US +1 206 266 4064 fax: +1 206 266 7010 Discover this with “whois”.

14. June 21-25, 2004 Lecture 1: Basic Skills 14 Ping! Are you awake? Is a computer on the network? Use ping to find out % ping www.cs.wisc.edu PING marzipan.cs.wisc.edu (128.105.7.11) 56(84) bytes of data. 64 bytes from marzipan.cs.wisc.edu (128.105.7.11): icmp_seq=1 ttl=63 time=0.327 ms 64 bytes from marzipan.cs.wisc.edu (128.105.7.11): icmp_seq=2 ttl=63 time=0.241 ms --- marzipan.cs.wisc.edu ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 2611ms rtt min/avg/max/mdev = 0.241/0.276/0.327/0.036 ms Note that a computer can have multiple names

15. June 21-25, 2004 Lecture 1: Basic Skills 15 Internet routes Between you and a computer on the network, there is an often complex route. % traceroute www.cs.uwm.edu traceroute to miller.cs.uwm.edu (129.89.143.24), 30 hops max, 40 byte packets 1 svi-121.cisco1.cs.wisc.edu.105.128.in-addr.arpa (128.105.121.248) 0.423 ms 0.242 ms 0.227 ms 2 rh-cssc-b280c-2-core-vlan-492.net.wisc.edu (144.92.128.186) 0.404 ms 4.985 ms 0.489 ms … snip… 6 r-uwmilwaukee-isp-atm1-0-1.wiscnet.net (140.189.8.2) 2.730 ms 2.603 ms 2.689 ms 7 space-needle-mke.csd.uwm.edu (216.56.1.194) 2.836 ms 2.718 ms 2.748 ms 8 miller.cs.uwm.edu (129.89.38.24) 2.754 ms * 2.796 ms

16. June 21-25, 2004 Lecture 1: Basic Skills 16 Port numbers A port number indicates which program to talk to on a computer. Some port numbers are standard:  HTTP (web): port 80  SMTP (mail): port 25  Ping: port 7 Some port numbers are assigned dynamically when you run a server.

17. June 21-25, 2004 Lecture 1: Basic Skills 17 Netstat Netstat can answer the question: is a program running on a port on the local computer. netstat --protocol=inet –l tcp 0 0 *:finger *:* LISTEN -l meant “listening for connections”. Look for active connections: netstat --protocol=inet | grep ssh % netstat --protocol=inet | grep ssh tcp 0 0 chopin.cs.wisc.edu:ssh ppp-67-38-160- 108:20715 ESTABLISHED tcp 0 0 chopin.cs.wisc.edu:ssh 68.185.181.47:1176 ESTABLISHED …

18. June 21-25, 2004 Lecture 1: Basic Skills 18 Telnet Telnet isn’t just for remote access to a computer Telnet can tell you if some remote services are running correctly. Is ssh running?  Find ssh port number in /etc/services. It’s 22.  telnet 22. Example: telnet beak.cs.wisc.edu 22 Trying 128.105.146.14... Connected to beak.cs.wisc.edu (128.105.146.14). Escape character is '^]'. SSH-1.99-OpenSSH_3.6.1p2 ^] (That is control-right bracket) telnet> quit

19. June 21-25, 2004 Lecture 1: Basic Skills 19 Telnet & HTTP (Just for fun) % telnet www.cs.wisc.edu 80 Trying 128.105.7.31... Connected to www.cs.wisc.edu (128.105.7.31). Escape character is '^]'. GET http://www.cs.wisc.edu/~roy/index.html Alain Roy … snip …  I typed the lines in red, and got back a web page.

20. June 21-25, 2004 Lecture 1: Basic Skills 20 Why are we telling you this? We have a shameful secret:  Maintaining a grid is hard work. Things break. You will get involved in debugging them. You can use all of these commands to figure out where a problem lies.

21. June 21-25, 2004 Lecture 1: Basic Skills 21 Example Bad complaint: Waaah!!! My grid server-thingy is down Good complaint: The Globus gatekeeper isn’t running, even though the computer is up.  Is the computer up? Use ping  Is the computer accessible on the network? Use traceroute  Can you telnet to the gatekeeper?

22. June 21-25, 2004 Lecture 1: Basic Skills 22 Grid Without a Grid (GWAG) Now we’re going to learn how to do grid work without using grid technology. What is a grid? We’ll tell you this afternoon. For now:  Run jobs on other computers  Transfer data  Discover information  Do it all securely Could we have thought of a worse acronym? GWAG?

23. June 21-25, 2004 Lecture 1: Basic Skills 23 GWAG: Security Explain OpenSSL Explain OpenSSH Explain HTTPS Note password authentication Describe ssh keygen stuff, to eliminate passwords. Note how this has to be set up on a per-computer basis. This needs to be fleshed out.

24. June 21-25, 2004 Lecture 1: Basic Skills 24 GWAG: Job submission ssh can run jobs remotely:  ssh roy@beak.cs.wisc.edu ls This expects a few things:  Your program already exists on the remote computer  A shell is invoked to set up your environment, so you can find ls.

25. June 21-25, 2004 Lecture 1: Basic Skills 25 GWAG: File Transfer How do you transfer data to be executed? scp executable roy@beak.cs.wisc.edu:.  Note the final period at the end of that line How do you transfer the output back? scp roy@beak.cs.wisc.edu:output. Submitting your job takes a minimum of three commands. If you don’t set up your keys, you’ll type your password three times.

26. June 21-25, 2004 Lecture 1: Basic Skills 26 GWAG: Issues Issues you will find:  How do you run three jobs sequentially that require data transfer? You just run each command in a row.  How do you run three jobs in parallel that require data transfer? Make each transfer/job/transfer a script. Run the scripts in parallel from the command line  How do you deal with failures? What if scp or ssh returns an error? What if scp or ssh never returns? (It hangs)  How do you keep track of everything? In an ad-hoc way?

27. June 21-25, 2004 Lecture 1: Basic Skills 27 GWAG: Dealing With the Issues We will talk about grid solutions to these problems. Keep an eye out for:  GSI for security  GRAM or Condor-G for transfer of input/output files  Condor-G for job reliability  DAGMan for running sets of jobs