1. Computer Systems Lab Courses

2. 2CS 363 GMU Spring 20052 Programming Languages Sebesta Fig. 1.2 Languages are an abstraction used by the programmer to express an idea interface to the underlying computer architecture

3. CS 363 GMU Spring 20053 Why study Programming Languages? Increases ability to express ideas in a language –wide variety of programming features Improves ability to choose appropriate language –Each language has strengths and weaknesses in term of expressing ideas Improves ability to learn new languages –different paradigms, different features –What does the future of programming languages hold? Improves understanding of significance of implementation Provides ability to design new languages –Domain specific languages increasingly popular

4. CS 363 GMU Spring 20054 Language Paradigms Imperative –Central features are variables, assignment statements, and iteration –Ex: C, Pascal, Fortran Object-oriented –Encapsulate data objects with processing –Inheritance and dynamic type binding –Grew out of imperative languages –Ex: C++, Java Functional –Main means of making computations is by applying functions to given parameters –Ex: LISP, Scheme, Haskell

5. CS 363 GMU Spring 20055 Language Paradigms Logic –Declarative  Rule-based – implicit control flow –Ex: Prolog Dataflow –Declarative  Model computation as information flow – implicit control flow –Inherently parallel Event-Driven –Continuous loop with handlers that respond to events generated in unpredictable order, such as mouse clicks –Often an add-on feature –Ex: Java Concurrent –Multiple interacting processes –Often an add-on feature –Ex: Java, High Performance Fortran (HPF), Linda

6. CS 363 GMU Spring 20056 Programming Domains Scientific applications –One of the earliest uses of computers –Large number of floating point computations –Long running –Imperative (Fortran, C) and Parallel (High Performance Fortran) Business applications –Produce reports, use decimal numbers and characters –Increasingly toward web-centric (Java, Perl, XML-based languages) –Imperative (Cobol) and domain specific (SQL) Artificial intelligence –Model human behavior and deduction –Symbol manipulation –Functional (Lisp) and Logical (Prolog) Systems programming –Need efficiency because of continuous use –Parallel and event driven –Imperative (C) …

7. CS 363 GMU Spring 20057 Influences on Language Design Programming methodologies –1950s and early 1960s: Simple applications; worry about machine efficiency –Late 1960s: People efficiency became important; readability, better control structures Structured programming Top-down design and step-wise refinement –Late 1970s: Process-oriented to data-oriented data abstraction –Middle 1980s: Object-oriented programming

8. CS 363 GMU Spring 20058 Compilers Scanner (lexical analysis) Parser (syntax analysis) Code Optimizer Semantic Analysis (IC generator) Code Generator Symbol Table Source language tokensSyntactic structure Syntactic/semantic structure Machine language Comp uter Input Data Output

9. CS 363 GMU Spring 20059 Interpreters Source language Interpreter Input Data Output

10. CS 363 GMU Spring 200510 What makes a language successful? Expressive Power –Included features impact programmer use Ease of use for Novice –Pascal, Basic, Logo Ease of Implementation Excellent Compilers Economics, Patronage, Legacy

11. 11 Comparative Languages

12. CS 363 GMU Spring 200512 FORTRAN - 1957 FORmula TRANslating systems FORTRAN I - 1957 (FORTRAN 0 - 1954 - not implemented) –Designed by John Backus for the new IBM 704, which had index registers and floating point hardware –Environment of development: Computers were small and unreliable Applications were scientific No programming methodology or tools Machine efficiency was most important

13. CS 363 GMU Spring 200513 FORTRAN FORTRAN 90 - 1990 –Modules –Dynamic arrays –Pointers –Recursion –CASE statement –Parameter type checking

14. CS 363 GMU Spring 200514 FORTRAN Contributions –First widely used programming language –Changed the way people interacted with computers –Set the standard for compilers Goal: generate machine code similar to that of machine language programmers  highly optimized compiler Much of the theory of compilers was developed in this compiler.

15. CS 363 GMU Spring 200515 Sample Fortran subroutine defcolor(rgb,nframe) implicit none integer nframe integer ihpixf, jvpixf parameter(ihpixf = 256, jvpixf = 256) ! pixel size character*1 rgb(3,ihpixf,jvpixf) ! RGB data array local integer i, j, idummy do 100 j = 1, jvpixf do 100 i = 1, ihpixf idummy = i*3*nframe + j*2 idummy = mod(idummy,256) ! assuming color depth is 256 (0--255) rgb(1,i,j) = char(idummy) ! red idummy = i*1*nframe + j*3 idummy = mod(idummy,256) rgb(2,i,j) = char(idummy) ! green idummy = i*5*nframe + j*7 idummy = mod(idummy,256) rgb(3,i,j) = char(idummy) ! blue 100 continue return end

16. CS 363 GMU Spring 200516 Sample Fortran real*4 one,eps, ht, tf real*8 one8, eps8 one = 1. one8 = 1. eps = 1. eps8 = 1. ht = 100000 tf = 24. print *, 'Test for precision of real*4, based on 1+eps=1' 10 eps = eps/2. if (one.ne. one+eps) go to 10 eps = 2.*eps print *,' relative precision is ',eps print * print *,'Test for precision of real*4,', + 'based on 100000+eps=100000' eps = 1. 15 eps = eps/2. if (ht.ne. ht+eps) go to 15 eps = 2.*eps print *,' relative precision is ',eps print * print *,'Test for precision of real*4,', + 'based on 24+eps=24' eps = 1. 17 eps = eps/2. if (tf.ne. tf+eps) go to 17 eps = 2.*eps print *,' relative precision is ',eps print * print *, 'Test for precision of real*8, based on 1+eps=1' 20 eps8 = eps8/2. if (one8.ne. one8+eps8) go to 20 eps8 = 2.*eps8 print *,' relative precision is ',eps8 end

17. CS 363 GMU Spring 200517 LISP - 1959 LISt Processing language AI research needed a language that: –Processed data in lists (rather than arrays) –Allowed symbolic computation (rather than numeric) Only two data types: atoms and lists Syntax is based on lambda calculus No variables or assignment Control via recursion and conditional expressions (A B C D) – apply function A to arguments B C D

18. CS 363 GMU Spring 200518 LISP Pioneered functional programming First interpreters slow COMMON LISP and Scheme are contemporary dialects of LISP ML, Miranda, and Haskell are related languages

19. CS 363 GMU Spring 200519 LISP Sample Problem: remove the first occurrence of atom A from list L (define (remove L A) (cond ( (null? L) '() ) ( (= (car L) A) (cdr L)) ; Match found! (else (cons (car L)(remove (cdr L) A))) ; keep searching )

20. CS 363 GMU Spring 200520 BASIC - 1964 Beginner’s All-purpose Symbolic Instruction Code Designed by Kemeny & Kurtz at Dartmouth Design Goals: –Easy to learn and use for non-science students –“pleasant and friendly” –Fast turnaround for homework –Free and private access –User time is more important than computer time Current popular dialect: Visual BASIC First widely used language with time sharing

21. CS 363 GMU Spring 200521 APL and SNOBOL Characterized by dynamic typing and dynamic storage allocation APL (A Programming Language) 1962 –Designed as a hardware description language (at IBM) –Highly expressive (many operators, for both scalars and arrays of various dimensions) –Programs are very difficult to read SNOBOL(1964) –Designed as a string manipulation language (at Bell Labs) –Powerful operators for string pattern matching

22. CS 363 GMU Spring 200522 Snobol Example * Find biggest words and numbers in a test string * BIGP = (*P $ TRY *GT(SIZE(TRY,SIZE(BIG))) $ BIG FAIL STR = 'IN 1964 NFL ATTENDANCE JUMPED TO 4,807,884; ‘ 'AN INCREASE OF 401,810.' P = SPAN('0123456789,') BIG = STR BIGP OUTPUT = 'LONGEST NUMBER IS ' BIG P = SPAN('ABCDEFGHIJKLMNOPQRSTUVWXYZ') BIG = STR BIGP OUTPUT = 'LONGEST WORD IS ' BIG END

23. CS 363 GMU Spring 200523 ALGOL 68 - 1968 From the continued development of ALGOL 60, but it is not a superset of that language Features: –User-defined data structures –Reference types –Dynamic arrays (called flex arrays) Contribution: –Had even less usage than ALGOL 60 BUT had strong influence on subsequent languages, especially Pascal, C, and Ada

24. CS 363 GMU Spring 200524 Important ALGOL Descendants Pascal - 1971 –Designed by Wirth, who quit the ALGOL 68 committee (didn't like the direction) –Designed for teaching structured programming –Small, simple, nothing really new –From mid-1970s until the late 1990s, it was the most widely used language for teaching programming in colleges

25. CS 363 GMU Spring 200525 Pascal Sample program fibonacci(input, output); type natural 0..maxint; var fnm1,fnm2, fn,n,i: natural begin readln(n); if n <= 1 then writeln(n) {F 0 = 0 and F 1 = 1} else begin {compute F n } fnm2 := 0; fnm1 := 1; for i := 2 to n do begin fn := fnm1 + fnm2; fnm2 := fnm1; fnm1 := fn; end writeln(fn); end end.

26. CS 363 GMU Spring 200526 Important ALGOL Descendants C - 1972 –Designed for systems programming (at Bell Labs by Dennis Richie) –Evolved primarily from B, but also ALGOL 68 –Powerful set of operators, but poor type checking –Initially spread through UNIX

27. CS 363 GMU Spring 200527 C Sample #include main() { int fnm1,fnm2, fn,n,i; scanf(“%d”,&n); if (n <= 1) printf(“%d\n”, n) /* F 0 = 0 and F 1 = 1*/ else { /* compute F n */ fnm2 = 0; fnm1 = 1; for (i = 2; i<=n ;++i) { fn = fnm1 + fnm2; fnm2 = fnm1; fnm1 = fn; } printf(“%d\n”,fn); }

28. CS 363 GMU Spring 200528 Prolog - 1972 Developed at the University of Aix-Marseille, by Comerauer and Roussel, with some help from Kowalski at the University of Edinburgh Applications in AI, DBMS Based on formal logic Non-procedural Can be summarized as being an intelligent database system that uses an inferencing process to infer the truth of given queries Inefficient execution

29. CS 363 GMU Spring 200529 Prolog Sample parent(X,Y) :- mother(X,Y). parent(X,Y) :- father(X,Y). sibling(X,Y) :- mother(M,X), mother(M,Y), father(D,X), father(D,Y),X \== Y. haschildren(X) :- parent(X,_). grandparent(X,Y) :- parent(X,M),parent(M,Y). cousin(X,Y) :- parent(A,X),parent(B,Y),sibling(A,B). mother(anne,mary). mother(anne,liz). mother(anne,susan). mother(anne,virginia). mother(elizabeth,russ). mother(mabel,anne). father(james,zelie). father(james,harry). father(james,ned). father(john,will). father(john,russell). father(jim,rachel). father(jim,maggie). father(tim,patrick). grandparent(anne,Y).

30. CS 363 GMU Spring 200530 Ada - 1983 (began in mid- 1970s) Huge design effort, involving hundreds of people, much money, and about eight years Environment: More than 450 different languages being used for DOD embedded systems (no software reuse and no development tools) Named for Ada Lovelace (1815-1851) – first programmer (worked with Charles Babbage).

31. CS 363 GMU Spring 200531 Ada Sample package ArrayCalc is type Mydata is private; function sum return integer; procedure setval(arg:in integer); private size: constant:= 99; type myarray is array(1..size) of integer; type Mydata is record val: myarray; sz: integer := 0; end record; v: Mydata; end; package body ArrayCalc is function sum return integer is temp: integer; -- Body of function sum begin temp := 0; for i in 1..v.sz loop temp := temp + v.val(i); end loop; v.sz:=0; return temp; end sum; procedure setval(arg:in integer) is begin v.sz:= v.sz+1; v.val(v.sz):=arg; end setval; end; with Text_IO; use Text_IO; with ArrayCalc; use ArrayCalc; procedure main is k, m: integer; begin -- of main get(k); while k>0 loop for j in 1..k loop get(m); put(m,3); setval(m); end loop; new_line; put("SUM ="); put(ArrayCalc.sum,4); new_line; get(k); end loop; end;

32. CS 363 GMU Spring 200532 Smalltalk - 1972-1980 Developed at Xerox PARC, initially by Alan Kay, later by Adele Goldberg First full implementation of an object- oriented language (data abstraction, inheritance, and dynamic type binding)

33. CS 363 GMU Spring 200533 C++ - 1985 Developed at Bell Labs by Stroustrup Evolved from C and SIMULA 67 Facilities for object-oriented programming, taken partially from SIMULA 67, were added to C Also has exception handling A large and complex language, in part because it supports both procedural and OO programming Rapidly grew in popularity, along with OOP ANSI standard approved in November, 1997

34. CS 363 GMU Spring 200534 C++ Related Languages Eiffel - a related language that supports OOP –(Designed by Bertrand Meyer - 1992) –Not directly derived from any other language –Smaller and simpler than C++, but still has most of the power Delphi (Borland) –Pascal plus features to support OOP –More elegant and safer than C++

35. CS 363 GMU Spring 200535 Java (1995) Developed at Sun in the early 1990s Based on C++ –Significantly simplified (does not include struct, union, enum, pointer arithmetic, and half of the assignment coercions of C++) –Supports only OOP –Has references, but not pointers –Includes support for applets and a form of concurrency

36. CS 363 GMU Spring 200536 Scripting Languages JavaScript –HTML-embedded at client side and executed by the client (i.e. web browser) –create dynamic HTML documents PHP –HTML enabled server side –Interpreted by server when a document in which it is embedded is requested. –Output of interpretation is HTML that replaces the PHP

37. 37 Supercomputer Applications

38. Topics Parallel computing and High Performance Computing Clusters MPI – Message Passing Interface –Passing messages among processes OpenGL and computer graphics applications

39. Parallel Computing Overview ● a high performance parallel computer is a computer that can solve large problems in a much shorter time than a single desktop computer. ● characterized by fast CPUs, large memory, a high speed interconnect, and high speed input/output. ● able to speed up computations; both by making the sequential components run faster and by doing more operations in parallel.

40. ● High performance parallel computers are in demand because there is a need for tremendous computational capabilities in science, engineering, and business. ● There are applications that require gigabytes of memory and gigaflops of performance. ● Today, application scientists are striving for terascale performance, to permit an even larger class of problems to be solved.

41. ● Terascale refers to computers that perform more than one trillion floating-point operations per second, called "teraflops" ● High performance parallel computers are used in a wide variety of disciplines. – Meteorologists use them for the prediction of tornadoes and thunderstorms – help computational biologists analyze DNA sequences – used by pharmaceutical companies in the design of new drugs,

42. – by oil companies for their seismic exploration, – Wall Street for the analysis of financial markets – NASA uses them for aerospace vehicle design, – the entertainment industry uses them for special effects in movies and commercials ● The common characteristic of all these complex scientific and business applications is the need to perform computations on large datasets or large equations.

43. Parallelism in Computer Programs ● It used to be thought that computer programs were sequential in nature ● algorithm is defined as the "sequence of steps" necessary to carry out a computation. ● In the first 30 years of computer use, programs were run sequentially because of this thinking. ● the 1980's saw great successes with parallel computers. ● Dr. Geoffrey Fox published a book entitled Parallel Computing Works! helped lead to a reversal in thinking.

44. ● Parallel computing is what a computer does when it carries out more than one computation at a time using more than one processor. ● If one processor can perform the arithmetic in time t, then ideally p processors can perform the arithmetic in time t/p. ● The benefit of parallelism is that it allows researchers to do computations on problems they previously were unable to solve.

45. Comparison of Parallel Computers The hardware components of parallel computers: * kinds of processors * types of memory organization * flow of control * interconnection networks We'll see what is common to these parallel computers, and what makes each one of them unique.

46. Processors There are three types of parallel computers: 1. computers with a small number of powerful processors 2. computers with a large number of less powerful processors 3. computers that are medium scale in between the two extremes

47. A Small Number of Powerful Processors They are general-purpose computers that perform especially well on applications that have large vector lengths. The examples of this type of computer are the Cray SV1 and the Fujitsu VPP5000.

48. A Large Number of Less Powerful Processors Computers with a large number of less powerful processors, named a Massively Parallel Processor (MPP), typically have thousands of processors. The processors are usually proprietary and air-cooled. Because of the large number of processors, the distance between the furthest processors can be quite large requiring a sophisticated internal network that allows distant processors to communicate with each other quickly.

49. Medium Scale Computers Medium scale computers typically have hundreds of processors. The processor chips are usually not proprietary; rather they are commodity processors like the Pentium III. These are general-purpose computers that perform well on a wide range of applications. The most common example of this class is the Linux Cluster.

50. Trends and Examples Decade Processor Type Computer Example 1970s Pipelined, Proprietary Cray-1 1980s Massively Parallel, Proprietary Thinking Machines CM2 1990s Superscalar, RISC, Commodity SGI Origin2000 2000s CISC, Commodity Workstation Clusters

55. Background on MPI MPI - Message Passing Interface –Library standard defined by a committee of vendors, implementers, and parallel programmers –Used to create parallel SPMD programs based on message passing 100% portable: one standard, many implementations Available on almost all parallel machines in C and Fortran Over 100 advanced routines but 6 basic

56. Key Concepts of MPI Used to create parallel SPMD programs based on message passing –Normally the same program is running on several different processors –Processors communicate using message passing Typical methodology: start job on n processors do i=1 to j each processor does some calculation pass messages between processor end do end job

57. Summary MPI is used to create parallel programs based on message passing Usually the same program is run on multiple processors The 6 basic calls in MPI are: –MPI_INIT( ierr ) –MPI_COMM_RANK( MPI_COMM_WORLD, myid, ierr ) –MPI_COMM_SIZE( MPI_COMM_WORLD, numprocs, ierr ) –MPI_Send(buffer, count,MPI_INTEGER,destination, tag, MPI_COMM_WORLD, ierr) –MPI_Recv(buffer, count, MPI_INTEGER,source,tag, MPI_COMM_WORLD, status,ierr) –call MPI_FINALIZE(ierr)

58. MPI helloworld.c #include main(int argc, char **argv) { int numtasks, rank; MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, & numtasks); MPI_Comm_rank(MPI_COMM_WORLD, &rank); printf("Hello World from process %d of %d\n“, rank, numtasks); MPI_Finalize(); }

59. MPI_COMM_WORLD MPI_INIT defines a communicator called MPI_COMM_WORLD for every process that calls it. All MPI communication calls require a communicator argument MPI processes can only communicate if they share a communicator. A communicator contains a group which is a list of processes Each process has it’s rank within the communicator A process can have several communicators

60. Sample Run and Output A Run with 3 Processes: –manjra> mpirun -np 3 -machinefile machines.list helloworld  Hello World from process 0 of 3  Hello World from process 1 of 3  Hello World from process 2 of 3  A Run by default –manjra> helloworld  Hello World from process 0 of 1

61. Sample Run and Output A Run with 6 Processes: –manjra> mpirun -np 6 -machinefile machines.list helloworld Hello World from process 0 of 6 Hello World from process 3 of 6 Hello World from process 1 of 6 Hello World from process 5 of 6 Hello World from process 4 of 6 Hello World from process 2 of 6 Note: Process execution need not be in process number order.

62. Image Processing Edge Detection

63. Parallel “Game of Life”..o..............oo......oo.............ooo.ooo....ooo...................oo.....o...........o......oo...o...............

64. Computer Architecture

65. Classification of Computer Architectures 1. Basic SISD Architectures: Traditional Machines 1. Zero Address: Stack Approach 2. One Address: Accumulator Approach 3. Two Address: Register/Memory Address Approach 2. Advanced Architectures: RISC Machines 3. SIMD Architectures: Vector Pipelines and Parallelism 4. MIMD Architectures: Distributed Processing (PVM), Hypercube Architecture

66. Digital Logic Level 1. Logic gates, Boolean Algebra, Circuit Design, Multiplexers 2. Parity, Error correction (Hamming Code), and Data Compression (Huffman Code)

67. Microprogramming and Microachitectures 1. Microprocessors and Instruction formats 2. Opcode design: Expanding opcodes 3. Target Machine verses hardware considerations 4. Registers and word length 5. Example Mic-1 Microprogram Level Interpreter

68. Assembly Language level 1. Assembly Language Instruction Sets, Introduction to SPIM 2. Branching and Loops 3. Addressing Techniques 4. Input and Output 5. Program Tuning