Unix Programming Environment Part 2 – An Introduction to Unix Systems Dept. of CSE BUAA

Unix Programming Environment Dept. of CSE, BUAA Agenda  1. Overview  2. A Brief History of Unix Systems  3. Organizations and Standards about Unix  4. Unix Nowadays  Internal structure of modern Unix, system interfaces, and tools  5. Summary - Unix’s legend  6. The mandate for change  7. Looking back, looking forward  8. Supplementary Readings

Unix Programming Environment Dept. of CSE, BUAA 1. Overview (1)  Since its inception in 1969, the Unix system has grown into a versatile operating platform:  small embedded processors  workstations and desktop systems  high-performance multi-processor systems  “The UNIX system” consists of a collection of user programs, libraries, and utilities, running on the UNIX operating system kernel, which provides a run-time environment and system services for these applications.  Baseline releases:  System V Release 4( SVR4)  UC’s Berkeley Software Distribution (4.xBSD )  Linux  CMU’s Mach

Unix Programming Environment Dept. of CSE, BUAA 1. Overview (2) - How UNIX is organized The UNIX system is functionally organized at three levels:  The kernel, which schedules tasks and manages storage;  The shell, which connects and interprets users' commands, calls programs from memory, and executes them;  The tools and applications that offer additional functionality to the operating system  Tools can be added or removed from a UNIX system, depending upon the applications required.

Unix Programming Environment Dept. of CSE, BUAA 2. A Brief History of Unix (1)  2.1 The Beginning  In the late 1960s, BTL & GE & MIT:  Multics: Multiplexed Information and Computing System  One of the principal developers, Ken Thompsons, Game program called “Space Travel”:  PDP-7: lacking a program development environment  Honeywell 635 running the GECOS: cross-assembler  Ken Thompsons & Dennis Ritchie: develop an operating environment for PDP-7  A file system: the ancestor of s5fs  A process subsystem: like “fork”  A simple command interpreter called the “shell”  Brian Kernighan called this system as “Unics”( Uniplexed Information and Computing system) => “UNIX”

Unix Programming Environment Dept. of CSE, BUAA 2. A Brief History of Unix (2)

Unix Programming Environment Dept. of CSE, BUAA 2. A Brief History of Unix (2)  Develop a text-processing system for the patent department at BTL:  Port UNIX to the PDP-11  Several text-processing utilities including “ed”, “runoff”, etc  A new language called “B”:  BCPL, CPL, Algol60  B: an interpretive language  To improve the performance, Ritchie evolved it into “C”: compilable and supporting data types and data structures  1973 – 1974: having a tremendous impact on Unix’s future success  Unix Programmer Manual  Unix was rewritten in C( resulting in v4 )  The first Unix paper on ACM SOSP – “The Unix Time Sharing System”( Supplementary reading )  Distributing Unix to outside including its source code

Unix Programming Environment Dept. of CSE, BUAA 2. A Brief History of Unix (3)  2.2 Proliferation  As a result of antitrust litigation by the Dept. of Justice, AT&T distributed Unix for educational and research purpose royalty- free: one of such licensees was U.C. at Berkeley at  Mel Ferentz: => USENIX Conference  John Lions from Australia: lecture notes =>  John Lions from Australia: lecture notes =>  Many people ported UNIX to various architectures: Interdata 8/32, Interdata 7/31, VM/370, VAX-11( UNIX/32V ), Intel 8086( by SCO & MS ), etc  The spirit of cooperation between its keepers and users was a key factor in the rapid growth and rising popularity of UNIX.  Version 7 Unix, released in 1979, was the first truly portable UNIX system.

Unix Programming Environment Dept. of CSE, BUAA 2. A Brief History of Unix (4)  2.3 Baseline Release 1: BSD  Utilities: ex->vi, a Pascal compiler, C Shell( job control, command history )  1978: VAX-11/780( 32-bit architecture, 4G address space, but only 2M physical memory) => a paging-based virtual memory system for UNIX  The Defense Advanced Research Projects Agency( DARPA ) funded the development of Unix systems at Berkeley:  Integrating TCP/IP network protocol suite => Internet  4.4BSD is the last release from CSRG at Berkeley.  FreeBSD and OpenBSD, basing on 4.4BSD, still evolve now.  Contributions from BSD: VM, TCP/IP, Fast file system( FFS), a reliable signals implementation, and the sockets facility, a log- structured file system, etc

Unix Programming Environment Dept. of CSE, BUAA 2. A Brief History of Unix (5)  2.4 Baseline Release 2: System V  After “baby Bells” were born, AT&T was allowed to enter the computer business in  AT&T marketed Unix aggressively: System III at 1982, System V in 1983, SVR2 in 1985, SVR3 in 1987:  A VM implementation quite different from that of BSD: “regions”(!!!)  System V IPC: shared memory, semaphores and message queues  The STREAMS framework for device drivers and network protocols: XTI  Shared libraries, remote file sharing, etc

Unix Programming Environment Dept. of CSE, BUAA 2. A Brief History of Unix (6)  SVR4, developed jointly by AT&T and Sun, was first released in 1989:  Integrating features from SVR3, 4BSD, SunOS and XENIX  Virtual File System( VFS ) and Virtual Memory( VM ) from Sun  Real-time scheduling classes  Supporting Symmetric Multiple Processors( SMP )  Kernel-level threads  Fine-grained locks

Unix Programming Environment Dept. of CSE, BUAA 2. A Brief History of Unix (7)  2.4 Baseline Release 3: Mach  The Unix kernel is small and simple, yet offered many useful facilities. As more and more features were incorporated into the kernel, the internal elegancy disappeared.  The microkernel approach: export a few simple abstractions, do provide most of the functionality thorough a collection of user- level tasks.  But the performance becomes the critical hurdle for the “microkernel” architecture.

Unix Programming Environment Dept. of CSE, BUAA 3. Organizations and Standards  Organization:  UI & OSF  Unix International( UI ): AT&T and Sun, System V-based systems  Open Software Foundation( OSF ): Motif, OSF/1, DCE  USENIX:  But the growth, and even survival, of UNIX has been jeopardized by Microsoft Windows.  Standards:  System V Interface Description( SVID ) from USL  POSIX from IEEE  XPG from X/Open

Unix Programming Environment Dept. of CSE, BUAA 4. Unix Nowadays (1) Internal Structure of Traditional Unix

Unix Programming Environment Dept. of CSE, BUAA 4. Unix Nowadays (2) Internal Structure of modern Unix

Unix Programming Environment Dept. of CSE, BUAA 4. Unix Nowadays (3)

Unix Programming Environment Dept. of CSE, BUAA 4. Unix Nowadays (4)  Summary  Architecture: Functions, Internal Organization  Dynamic loading

Unix Programming Environment Dept. of CSE, BUAA 4. Unix Nowadays (5) - Standards (1) Unix

Unix Programming Environment Dept. of CSE, BUAA 4. Unix Nowadays (6) - Standards (2)  X/Open Common Applications Environment (CAE) Portability Guide Issue 3 (XPG3) and Issue 4 (XPG4)  SUS( Single UNIX Specification ), SUSv2  XNS4: Networking Services Issue4   Notes:  1. The developers of SVID3( UNIX Systems Laboratories) are no longer in business, and this specification defers to POSIX and X/Open CAE.  2. POSIX Standards: we can get them from the library.

Unix Programming Environment Dept. of CSE, BUAA 4. Unix Nowadays (7) - Standards (3)

Unix Programming Environment Dept. of CSE, BUAA 4. Unix Nowadays (8)  Software Development Toolset for Unix  GNU Development Toolset – gcc, ld, gdb, building system

Unix Programming Environment Dept. of CSE, BUAA 5. Unix’s Legend (1)

Unix Programming Environment Dept. of CSE, BUAA 5. Unix’s Legend (2) Brian Kernighan Dennis Ritchie Ken Thompson Bill Joy Linus Torvalds Unix Creators & Gurus

Unix Programming Environment Dept. of CSE, BUAA What’s GNU? Richard Stallman – the founder and author of many GNU programs Free Software Foundation GPL - GPL - Copyright is called…Copyleft. Linux => GNU/Linux

Unix Programming Environment Dept. of CSE, BUAA 6. The Mandate for Change  1. Functionalities:  Pipe -> System V IPC -> POSIX IPC  Multithread  2. Networking  TCP/IP Protocol Stack  Distributed File System: NFS from Sun, AFS/Coda from CMU  RPC-based Services: NIS from Sun, DCE from OSF  3. Performance  FFS, Log FS  SMP Architecture  4. Computation Paradigm Shifts  A Host + terminals – networked workstations – C/S model – Distributed object/component-based computation

Unix Programming Environment Dept. of CSE, BUAA 7. Looking back, Looking forward( 1 )  1. “Small is beautiful”:  2. Flexibility: the core is a framework  Simplicity, clarity and flexibility  Methodology: combination, interface-oriented  3. what’s wrong with Unix?  Although initially simple, it did not remain the way: Standard I/O  No uniform user interfaces  Ritche said: “Unix is simple and coherent, but it takes a genius to understand and appreciate its simplicity.”  The building-block approach requires creativity and imagination to use effectively.  cat menu | grep shrimp | test -lt $10  Because of too many value-added features, Unix became chaotic. Standards like POSIX are partially successful now.

Unix Programming Environment Dept. of CSE, BUAA 7. Looking back, Looking forward( 2 )  Philosophy Matters and Traditions in Unix  Lessons of Unix can be applied in our programming practice  ‘KISS Principle’: Keep it simple, stupid! / Simplicity, clarity and generality  Write small pieces connected by clean interfaces.  Design programs to communicate easily with other programs.  Robustness is the child of transparency and simplicity.  Design for simplicity; add complexity only where you must.  Design for transparency; spend effort early to save effort later.  In interface design, obey the Rule of Least Surprise.  Programmer time is expensive; conserve it in preference to machine time.  Avoid hand-hacking; write programs to write programs when you can.  Use smart data so program logic can be stupid and robust.  Prototype, then polish. Get it working before you optimize it.

Unix Programming Environment Dept. of CSE, BUAA 7. Looking back, Looking forward( 3 )  Everything that can be a device-independent filter should be.  Data streams should, if at all possible, be textual (so they can be viewed and filtered with standard tools).  Database layouts should if at all possible be textual (human- editable).  Application protocols should if at all possible be textual (human-readable).  Complex front ends (user interfaces) should be cleanly separated from complex back ends.  Whenever possible, prototype in an interpretive language before coding C.  Mixing languages is better than writing everything in one, if using only that one will over-complicate the program.

Unix Programming Environment Dept. of CSE, BUAA 8. Supplementary Readings  The UNIX Time-Sharing System Dennis M. Ritchie and Ken Thompson The UNIX Time-Sharing System The UNIX Time-Sharing System  Early Unix history and evolution Dennis M. Ritchie Early Unix history and evolution Early Unix history and evolution  Themes in Operating Systems Dennis M. Ritchie Dennis M. Ritchie  25th Anniversary of Unix 25th Anniversary of Unix 25th Anniversary of Unix  An introduction to Solaris An introduction to Solaris An introduction to Solaris