CS 105 “Tour of the Black Holes of Computing”

Slides:



Advertisements
Similar presentations
© Original by Donald Acton; Changes by George Tsiknis Unix I/O Related Text Sections: 2nd Ed: and st Ed: and
Advertisements

1. I/O. I/O Unix I/O RIO (robust I/O) package Metadata, sharing, and redirection Standard I/O Conclusions and examples.
Carnegie Mellon 1 System-Level I/O : Introduction to Computer Systems 14 th Lecture, Oct. 12, 2010 Instructors: Randy Bryant and Dave O’Hallaron.
System-Level I/O November 4, 2003 Topics Unix I/O Robust reading and writing Reading file metadata Sharing files I/O redirection Standard I/O “The.
System-Level I/O Topics Unix I/O Robust reading and writing Reading file metadata Sharing files I/O redirection Standard I/O.
System-Level I/O October 8, 2008 Topics Unix I/O Robust reading and writing Reading file metadata Sharing files I/O redirection Standard I/O class14.ppt.
System-Level I/O May 28, 2008 Topics Unix I/O Robust reading and writing Reading file metadata Sharing files I/O redirection Standard I/O.
System-Level I/O November 11, 2004 Topics Unix I/O Robust reading and writing Reading file metadata Sharing files I/O redirection Standard I/O “The.
System-Level I/O Nov 14, 2002 Topics Unix I/O Robust reading and writing Reading file metadata Sharing files I/O redirection Standard I/O class24.ppt
System-Level I/O Topics Unix I/O Robust reading and writing Reading file metadata Sharing files I/O redirection Standard I/O.
I/O April 16, 2002 Topics Files Unix I/O Standard I/O Reading: (Beta) or (New) Problems: 12.9 (Beta) or 12.4 (New) class24.ppt
System-Level I/O May 22, 2006 Topics Unix I/O Robust reading and writing Reading file metadata Sharing files I/O redirection Standard I/O.
Recitation 11: I/O Problems Andrew Faulring Section A 18 November 2002.
Introduction to Operating Systems File I/O Most of the following slides are adapted from slides of Gregory Kesden and Markus Püschel of Carnegie Mellon.
System-level I/O Alan L. Cox Some slides adapted from CMU slides.
Chapter 39 Virtsualization of Storage: File and Directory Chien-Chung Shen CIS, UD
Input and Output Topics I/O hardware Unix file abstraction Robust I/O File sharing io.ppt CS 105 “Tour of the Black Holes of Computing”
System-Level I/O Topics Unix I/O Robust reading and writing
Carnegie Mellon 1 System-Level I/O / : Introduction to Computer Systems 15 th Lecture, Oct. 18, 2012 Instructors: Dave O’Hallaron, Greg Ganger,
1 System-Level I/O Andrew Case Slides adapted from Jinyang Li, Randy Bryant and Dave O’Hallaron.
Week 12 - Friday.  What did we talk about last time?  Exam 2 post mortem  Low level file I/O.
Shell (Part 2). Example r What if we want to support something like this: m ps –le | sort r One process should execute ps –le and another should execute.
Recitation 11: 11/18/02 Outline Robust I/O Chapter 11 Practice Problems Annie Luo Office Hours: Thursday 6:00 – 7:00 Wean.
Carnegie Mellon 1 System-Level I/O / : Introduction to Computer Systems 15 th Lecture, Oct. 16, 2014 Instructors: Greg Ganger, Greg Kesden,
Carnegie Mellon 1 System-Level I/O / : Introduction to Computer Systems 15 th Lecture, June 25, 2013 Instructors: Gregory Kesden.
System-Level I/O March 25, 2004 Topics Unix I/O Robust reading and writing Reading file metadata Sharing files I/O redirection Standard I/O “The.
Lee CSCE 312 TAMU 1 System Level I/O Instructor: Dr. Hyunyoung Lee Based on slides provided by Randy Bryant and Dave O’Hallaron.
Input and Output Topics I/O hardware Unix file abstraction Robust I/O File sharing CS 105 “Tour of the Black Holes of Computing”
System-Level I/O.
Input and Output Topics I/O hardware Unix file abstraction Robust I/O File sharing CS 105 “Tour of the Black Holes of Computing”
Laface 2007 File system 2.1 Operating System Design Filesystem system calls buffer allocation algorithms getblk brelse bread breada bwrite iget iput bmap.
CS 105 “Tour of the Black Holes of Computing”
CSE333 SECTION 3. Important Dates Jan 27 th – Homework 1 Due Feb 6 th – Midterm.
1 System-Level I/O. 2 Outline Unix I/O Reading File Metadata Sharing Files & I/O redirection Robust I/O Standard I/O Suggested Reading –10.1~10.3, 10.5~10.7,
CSCI 330 UNIX and Network Programming Unit VIII: I/O Management II.
Carnegie Mellon 1 Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition System-Level I/O : Introduction to Computer.
Carnegie Mellon /18-243: Introduction to Computer Systems Instructors: Bill Nace and Gregory Kesden (c) All Rights Reserved. All work.
Files Oct. 28, 2008 Topics Mapping file offsets to disk blocks File system buffering and you The directory hierarchy lecture-18.ppt “The course.
System-Level I/O Topics Unix I/O Robust reading and writing Reading file metadata Sharing files I/O redirection Standard I/O.
Carnegie Mellon 1 System-Level I/O / : Introduction to Computer Systems 15 th Lecture, June 24, 2014 Instructors: Gregory Kesden.
Week 12 - Thursday CS222.
Module 12: I/O Systems I/O hardware Application I/O Interface
Robust I/O package Chapter 11 practice problems
System-Level I/O.
Supplementary Material on Unix System Calls and Standard I/O
System-Level I/O : Introduction to Computer Systems
Introduction to Operating Systems File I/O
Files CMSC 15400: Introduction to Computer Systems Chapter 10
CS 105 “Tour of the Black Holes of Computing”
System-Level I/O Topics Unix I/O Robust reading and writing
CSCI 315 Operating Systems Design
System-Level I/O October 9, 2008
Unix Directories unix etc home pro dev motd snt unix ... slide1 slide2
Operating System Concepts
“The course that gives CMU its Zip!”
“The course that gives CMU its Zip!”
Virtual Memory CSCI 380: Operating Systems Lecture #7 -- Review and Lab Suggestions William Killian.
File I/O (1) Prof. Ikjun Yeom TA – Mugyo
Reference: Advanced Programming in the UNIX Environment, Third Edition, W. Richard Stevens and Stephen A. Rago, Addison-Wesley Professional Computing Series,
File I/O (1) Prof. Ikjun Yeom TA – Hoyoun
IPC Prof. Ikjun Yeom TA – Hoyoun
System-Level I/O.
Chapter 13: I/O Systems I/O Hardware Application I/O Interface
System-Level I/O CSCI 380: Operating Systems
CSCI 380: Operating Systems William Killian
Chien-Chung Shen CIS, UD
Instructors: David Trammell
Module 12: I/O Systems I/O hardwared Application I/O Interface
Lecture 26: File Systems CS April 29, 2019.
Presentation transcript:

CS 105 “Tour of the Black Holes of Computing” Input and Output Topics I/O hardware Unix file abstraction Robust I/O File sharing io.ppt

I/O: A Typical Hardware System CPU chip register file ALU system bus memory bus main memory bus interface I/O bridge I/O bus Expansion slots for other devices such as network adapters. USB controller graphics adapter disk controller mouse keyboard monitor disk

Abstracting I/O Low level requires complex device commands Vary from device to device Device models can be very different Tape: read or write sequentially, or rewind Disk: “random” access at block level Terminal: sequential, no rewind, must echo and allow editing Video: write-only, with 2-dimensional structure Operating system should hide these differences “Read” and “write” should work regardless of device Sometimes impossible to generalize (e.g., video) Still need access to full power of hardware

Unix Files A Unix file is a sequence of m bytes: B0, B1, .... , Bk , .... , Bm-1 All I/O devices are represented as files: /dev/sda2 (/usr disk partition) /dev/tty2 (terminal) Even the kernel is represented as a file: /dev/kmem (kernel memory image) /proc (kernel data structures)

Unix File Types Regular file: binary or text. Unix does not know the difference! Directory file: contains the names and locations of other files Character special file: keyboard and network, for example Block special file: like disks FIFO (named pipe): used for interprocess comunication Socket: used for network communication between processes

Unix I/O The elegant mapping of files to devices allows kernel to export simple interface called Unix I/O Key Unix idea: All input and output is handled in a consistent and uniform way Basic Unix I/O operations (system calls): Opening and closing files: open()and close() Changing the current file position (seek): llseek (not discussed) Reading and writing a file: read() and write()

Opening Files int fd; /* file descriptor */ if ((fd = open(“/etc/hosts”, O_RDONLY)) == -1) { fprintf(stderr, “Couldn’t open /etc/hosts: %s”, strerror(errno)); exit(1); } Opening a file informs kernel that you are getting ready to access that file Returns a small identifying integer file descriptor fd == -1 indicates that an error occurred; errno has reason strerror converts to English (Note: use strerror_r for thread safety) Each process created by a Unix shell begins life with three open files (normally connected to terminal): 0: standard input 1: standard output 2: standard error

Closing Files Closing a file tells kernel that you’re finished with it int fd; /* file descriptor */ int retval; /* return value */ if ((retval = close(fd)) == -1) { perror(“close”); exit(1); } Closing a file tells kernel that you’re finished with it Closing an already closed file is recipe for disaster in threaded programs (more on this later) Some error reports are delayed until close! Moral: Always check return codes, even for seemingly benign functions such as close() perror is simplified strerror/fprintf; see man page

Reading Files char buf[4096]; int fd; /* file descriptor */ unsigned int nbytes; /* number of bytes read */ /* Open file fd ... */ /* Then read up to 4096 bytes from file fd */ if ((nbytes = read(fd, buf, sizeof(buf))) == -1) { perror(“read”); exit(1); } Reading a file copies bytes from current file position to memory, then updates file position Returns number of bytes read from file fd into buf nbytes == -1 indicates error occurred; 0 indicates end of file (EOF) short counts (nbytes < sizeof(buf) ) are possible and are not errors!

Writing Files char buf[4096]; int fd; /* file descriptor */ unsigned int nbytes; /* number of bytes read */ /* Open the file fd ... */ /* Then write up to 4096 bytes from buf to file fd */ if ((nbytes = write(fd, buf, sizeof(buf)) == -1) { perror(“write”); exit(1); } Writing a file copies bytes from memory to current file position, then updates current file position. Returns number of bytes written from buf to file fd nbytes == -1 indicates that an error occurred As with reads, short counts are possible and are not errors! Here, transfers up to 4096 bytes from address buf to file fd

Simple Example #include "csapp.h" int main(void) { char c; while(Read(STDIN_FILENO, &c, 1) != 0) Write(STDOUT_FILENO, &c, 1); exit(0); } Copying standard input to standard output one byte at a time Note the use of error-handling wrappers for read and write (Appendix B in text)

Dealing with Short Counts Short counts can occur in these situations: Encountering (end-of-file) EOF on reads Reading text lines from a terminal Reading and writing network sockets or Unix pipes Short counts never occur in these situations: Reading from disk files, except for EOF Writing to disk files How should you deal with short counts in your code? Use the RIO (Robust I/O) package from your textbook’s csapp.c file (Appendix B) (But note that it handles EOF wrong on terminal) Use C stdio or C++ streams Write your code very, very carefully Ignore the problem and accept that your code is fragile

“Foolproof” I/O Low-level I/O is difficult because of short counts and other possible errors Textbook provides RIO package, a (fairly) good example of how to encapsulate low-level I/O RIO is set of wrappers that provide efficient and robust I/O in applications (e.g., network programs) that are subject to short counts. Download from csapp.cs.cmu.edu/public/ics/code/src/csapp.c csapp.cs.cmu.edu/public/ics/code/include/csapp.h

Implementation of rio_readn /* * rio_readn - robustly read n bytes (unbuffered) */ ssize_t rio_readn(int fd, void *usrbuf, size_t n) { size_t nleft = n; ssize_t nread; char *bufp = usrbuf; while (nleft > 0) { if ((nread = read(fd, bufp, nleft)) == -1) { if (errno == EINTR) /* interrupted by signal handler return */ nread = 0; /* and call read() again */ else return -1; /* errno set by read() */ } else if (nread == 0) break; /* EOF */ nleft -= nread; bufp += nread; return (n - nleft); /* return >= 0 */

Unbuffered I/O RIO provides buffered and unbuffered routines Especially useful for transferring data on network sockets Same interface as Unix read and write rio_readn returns short count only it encounters EOF Usually incorrect if reading from terminal rio_writen never returns a short count Calls to rio_readn and rio_writen can be interleaved arbitrarily on the same descriptor

Buffered Input Buffered: Efficiently read text lines and binary data from file partially cached in an internal memory buffer rio_readlineb reads text line of up to maxlen bytes from file fd and stores it in usrbuf. Especially useful for reading lines from network sockets rio_readnb reads up to n bytes from file fd Calls to rio_readlineb and rio_readnb can be interleaved arbitrarily on same descriptor Warning: Don’t interleave with calls to rio_readn with calls to *b versions

Buffered Example Copying the lines of a text file from standard input to standard output #include "csapp.h" int main(int argc, char **argv) { int n; rio_t rio; char buf[MAXLINE]; Rio_readinitb(&rio, STDIN_FILENO); while((n = Rio_readlineb(&rio, buf, MAXLINE)) != 0) Rio_writen(STDOUT_FILENO, buf, n); exit(0); }

I/O Choices Unix I/O Standard I/O RIO C++ streams Roll your own Most general and basic; others are implemented using it Unbuffered; efficient input requires buffering Tricky and error-prone; short counts, for example Standard I/O Buffered; tricky to use on network sockets Potential interactions with other I/O on streams and sockets Not all info is available (see later slide on metadata) RIO C++ streams Roll your own

I/O Choices, continued Unix I/O Standard I/O RIO C++ streams Buffered and unbuffered Nicely packaged Author’s choice for sockets and pipes But buffered version has problems dealing with EOF on terminals Non-standard, but built on Stevens’s work C++ streams Standard (sort of) Very complex Roll your own Time consuming Error-prone Unix Bible: W. Richard Stevens, Advanced Programming in the Unix Environment, Addison Wesley, 1993.

How the Unix Kernel Represents Open Files Two descriptors referencing two distinct open files Descriptor 1 (stdout) points to terminal, and descriptor 4 points to open disk file Descriptor table [one table per process] Open file table [shared by all processes] v-node table [shared by all processes] File A (terminal) stdin fd 0 File access stdout fd 1 Info in stat struct File pos File size stderr fd 2 refcnt=1 File type fd 3 fd 4 ... ... File B (disk) File access File size File pos File type refcnt=1 ... ...

File Sharing Two distinct descriptors sharing the same disk file through two distinct open file table entries E.g., Calling open twice with the same filename argument Descriptor table (one table per process) Open file table (shared by all processes) v-node table (shared by all processes) File A fd 0 File access fd 1 File pos File size fd 2 refcnt=1 File type fd 3 fd 4 ... ... File B File pos refcnt=1 ...

How Processes Share Files A child process inherits its parent’s open files. Here is the situation immediately after a fork Descriptor tables Open file table (shared by all processes) v-node table (shared by all processes) Parent's table File A fd 0 File access fd 1 File pos File size fd 2 refcnt=2 File type fd 3 fd 4 ... ... Child's table File B File access fd 0 File size fd 1 File pos fd 2 File type refcnt=2 fd 3 ... fd 4 ...

I/O Redirection Question: How does a shell implement I/O redirection? unix> ls > foo.txt Answer: By calling the dup2(oldfd, newfd) function Copies (per-process) descriptor table entry oldfd to entry newfd Descriptor table before dup2(4,1) Descriptor table after dup2(4,1) fd 0 fd 0 fd 1 a fd 1 b fd 2 fd 2 fd 3 fd 3 fd 4 b fd 4 b

File Metadata Metadata is data about data, in this case file data. Maintained by kernel, accessed by users with the stat and fstat functions. /* Metadata returned by the stat and fstat functions */ struct stat { dev_t st_dev; /* device */ ino_t st_ino; /* inode */ mode_t st_mode; /* protection and file type */ nlink_t st_nlink; /* number of hard links */ uid_t st_uid; /* user ID of owner */ gid_t st_gid; /* group ID of owner */ dev_t st_rdev; /* device type (if inode device) */ off_t st_size; /* total size, in bytes */ unsigned long st_blksize; /* blocksize for filesystem I/O */ unsigned long st_blocks; /* number of blocks allocated */ time_t st_atime; /* time of last access */ time_t st_mtime; /* time of last modification */ time_t st_ctime; /* time of last change */ };

Summary: Goals of Unix I/O Uniform view User doesn’t see actual devices Devices and files look alike (to extent possible) Uniform drivers across devices ATA disk looks same as IDE, EIDE, SCSI, … Tape looks pretty much like disk Support for many kinds of I/O objects Regular files Directories Pipes and sockets Devices Even processes and kernel data