1. Unix Programming Environment
Part 6-2 – Standard I/O Library, File and Directory
Prepared by Xu Zhenya( )
Draft – Xu Zhenya( 2002/10/01 )

2. Agenda 1. Overview 2. Standard I/O Routines 3. Files and Directories
Chapter 6
3. Files and Directories
Chapter 7( 1, 2, 3 )
Unix Programming Environment, Dept. of CSE, BUAA

3. Overview (1)

4. Overview (2) 1. POSIX I/O: 2. scatter /gather I/O: 3. Nonblocking I/O
ssize_t pread( int fd, void * buf, size_t nbytes, off_t offset );
ssize_t pwrite( int fd, const void * buf, size_t nbytes, off_t offset );
2. scatter /gather I/O:
ssize_t readv / writev ( int fd, const struct iovec *iov, int iovcnt );
struct iovec { caddr_t iov_base; int iov_len; }
3. Nonblocking I/O
O_NONBLOCK & O_NDELAY
4. I/O Multiplexing
select() & poll()
5. Async I/O
Performance: Commercial RDMS, concurrency model
SIGIO
Implementation: user-level, kernel-level
6. Memory-mapping file I/O
void *mmap(void *addr, size_t len, in prot, int flags, int fd, off_t off );
7. 64-bit files: off_t => offset_t, llseek

5. Standard I/O (1) 1. History 2. Streams and FILE objects 3. Buffering
The standard I/O library was rewritten by Dennis Ritchie around Surprisingly, little has changed in the standard I/O library after more than 27 years.
2. Streams and FILE objects
Linux: /usr/include/libio.h – struct _IO_FILE;
On some UNIX systems, < 256 /1024
STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO
3. Buffering
Fully buffered:
files that reside on disks
fflush(): forces a write of all user-space buffered data for the given output or update stream

6. Standard I/O (2) Line buffered:
perform I/O when a new line charater is encountered on input or output.
the stream refers to a terminal.
Unbuffered:
The standard I/O library does not buffer the characters.
stderr
Notes:
Default setting:
Standard error is always unbuffered.
All other streams are line buffered if they refer to a terminal device; otherwise they are fully buffered.
The size of line buffers are 128bytes, and the size of full buffers are 1K bytes.
setbuf(), setvbuf()
make sure that the buffer space still exists…

7. Standard I/O (3) 4. Reading the textbook: 5. Misc. P131: table6-3
FILE * freopen(const char *pathname, const char *type, FILE *fp );
Typically used to open a specified file as one of the predefined streams: standard input, output or error.
FILE * fdopen( int fd, const char * type );
Often used with descriptors that are returned by the functions that create pipes and network communication channels.

8. Standard I/O (4) int fileno( FILE * stream );
ferror(), feof(), clearerr()
Positioning a stream:
ftell(), fseek(), rewind()
fgetpos(), fsetpos()
Formatted output:
vprintf, vfprintf, vsprintf,
va_list; man vprintf; man 3 stdarg
Binary I/O
size_t fread( void *p_buf, size_t size, size_t nobj, FILE *fp );
Read and write a C structure?
Compilers: the binary layout of a structure: alignment, (#pragma pack(1))
CPUs’ architecture: Floating-point values
To exchange binary data : Network protocols

9. Standard I/O (5) 6. Temporary files 7. Cautions for Using Standard I/O
char * tmpnam( char * ptr );
If ptr is NULL, then return a pointer to a static area. => copy
If ptr is not NULL, then lengthof( ptr ) >= L_tmpnam ( <stdio.h> )
char *tempnam(const char *directory, const char *prefix );
to specify both the directory and a prefix of the generated pathname.
FILE * tmpfile( void );
A temporary file( wb+ ) that is automatically removed when it is close or on program termination.
7. Cautions for Using Standard I/O
Don’t mix standard I/O and system-level I/O.
If we write to the file at system-level, but then read at the standard I/O level => we might lose the changes.

10. File and Directory (1) File Descriptor 0 Process:
the file descriptor flags: CLOSE_ON_EXEC
a pointer to a file table entry
The file structure in the kernel:
the file status flags( read, write, append, sync, nonblocking, etc )
the current file offset, reference count and a pointer to the v-node table entry for this file
v-node structure:
On Solaris, we can use “crash” to trace all data structures in the diagram.

11. inode->i_op => ramfs.c::ramfs_get_inode
OO ::Object Model

12. File and Directory (2)

13. File Operations (1) 1. open & create
2. read & write, synchronize the file to disk, change the offset location, change the size of a file
read, write
sync, fsync
lseek, llseek
ftruncate, truncate
3. close, delete
close, unlink, remove

14. File Operations (2) 4. check accessiblility, files’ attributes
stat, fstat, lstat
access
umask, chmod, fchmod, chown, fchown, lchown, utime
5. fcntl, ioctl
6. processes
chdir, fchdir, getcwd, chroot
umask, dup, dup2

15. open and creat O_RDONLY, O_WRONLY, O_RDWR O_APPEND, O_TRUNC
int open( const char *pathname, int flags, mode_t mode );
O_RDONLY, O_WRONLY, O_RDWR
O_APPEND, O_TRUNC
O_CREAT, O_EXCL
O_SYNC,
O_NONBLOCK, O_NDELAY
int creat( const char *pathname, mode_t mode );
open( pathname, O_WRONLY | O_CREAT | O_TRUNC, mode );
Temporary files:
open( pathname, O_RDWR | O_CREAT | O_TRUNC, mode );

16. close, unlink, lseek, truncate
close, unlink, remove
DS in the kernel memory, links of an inode in the file system
When a process terminates, all open files are automatically closed by the kernel.
unlink: links–
If the name was not the last link to a file
If the name was the last link to a file and no process have the file open
If the name was the last link to a file and any process have the file open
remove deletes a name from the file system. It calls unlink for files, and rmdir for directories.
lseek
Symbol const: SEEK_SET(0), SEEK_CUR(1), SEEK_END(2)
Only recoding the current file offset within the kernel => NO I/O operations to take place.
A hole in the file
int truncate( const char *path, off_t length );
int ftruncate( int fd, off_t length );

17. read and write read write I/O Efficiency
The number of bytes actually read is less than the requested amount:
A regular file: the end of file
A terminal device: line
From a network endpoint:
Record-oriented devices like a magnetic tape
write
The return value is usually equal to the nbytes argument, otherwise an error has occurred: exceeding the size limit, etc
I/O Efficiency
Textbook, p148

18. fcntl int fcntl( int fd, int cmd, … /* int arg */ );
Change the properties of a file that is already opened.
Example:
flag = fcntl( fd, F_GETFL, 0 );
fcntl( fd, F_SETFL, flag | O_NONBLOCK );
The fcntl function is used for five different purposes:
Duplicate an existing descriptor
Get/set file descriptor flags
Get/set file status flags
Get/set asynchronous I/O ownership
Get/set record locks

19. ioctl int ioctl( int fd, int request, … );
manipulates the underlying device parameters of special files: device drivers, etc
Terminal I/O => POSIX.1 standard

20. stat, fstat and lstat dev_t st_dev; /* device */
int stat( const char *file_name, struct stat *buf );
int fstat( int filedes, struct stat *buf );
int lstat( const char *file_name, struct stat *buf );
struct stat {
dev_t st_dev; /* device */
ino_t st_ino; /* inode */
mode_t st_mode; /* protection */
nlink_t st_nlink; /* number of hard links */
uid_t st_uid; /* user ID of owner */
gid_t st_gid; /* group ID of owner */
dev_t st_rdev; /* device type (if inode device) */
off_t st_size; /* total size, in bytes */
unsigned long st_blksize; /* blocksize for filesystem I/O */
unsigned long st_blocks; /* number of blocks allocated */
time_t st_atime; /* time of last access */
time_t st_mtime; /* time of last modification */
time_t st_ctime; /* time of last change */ };
Textbook: p156, example: checkmail.c
File type: S_ISREG, S_ISDIR
Permission: S_IRUSR
Unix Programming Environment, Dept. of CSE, BUAA

21. access, umask and chmod int access( const char *pathname, int mode);
checks whether the process would be allowed to read, write or test for existence of the file (or other file system object).
a symbolic link: the file referred to by this symbolic link
R_OK, W_OK, X_OK, F_OK
umask, chmod, fchmod, chown, fchown, lchown, utime

22. File Sharing In one process Among processes int dup( int oldfd );
int dup2( int oldfd, int newfd );
Textbook, p163, system()
I/O redirection, some Servers like WWW
Among processes
O_CREAT & O_EXCL
Atomic Operations:
lseek and write => pread & pwrite
the offset of a open file in the kernel

23. Directory DIR *opendir(const char *name);
1. directory format ( textbook, p153 )
struct dirent {
#ifndef __USE_FILE_OFFSET64
__ino_t d_ino;
__off_t d_off;
#else
__ino64_t d_ino;
__off64_t d_off;
#endif
unsigned short int d_reclen;
unsigned char d_type;
char d_name[256]; /* We must not include limits.h! */ };
2. accessing a directory
DIR *opendir(const char *name);
struct dirent *readdir(DIR *dir);
void rewinddir(DIR *dir);
int closedir(DIR *dir);
exec(): close all directory stream
3. creating and deleting, renaming a directory
mkdir, rmdir, rename

24. Symbol Link
readlink, symlink

25. mmap() – (1)
void * mmap(void *start, size_t length, int prot , int flags, int fd, off_t offset);
1. Map a file or a POSIX shared memory object into the calling process’s address space. The underlying objects may be:
a regular file
a special device(/dev/zero): anonymous mapping
shm_open(): unrelated process ( POSIX )
2. function list:
mmap, munmap, msync, mprotect
3. Linux: shared library, executable binary files, read/write operations.

26. 1. vm_area, vm_area driver = filemap
2. vm_area is linked to the inode, its page too.
3. when page fault during accessing the memory, the fault handler will call the driver, which should invoke the inode’ mmap() to load from the file.

27. mmap() – (2) Notes for mmap():
1. flag == MAP_PRIVATE: don’t change the underlying object; during the first writing, the kernel would dup a private copy. It imply that we can see the change before the first time writing.
2. flag == MAP_SHARED: the underlying object will be changed.
3. the mapped length & the size of the underlying object
PAGESIZE: sysconf( _SC_PAGESIZE );
SIGBUS: out of the length of the underlying object
SIGSEGV: out of the mapped segment
Use truncate & ftruncate frist to resize the underlying object
4. Fork(): if flag == MAP_SHARED, the child process inherits the mapped memory.

28. mmap() – (3) 5. how to use mmap()
Directly access the memory, not using read() & write()
!!!! But the system calls read() & write are atomic operations
Not all FD can be mapped into memory. Like a terminal or socket
Implement the shared memory among unrelated processes
The underlying object provides the initial values for the mapped memory.
Any change will be write back to the underlying object.
6. how to use anonymous mapping?
On SYSV, we can use /dev/zero to map: ZFOD( zero-fill-on-demand)
After loading the program, BSS, heap and stack will use anonymous mapping.
In the applications, a parent and children processes can use anonymous mapping to share memory without creating or opening a real file.
7. truncate & ftruncate to resize the underlying object