CS 3733 Operating Systems Notes: Files

ThursdayClass TuesdayClass ThursdayClass

Unix uses device independence.

I/O is done through device drivers that have a standard interface.

5 main system calls for I/O:

open
close
read
write
ioctl

Types of files:

regular
directory
character special: like a terminal or tape
block special: like a disk drive
FIFO: also called a named pipe
symbolic link: a file that points to another file
socket: for network communication
others

Directories

Figure 3.2 (page 86): Structure of a typical UNIX filesystem

Current Working Directory

  #include <unistd.h>

  char *getcwd(char *buf, size_t size);

Example 3.2 (page 80) outputs the pathname of the current working directory. It uses PATH_MAX to determine the size of the buffer needed.

Example 3.3 (page 81) determines the implementation's maximum pathname length relative to the root directory using pathconf.

Reading Directories

  #include <sys/types.hi>
  #include <dirent.h>

  DIR *opendir(const char *filename);
  struct dirent *readdir(DIR *dirp);
  void rewinddir(DIR *dirp);
  int closedir(DIR *dirp);

opendir provides a handle for the other functions.
readdir gets the next entry in the directory.
rewinddir restarts from the beginning.
closedir closes the handle.
Note that like strtok these are not reentrant.

Program 3.1 (page 83) lists the files in a directory.

Inodes
A directory entry contains only a name and an index into a table giving information about a file. The table and the index are both referred to as an inode.

This is what an inode looks like:

Figure 3.3 (page 87): Inodes are used to represent information about a UNIX file.

Exercise 3.3, page 87:
Suppose that the block size is 8K and pointers are 4 bytes. How large a file can be represented using single indirect pointers, double indirect pointers, and triple indirect pointers?
A block contains 2K pointers. A single indirect block can address 2K x 8K = 16 megabytes.
A double indirect block will contain 2K pointers to single indirect blocks for a total of 2K x 16 megabytes, or 32 gigabytes.
A triple indirect block can reference 2K times this, or 64 terabytes.

Inodes are read using the system call stat

  #include <sys/types.h>
  #include <sys/stat.h>

  int stat(const char *path, struct stat *buf);
  int fstat(int fildes, struct stat *buf);
  int lstat(const char *path, struct stat *buf);

stat is given the name of a file.
fstat is used for open files.
lstat does the same thing as stat except that if the file is a symbolic link, it gives information about the link, rather than the file it is linked to.
The contents of the struct stat are system dependent, but most systems have similar fields. Below are some of the fields for the Solaris 7:

  dev_t           st_dev;   /* ID of device containing directory entry */
  ino_t           st_ino;   /* Inode number */
  mode_t          st_mode;  /* File mode (type and permissions) */
  nlink_t         st_nlink; /* Number of links */
  uid_t           st_uid;   /* User ID of the file's owner */
  gid_t           st_gid;   /* Group ID of the file's group */
  dev_t           st_rdev;  /* ID for special files */
  off_t           st_size;  /* File size in bytes */
  timestruc_t     st_atim;  /* Time of last access */
  timestruc_t     st_mtim;  /* Time of last data modification */
  timestruc_t     st_ctim;  /* Time of last file status change */
  blksize_t       st_blksize;  /* Preferred I/O block size */
  blkcnt_t        st_blocks;   /* Number st_blksize blocks allocated */

The timestruc_t is the same as the struct timespec structure which contains two fields:

   time_t  tv_sec;   /* seconds */
   long    tv_nsec;  /* nanoseconds */

Example 3.6 (page 89) takes a directory name passed on the command line and lists the contents of that directory along with the time it was last modified.
You will notice that st_mtime is not a field listed above. st_mtime is the legacy statbuf field that contained the modification time in seconds since January 1, 1970, UTC. It is defined in sys/stat.h with:
#define st_mtime st_mtim.tv_sec

Links

A link is an association between a filename and an inode.
UNIX has two types of links: hard and symbolic (also called soft).
Directory entries are called hard links because they directly link filenames to inodes.
Each inode contains a count of the number of hard links to the inode.
When a file is created, a new directory entry is created an a new inode is assigned.
Additional hard links can be created with
ln newname oldname
or with the link system call.
A new hard link to an existing file creates a new directory entry but assigns no other additional disk space.
A new hard link increments the link count in the inode.
A hard link can be removed with the rm command or the unlink system call.
These decrement the link count.
The inode and associated disk space are freed when the count is decremented to 0.

Figure 3.4 (page 91) shows a simple hard link and the associated inode.
Figure 3.5 (page 92) shows two hard links to the same file.


Figure 3.4 (page 91): A directory entry, inode, and data block for a simple file.	Figure 3.5 (page 92): Two hard links to the same file as in Figure 3.4.

Exercise 3.4, page 91, modified
What would happen to Figure 3.5 after the following sequence of operations:
open("/dirA/name1");
read
close
modify memory image of the file
unlink("/dirA/name1");
open("/dirA/name1");
write
close
Answer: The result is Figure 3.6 (page 93).


Figure 3.5 (page 92): Two hard links to the same file as in Figure 3.4.	Figure 3.6 (page 93): The situation after editing the file. The original file had inode 12345 and two hard links prior to being edited.

Some editors make a backup copy of the file being edited. One way to do this is with the following:
open("/dirA/name1");
read
close
modify memory image file file
rename("/dirA/name1","dirA/name1.bak");
open("/dirA/name1");
write
close
The result is in Figure 3.7 (page 93).


Figure 3.5 (page 92): Two hard links to the same file as in Figure 3.4.	Figure 3.7 (page 93): The situation after editing the file with an editor that makes a backup copy.

How to do it right:
open("/dirA/name1");
read
close
modify memory image file file
open("/dirA/name1",O_WRONLY|O_TRUNC);
write
close

Symbolic Links

A symbolic link is a special type of file that contains the name of another file.
A reference to the name of a symbolic link causes the operating system to use the name stored in the file, rather than the name itself.
Symbolic lines are created with the command:
ln -s newname oldname
Symbolic links do not affect the link count in the inode.
Unlink hard links, symbolic links can span filesystems.

Figure 3.8 (page 94) shows a file with a symbolic link.

Figure 3.8 (page 94): An ordinary file with a symbolic link to it.

Figure 3.9 (page 95) shows the result of making an editing change in the file /dirB/name2 when the editor changes the inode associated with the file read in.

Figure 3.9 (page 95): The situation after editing a file that has a symbolic link.

File Handles
Two methods of doing I/O in UNIX:
File descriptors: this is part of the UNIX specification, POSIX, etc.
File pointers: this is from the C standard

File Descriptors
Use the POSIX names STDIN_FILENO, STDOUT_FILENO and STDERR_FILENO.

open, close, read, write,ioctl

  #include <sys/types.h>
  #include <sys/stat.h>
  #include <fcntl.h>

  int open(const char *path, int  oflag, ...);

Common oflag values: O_RDONLY O_WRONLY O_RDWR O_CREAT
If O_CREAT is used, must include third argument for permissions.

Figure 3.10 (page 98): The layout of the permissions mask.

`S_IRUSR`	read permission bit for owner
`S_IWUSR`	write permission bit for owner
`S_IXUSR`	execute permission bit for owner
`S_IRWXU`	read, write, execute for owner

`S_IRGRP`	read permission bit for group
`S_IWGRP`	write permission bit for group
`S_IXGRP`	execute permission bit for group
`S_IRWXG`	read, write, execute for group

`S_IROTH`	read permission bit for others
`S_IWOTH`	write permission bit for others
`S_IXOTH`	execute permission bit for others
`S_IRWXO`	read, write, execute for others

`S_ISUID`	set user ID on execution
`S_ISGID`	set group ID on execution

Table 3.1, Page 99: POSIX.1 symbolic names for file permission bits.

Example 3.10 (page 98) show simple code to create a new file.

The File Descriptor Table is part of the user program area and can be thought of as an array of pointers indexed by the file descriptors.
The pointers point to entries in the System File Table.
The System File Table is in the kernel area.
It contains an entry for each open file.
Entries contain pointers to a table of inodes kept in memory.
Entries contain other information including the current file offset and a count of the number of file descriptors that are using this entry.
When a file is closed the count is decremented. The entry is freed when the count becomes 0.
The In-Memory Inode Table contains copies of the inodes that are being used.

Figure 3.11 (page 100): Relationship between the file descriptor table, the system file table and the in-memory inode table.

File Pointers and Buffering
Use fopen, fclose, fread, fwrite, fprintf, fscanf, etc.
Example 3.11 (page 101) shows how to open a file file output using file pointers.

Figure 3.12 (page 102): Schematic use of a file pointer after fopen.

I/O using file pointers will read from or write to the buffer.
The buffer will be filled or emptied when necessary.
A write may fill part of the buffer without causing any physical I/O to the file.
The amount of buffer may vary.
If a write is done to standard output and them the program crashes, the data written may not show up on the screen.
Standard error is not buffered.
Interleaving output to standard output and standard error may cause output to appear in an unpredictable order.
You can force the physical output to occur with an fflush.

Inheritance of File Descriptors
When fork creates a child, the child inherits a copy of the parents address space, including the file descriptor table.

Example 3.14 (pages 103-104) shows a code segment in which a child inherits a file descriptor of an open file.
Figure 3.13 (page 105) shows the file descriptor tables and system file table corresponding to this code.

Figure 3.13 (page 105): If the parent opens my.dat before the fork, both parent and child share the system file table entry.

Example 3.15 (page 104) shows a similar code segment in which the fork is done before the open.
Figure 3.14 (page 106) shows the relationship between the file descriptor tables and the system file table for this example.

Figure 3.14 (page 106): If the parent and child open my.dat after the fork, the two processes use different system file table entries.

Redirection
Example 3.16 (page 107): Consider the following command:
cat > my.file
Figure 3.15 (page 107) show the file descriptor table for this.

Figure 3.15 (page 107): The status of the file descriptor table before and after redirection for the process that is executing cat > my.file

Redirection can be done by copying one entry of the file descriptor table into another.
This is accomplished with the dup2 system call.

#include <udistd.h>

int dup2(int fildes, int fildes2);

It closes fildes2 and then copies the pointer of entry fildes into the entry fildes2.
Example 3.17 (page 108) shows how this can be done.

Pipes
From the command line:
ls -l | sort -nr +4 | head
produces a list of files sorted by size with only the largest ones displayed.
This is done with the pipe system call:

#include <unistd.h>

int pipe(int fildes[2]);

This creates a communication buffer that the caller can access through file descriptors fildes[0] and fildes[1].
The data written to fildes[1] is read from fildes[0] on a first-in-first-out bases.
The pipe has no external or permanent name and can only be accessed through its two file descriptors.
The pipe can only be used by the process that created it and its descendents.

Example 3.20 (pages 110-111) shows how this can be used to implement
ls -l | sort -nr +4 | head

Figures 3.18 to 3.20 (pages 111-112) show the state of the file descriptor table for Example 3.20.

Figure 3.18 (page 111): The status of the file descriptor table after the fork has been executed in Example 3.20.

Figure 3.19 (page 112): The status of the file descriptor table after the pup2's have been executed in both processes of Example 3.20.

Figure 3.20 (page 112): The status of the file descriptor table just before the execl's have been executed in both processes of Example 3.20.

read and write

#include <unistd.h>

ssize_t read(int fildes, void *buf, size_t nbyte);
ssize_t write(int fildes, const void *buf, size_t nbyte);

The type size_t is an unsigned integer type and ssize_t is a signed integer type.
The read requests that nbyte bytes be read from the file with descriptor fildes and placed in buf.
- The caller must provide a buffer that is large enough to hold nbyte bytes of data.
- On success, read returns the number of bytes actually read, which may be less the nbyte.
- On failure, read returns -1 and sets errno.
The write attempts to output nbyte bytes from buf to the file descriptor given by fildes.
- On success, write returns the number of bytes actually written, which may be less then nbyte.
- On failure, write returns -1 and sets errno.

Example 3.21 (page 113) shows simple code for reading form from_fd and writing to to_fd

This code is not correct since it is not guaranteed that the write will actually write all of the bytes that it is requested to write.

Program 3.2 (pages 114-115) show how to do this correctly.
The program takes source and destination as command line parameters.
An error occurs if the destination file already exists.
A loop for the write handles the case of partial writes.

Non-blocking I/O Various control flags can be changed using fcntl:

#include <sys/types.h>
#include <unistd.h>
#include <fcntl.h>

int fcntl(int fildes, int cmd, /* arg */ ...);

Example 3.22 (page 116) is a program segment that sets an open file descriptor to non-blocking.
Example 3.23 (page 117) sets it to blocking again.

You can use non-blocking I/O to monitor 2 file descriptors, but this solution uses busy waiting.

select

  #include <sys/time.h>
  #include <sys/types.h>
 
  int select(int nfds, fd_set *readfds, fd_set *writefds,
             fd_set *exceptfds, struct timeval *timeout);

  void FD_SET(int fd, fd_set *fdset);
  void FD_CLR(int fd, fd_set *fdset);
  int FD_ISSET(int fd, fd_set *fdset);
  void FD_ZERO(fd_set *fdset);

Example 3.24 (pages 118-119) shows how to use this to monitor two file descriptors.

FIFOs
FIFOs, or named pipes allow you to use pipes between processes that are not related.
You can create a named pipe with the mkfifo command or the mkfifo system call.

#include <sys/types.h>
#include <sys/stat.h>

int mkfifo(const char *path, mode_t mode);

The programs readfifo and writefifo illustrate how these can be used.

readfifo takes one command line parameter, the name of a fifo.
- It creates it if it does not already exist.
- It reads from the fifo until an end of file occurs.
- It outputs what it has read to standard output.
- It blocks on the open for read until another process opens it for write.
writefifo also takes one command line parameter, the name of a fifo.
- It creates the fifo if it does not already exist.
- It writes to the fifo whatever it read from standard input.
- It exists when it receives and end-of-file on standard input.
- It blocks on the open for write until another process opens it for read.

Get readfifo.c or writefifo.c.

Note: There is something wrong with these programs. What is it?