Assignment 7: Reading Unix V6 Filesystems

Assignment 7: Reading Unix V6 Filesystems

Unix V6 was an early version of the Unix operating system, which was the predecessor of Linux. Unix V6 was released in 1975 and was the first version of Unix to see widespread usage outside Bell Labs. It was a beautiful piece of software: simple and very small (less than 64 KBytes of code!), yet with many of the most powerful features found in modern Linux systems. For this assignment, we have created images of Unix V6 disks in a format virtually identical to how they would have appeared in 1975. Your task for this assignment is to recreate enough of the V6 file system code to read files from the disk. You will work in C for this assignment, since that was the language used for Unix V6.

Learning Goals

  • Understand the different layers of the Unix V6 filesystem and how they work together to access files.
  • Learn how to navigate a substantial code base, adding new features and functions, and testing their behavior.
  • Appreciate the complexity and trade-offs of implementing a filesystem.
  • See how layering can be used to encapsulate knowledge in a modular fashion.

Getting Started

To get started on this assignment, login to a myth machine and invoke the following command:

git clone /afs/ir/class/archive/cs/cs111/cs111.1236/repos/assign7/$USER assign7

This will create a new folder called assign7 in your current working directory and clone the starter code into that folder. There are quite a few files in the starter code; we'll introduce them as needed in the rest of this writeup.

As usual, the directory contains a Makefile; if you type make, it will compile your code together with two test frameworks, test, and diskimageaccess.

Both of the frameworks operate on disk images. A disk image is a file that mirrors the structure of a Unix V6 disk: the first 512 bytes of the file represent block 0 of the disk, the next 512 bytes represent block 1, and so on. We have created several disk images in the directory samples/disk_images.

The first test framework, test, behaves like test programs in previous assignments. You can invoke ./test with an argument giving the name of a test to run (invoke it with no arguments to see a list of available tests). There is also a program samples/test_soln, which is the same as test except that it contains our solution code for the assignment. You can compare the output from these two programs to make sure that they are the same.

The second test framework, diskimageaccess, uses your filesystem code to print information about all of the files in a disk image. To use diskimageaccess, type

./diskimageaccess <option> samples/disk_images/<image>

The <image> argument selects a disk image to read, such as basic.img, and <option> indicates which test to perform. We will use two options to test your submission:

  • -i: tests the inode and file layers. This test will read all of the inodes in order of i-number and print out the contents of the inode, along with a SHA-1 checksum that uniquely summarizes the contents of each file.
  • -p: tests the directory and pathname layers. This test will walk the entire file system directory tree, starting at the root directory, and print out all of the paths that it finds, along with inode and checksum information for each file or directory found.

diskimageaccess provides other options besides these; run ./diskimageaccess -h to see a full list. There is also a program samples/diskimageaccess_soln that contains our solution code, so you can compare its output to that of diskimageaccess.

If you invoke tools/sanitycheck, it will run a series of tests on your solution using both test and diskimageaccess. Try this now: the starter code should compile but almost all the tests will fail.

Structure of the Code

One of the best ways to manage complexity in a large software system is to divide the system into layers. Each layer solves a particular problem for the overall system and provides a simple API for higher layers to use. Higher layers can use the functions of lower layers without needing to understand the details of how they are implemented.

The Unix V6 file system code is structured in five layers; you will write code to implement four of them. For each layer there is a header file that defines the interface to the layer and a corresponding .c file that implements the interface. Working from bottom to top, the layers are:

Block layer (diskimg.h and diskimg.c)
The lowest layer of the file system manages the details of communication with the disk. It encapsulates knowledge about how to read and write individual sectors, so no functions above this layer have to know about the nitty-gritty of communicating with disk hardware.

Inode layer (inode.h, inode.c)
This layer encapsulates information about how to locate inodes on disk and read them into memory; it uses the block layer for disk I/O. It also knows how to traverse the multi-level index structure of an inode to translate logical block numbers to physical disk sectors.

File layer (file.h and file.c)
This layer is responsible for reading data from files into memory. It is a relatively thin layer: it uses the inode layer to determine which block(s) to read, then it uses the block layer to read the blocks.

Directory layer (directory.h and directory.c) This layer encapsulates knowledge about the structure of directories. It provides a function that will lookup a file name in a directory and return the inumber of the file, if the name exists in that directory.

Pathname layer (pathname.h, pathname.c) The highest layer provides the ability to lookup hierarchical file paths starting from the root directory. It knows how to parse paths such as /usr/class/cs111/hello.txt into elements such as usr and class, and it uses the directory layer to look up each element in the appropriate directory.

Read through the header files for each of the layers now to get a more precise feel for what each of the functions does. The block layer has already been fully implemented for you in diskimg.c; it reads and writes blocks from a disk image file rather than an actual disk, but the API is the same as it would be for a real disk. You will implement the other four layers.

The starter code contains four additional header files describing Unix V6 structures:

  • ino.h defines the internal structure of an inode.
  • direntv6.h defines the structure of a directory entry.
  • filesys.h defines the layout of the file system superblock and provides several useful constants.
  • unixfilesystem.h describes the overall layout of a disk, such as where inodes are stored. The first three of these files are slightly modified copies of the original header files from Unix V6.

Error Handling

Real systems (and especially operating systems) must deal with a variety of error conditions. Most of the filesystem functions can return errors. For example, diskimg_readsector can return an error (-1 return value) if there is a problem reading the disk, and directory_findname will return an error if it cannot find the given name in the directory. You must check for errors in two ways:

  • First, you must detect appropriate error conditions in all of the code you write and return -1 when errors occur (e.g., your implementation of directory_findname must return -1 if the given name cannot be found).
  • Second, you must check for errors returned by any of the functions you invoke, such as diskimg_readsector. When a child function returns an error, the parent function must usually return an error as well.

In addition, before returning an error from any function, you must print a descriptive message to stderr describing the problem that occurred. Be precise: for example, "file_getblock couldn't read disk block 47" is a much more useful message than "oops!" (sadly, messages with about as much useful information as "oops!" occur distressingly often in production code...). Your error messages should include any available information that makes it easier to understand the problem, such as the number of a disk block that couldn't be read, the name of a file that couldn't be found, and so on.

Your error messages do not necessarily need to duplicate those generated by our sample solution, but they must contain appropriate information. Matching the message from the sample solution may be the easiest way to go.

Implementaton Milestones

We recommend that you implement the layers from the bottom up, testing each layer before you move on to the next.

Milestone 1: Inode Layer

Implement this layer in four stages:

  1. Write the code for inode_iget in inode.c. The comment in inode.h describes exactly what this function must do. Your code will invoke diskimg_readsector to read blocks from disk. You will need to consult the file unixfilesystem.h, which describes the disk layout. Test your code with the iget and bogus_inumber tests implemented by test. Compare the output of these tests with the output generated by the sample solution in samples/test_soln.

  2. Implement enough of inode_indexlookup to handle blocks in small files (those with no indirect blocks) and test it with the index_direct test.

  3. Add code to inode_indexlookup to handle indirect blocks and test it with the index_indirect test.

  4. Complete the implementation of inode_indexlookup by writing code to handle doubly-indirect blocks; test it with the index_double_indirect test.

Milestone 2: File Layer

Implement file_getblock in file.c and test it with the getblock test. Once this test passes, you should be able to run ./diskimageaccess -i; try this with each of the images in samples/disk_images and make sure the output matches the output generated by samples/diskimageaccess_soln.

Milestone 3: Directory Layer

Implement the directory_findname function in directory.c, then test it with the findname test.

Mileston 4: Pathname Layer

Implement the pathname_lookup function in pathname.c, then test it with the path test. Once this test passes, you should be able to run ./diskimageaccess -p on each of the disk images in samples/disk_images. Make sure the output matches that from samples/diskimageaccess -p.

Once you have completed all of these milestones sanitycheck should pass as well.

Additional Information

Integer sizes

Back in the 1970s, storage space — both on disk and in main memory — was at a premium. As a result, the Unix v6 file system goes to lengths to reduce the size of data it stores. You'll notice that many integer values in the structs we provided are stored using only 16 bits, rather than today's more standard 32 or 64. (In our code we use the type int16_t from stdint.h to get a 16-bit integer, but back in the '70s, the C int type was 16 bits wide.)

In another space-saving move, the designers of the file system stored the inode's size field as a 24-bit integer. There's no 24-bit integer type in C, so this value is represented using two fields in the inodestruct: i_size1, which contains the least-significant 16 bits of the value, and i_size0, which contains the most-significant 8 bits of the value. We provide a function inode_getsize in inode.c that assembles these two fields into a normal C integer for you.

Inode structure

The structure of an inode is slightly different for this project than the way it was discussed in lecture (this project is based on a more primitive version of Unix than the lectures). For this project, there are only 8 block pointers in an inode, instead of 12 in lecture. Furthermore, the block pointers are interpreted in one of two different ways, depending on whether the file is large or small. For small files ((ino->imode & ILARG) == 0) each of the eight pointers is a direct pointer. For large files, the first seven pointers refer to indirect blocks and the eighth pointer refers to a doubly indirect block.

Disk blocks hold 512 bytes in this project. Block pointers take two bytes, so each indirect or doubly-indirect block holds 256 block pointers.

The first inode

Since there is no inode with an inumber of 0, the designers of the file system decided not to waste the 32 bytes of disk space to store it. The first inode in the first inode block has inumber 1; this inode corresponds to the root directory for the file system The symbol ROOT_INUMBER gives the number of the first stored inode; you should use this in your code, not the number 1. See unixfilesystem.h for details.

Be careful not to assume that the first inode has an inumber of 0. Off-by-one errors are the worst.

Inode's i_mode

The 16-bit integer i_mode in the inode struct isn't really a number; rather, the individual bits of the field indicate various properties of the inode. ino.h contains #defines that describe what each bit means.

For instance, we say an inode is allocated if it describes an existing file. The most-significant bit (i.e. bit 15) of i_mode indicates whether the inode is allocated or not. So the C expression (i_mode & IALLOC) == 0 is true if the inode is unallocated and false otherwise.

Similarly, bit 12 of i_mode indicates whether the file uses the large file mapping scheme. So if (i_mode & ILARG)!= 0, then the inode's i_addr fields point at indirect and doubly-indirect blocks rather than directly at the data blocks.

Bits 14 and 13 form a 2-bit wide field specifying the type of file. This field is 0 for regular files and 2 (i.e. binary 10, or the constant IFDIR) for directories. So the expression (i_mode & IFMT) == IFDIR is true if the inode is a directory, and false otherwise.

Type casting

Try to minimize the use of type casting. For example, when implementing inode_iget, past students have sometimes declared generic character buffers (e.g. char buffer[DISKIMG_SECTOR_SIZE]) to store the contents of a sector in the inode table, and then gone on to use type casting and manual pointer arithmetic to replicate the contents of an inode in the space addressed by inp.

Don’t do that. Instead, declare an array of struct inode records (you know that the contents of sectors in the inode table store inode records, right?) and pass the address of that array to diskimg_readsector. That way, you can manipulate the contents of a strongly typed array of struct inodes instead of a weakly typed array of chars. Stated differently, you don't need to use the void * trickery you learned in CS107 here. You should only use void *, manual pointer arithmetic, and memcpy when you don’t know the data types that are being stored, copied, and passed around. In the case of inode_iget, you know the sectors you’re examining contain arrays of struct inodes.

Similarly, your implementation of directory_findname should declare an array of struct direntv6 instead of a character array of length 512, and you should be able to take advantage of the fact that an indirect block contains an array of 256 two-byte integers (uint16_t).

Constants

Make sure to use the constants defined in the header files and define your own constants where appropriate. For instance, use INODE_START_SECTOR and ROOT_INUMBER from unixfilesystem.h instead of hardcoding numbers such as 2 and 1 in your code. There are calculations throughout your code, and your goal should be to make them as easy to follow as possible.

Memory Allocation

You should not need to use dynamic memory allocation on this assignment (i.e. malloc, realloc, or free).

Debugging

When debugging with the test program, you may find it useful to run either or both of the following commands:

samples/diskimageaccess_soln -i samples/disk_images/basicExtended.img
samples/diskimageaccess_soln -p samples/disk_images/basicExtended.img

These commands will print out the contents of the disk image used by the tests (in two different ways). Seeing the actual disk structure may help you figure out what your code is doing.

Submitting and Grading

Once you are finished working and have saved all your changes, submit by running the submit tool in the tools folder: tools/submit. We recommend you do a trial submit in advance of the deadline to allow time to work through any snags. You may submit as many times as you would like; we will grade the latest submission. Submitting a stable but unpolished/unfinished is like an insurance policy. If the unexpected happens and you miss the deadline to submit your final version, this previous submit will earn points. Without a submission, we cannot grade your work.

We will grade your assignment with the following criteria:

  • Code test results: we will run the sanity check tests as well as potential additional test cases to confirm your functions work as intended and implement the required error checking. You have access to all of the disk images we'll use for grading.
  • Clean build and clean valgrind reports (eg. no memory leaks or errors).
  • Code style: we will manually review your code and give feedback on coding style and design.

Credits

Mendel Rosenblum created this assignment. Improvements (especially to the writeup) have been contributed by Jerry Cain, Chris Gregg, John Ousterhout, and Nick Troccoli.