Assignment 3: Thread Dispatcher

Up until now you have used the std::thread class to create threads; these threads are implemented by the operating system. We'll refer to these threads as system threads because they are implemented by the operating system. For this project you will create a new implementation of threads in C++; these threads are implemented entirely at user-level (without the knowledge of the operating system). We'll refer to these threads as user-level threads. You will create a thread dispatcher that runs in a single system thread, which is what you get when a new process starts. Your dispatcher will use that system thread to create and run any number of user-level threads. These user-level threads will have all the features of system threads (each has its own stack, they can be scheduled independently, and in Assignment 4 you will implement locks and condition variables for them), but the operating system doesn't know about them: it only knows about the one system thread. Your code to implement user-level threads will be very similar to the code that implements system threads in an operating system running on a single core.

This assignment is an example of virtualization: you'll take one system thread and use it to implement multiple user-level threads, each of which has (almost) the same capabilities as a system thread.

Here are the learning goals for this assignment:

• Learn how dispatching works and what code is required to create, execute, and delete threads.
• Learn how timer interrupts can be used to implement a round-robin scheduler.

Getting Started

To get started on this assignment, login to the myth cluster and clone the starter repo with this command:

git clone /afs/ir/class/archive/cs/cs111/cs111.1246/repos/assign3/$USER assign3

This will create a new directory assign3 in your current directory and it will clone a Git starter repository into that directory. Do your work for the assignment in this directory . The files thread.hh and thread.cc contain a skeleton for all of the code you need to write. Open those files in an editor; you'll see various places with comments such as "You have to implement this". You'll eventually replace each of those comments with code. The other files in the directory provide libraries that you'll use in your solution (such as stack.hh and stack.cc) or examples that will be discussed in section (such as static.cc and interrupt.cc).

The directory contains a Makefile; if you type make, it will compile your code along with the libraries and a test program, creating an executable test. You can invoke the command tools/sanitycheck to run a series of basic tests on your solution. Try this now: the starter code should compile but almost all the tests will fail.

The Thread Class

In this assignment you will create a C++ class Thread with the following methods, all of which are declared in thread.hh:

Thread(std::function<void()> main)

This is the constructor. It creates a new thread that will run main as its top-level function (the function must take no arguments and return no result). The new thread will be added to the ready queue, so the dispatcher will eventually execute it. If main returns, the thread will be terminated just as if it had invoked Thread::exit.

~Thread()

This is the destructor for Thread objects. It will be invoked only from Thread::exit (it is declared private in thread.hh, which prevents code outside the Thread class from invoking it).

void schedule()

When this method is invoked, the associated thread is added to the back of the ready queue; the thread must not currently be on the ready queue.

void Thread::redispatch()

Dispatches the next ready thread without rescheduling the current thread; the current thread will block unless schedule was invoked for it. This method must be invoked with interrupts disabled. This is a static method.

void Thread::exit()

Terminates the thread that is currently running and destroys its Thread object. This does not exit the entire application: other user-level threads can continue to run. This is a static method.

void Thread::yield()

Invokes schedule() on the current thread, followed by redispatch(): the current thread will remain ready, but other threads will get a chance to execute. This is a static method.

Thread* Thread::current()

Returns a pointer to the thread that is currently executing, or nullptr if no thread has ever been dispatched. This is a static method.

C++ Proficiency: Static Methods and Variables

Note that most of the methods of the Thread class are declared as static; this means that the methods are associated with the Thread class rather than a specific Thread object. For a normal method, such as schedule, you invoke it using an instance of the class, like this:

Thread *thread = new Thread(myFunc);
...
thread->schedule();

A static method is declared with the keyword static. For example, here is the declaration of current in thread.hh:

static Thread *current();

You invoke it using the class, like this:

Thread *t = Thread::current();

When a normal method is executing, it can access instance variables in that object directly using their names. When a static method executes, there is no "current object" so it can't reference instance variables directly. However, static methods often have access to objects using other variables, and they can use those pointers to read and write instance variables.

Static methods are useful in situations where there isn't an object available to use for invoking the method, or where the functionality provided by the method doesn't relate to a specific instance of the class. The static methods in the Thread class are declared as static because they can be invoked in places where there is no Thread object available; most of the static methods operate implicitly on the thread that is currently running.

Classes can also contain static variables. A static variable is one that is declared with the keyword static. For example, in the Thread class we have defined the ready queue as a static variable with this declaration:

static std::queue<Thread*> ready_;

Normal variables defined in classes (called instance variables or member variables) are part of objects: there is a separate instance of the variable in each instance of the object. When a method refers to an instance variable, it accesses the variable associated with the current object. A static variable has only one instance, whose storage is allocated outside any object. When a method refers to a static variable, it will access the same variable regardless of the current object. Static variables are useful for holding information that is shared across all of the objects of the class; for example, the ready queue is shared by all threads. Static methods often access static variables (for example, the redispatch method will need to access ready_.)

A quirk of static variables is that they must be defined as well as declared. The code above declares the variable ready_ but does not actually define it (i.e. allocate storage for it). The definition of ready_ is in the file thread.cc:

std::queue<Thread*> Thread::ready_;

Notice that the variable is named with the prefix Thread::, similar to a static method.

In the section for this assignment you will discuss the program static.cc, which illustrates static variables and methods. The code for static.cc is present in your starter repo and make will build the static executable from static.cc, so you can run this program yourself to see how it works.

C++ Proficiency: this

In this assignment you will sometimes find that a C++ method needs a pointer to the object on which it was invoked; a special variable named this is implicitly available in every non-static method to provide that pointer. For example, if a class contains an instance variable foo, you can access that variable in a method either as foo or as this->foo (this example is just an illustration of how this works; we don't recommend actually using the this->foo notation).

There is no this variable in class static methods, since these methods are not invoked on a specific object.

Stack Management

We have created a Stack class to help you manage stacks and context switching. A Stack object contains enough space for a moderate size call stack, plus a place to store the stack pointer register when the stack is not active. The constructor for the class looks like this:

Stack(void(*start)())

This constructor will initialize the object so that the function start will be invoked the first time stack_switch is called to activate the stack.

void stack_switch(Stack *current, Stack *next)

This function performs a context switch by switching execution from one stack to another. current is the Stack object that is active (the stack pointer register points somewhere in this object), and next is the stack to switch to. The function will save registers on the current stack, save the current stack pointer in current, switch to the stack pointer stored in next, and restore the registers saved on that stack. When the function returns, it will be executing on the new stack. The first time that stack_switch switches into a stack, it will start executing code at the start function for that stack; after that switching into a stack will cause execution to resume where it left off (returning from the stack_switch call that switched out of that stack). Note: it is fine for current and next to be the same. As a special case, it is also OK for current to be nullptr; when this happens, the current stack pointer register is not saved (you'll see later why this is needed). You will invoke this function in your implementation of Thread::redispatch.

The code that implements Stack is pretty simple (and interesting!); if you're curious, check out the code in stack.hh and stack.cc in the starter code.

In section you will go over a sample program two_stacks.cc, which illustrates how to use Stack objects and the stack_switch function. Your starter repo contains a copy of two_stacks.cc, and make will build the executable two_stacks from it. After you have discussed this example in section, and before you start implementing this assignment, answer Questions 1A-1C in questions.txt.

Implementation Milestones

You won't need to write very much code for this assignment: our implementation has only about 30 lines (excluding comments). However, it's a bit tricky to figure out what these lines are, and each line is important. So, we have have designed a series of milestones to guide you through the implementation in small steps. After each step you should be able to pass additional tests.

Milestone 1: the First Stack

Unfortunately, the trickiest part of this assignment comes at the beginning: how to create the first Thread and start it executing? It's going to take multiple milestones to do this right. In this milestone you'll create a stack and start executing code on that stack. Every Thread has its own private stack, which will be implemented with the Stack class described above. For starters you can declare a Stack instance variable in thread.hh.

The next step is to initialize the Stack in the Thread constructor. This is where things start getting tricky. When constructing a Stack you must specify a start function, which is the top-level function that will execute on that stack. You might think this can just be the function specified by the main argument to the Thread, and we do eventually want to invoke main. However, it isn't possible to pass main as the argument to the Stack constructor, for two reasons. First, the types are incompatible: the Stack constructor supports only a bare-bones function with no arguments and no results, but the main argument to the Thread constructor is a std::function, which is a more powerful type (e.g. it supports lambdas). Second, you will need to do additional initialization before invoking main, as well as cleanup if main should ever return.

The solution to these issues is to introduce a wrapper function, which is passed to the Stack constructor. The wrapper will be invoked when the Stack begins executing, and it will then invoke the main function. It's called a wrapper because it will (eventually) contain additional code that runs before calling main and after main returns. We have created a skeleton wrapper function in Thread called wrapper; the initial code in wrapper just prints a message and exits the entire process. In future milestones you will replace that code, but leave it there for now.

Now that you know about wrapper, you can add an initializer for the Stack instance variable in your Thread constructor, specifying wrapper as the start argument.

The last step for this milestone is to start executing code on the new Stack. As a temporary hack for this milestone, invoke stack_switch in the Thread constructor. Normally, stack_switch is invoked when running on one Stack, and it will switch to a different Stack; it takes the current and new Stacks as arguments. However, during the first call to stack_switch there is no current Stack: the current stack is one allocated by the Linux kernel for the system thread, which is not a Stack object and cannot be passed to stack_switch. For now, pass nullptr as the first argument to stack_switch, to indicate "no current stack". The result is that we abandon the system thread's stack; after the first call to stack_switch we'll use only Stacks.

Once you've added code to invoke stack_switch, compile your code and run the enter test with the following command:

./test enter

This should print out the message from wrapper before exiting, means you have successfully executed code using a Stack object. This may not seem like much (the enter sanity test won't pass until you complete Milestone 3), but it is actually significant progress!

Milestone 2: the Ready Queue

In Milestone 1 you invoked stack_switch from the Thread constructor, but this isn't how the constructor is supposed to behave. The constructor should add the new Thread to the ready queue; stack_switch should not be invoked until the next invocation of the redispatch method.

For this milestone, implement the schedule and redispatch methods, and call schedule instead of stack_switch in the Thread constructor. We have already declared the ready queue as a class static variable ready_. The schedule method should add the Thread to the ready queue; redispatch should remove the first Thread from the ready queue and stack_switch into its stack. For now, keep using nullptr as the first argument to stack_switch.

Once you have implemented the above changes, try running the enter test again. It should still print same message, but now the Thread has been properly dispatched using the ready queue.

Milestone 3: Into the First Thread

In this milestone you will replace the code in the wrapper function with code to invoke the Thread's main function. This is easy to do, except that wrapper doesn't currently have access to the function to invoke. Making this information available requires two steps. The first step is to modify the Thread class so that the constructor saves its main argument in the Thread (do this now). The second step is for wrapper to use the information saved in the Thread to invoke main.

Unfortunately, wrapper is a static function and it is invoked with no arguments, so it doesn't have immediate access to any Thread objects. It needs a way to find out which Thread is currently executing. It turns out that several other Thread methods, such as exit, yield, and redispatch, will also need a way to identify the current thread.

So, the main work for this milestone is to keep track of the Thread that is currently executing. To do this, you'll need to define a new variable in the Thread class that holds a pointer to the current Thread (should this be an instance variable or a class static variable?). Then you will need to add code to update this variable whenever the current thread changes. Once you've done this, implement Thread::current, which is a static method that returns a pointer to the current Thread. Finally, invoke Thread::current in the wrapper method and use the result to invoke the Thread's main program. In addition, call Thread::exit if main returns.

Now try running the enter test again. This time it should print a different message: "Entered thread". The message is printed by the main program specified in the test, so this means you have successfully dispatched into your first Thread; congratulations!

The enter test should now pass when run under sanitycheck.

Milestone 4: Multiple Threads

In this milestone you will add functionality to support multiple threads and switch between them. To do this, you must implement the yield method. In addition, you will need to fix the stack_switch simplification from Milestones 1 and 2, where you always passed nullptr as the prev argument. From now on, you should pass in nullptr only if there is no current thread (the first time you invoke stack_switch, or if the "current" thread just exited); otherwise you should pass the Stack of the current thread.

Note: if at any point the redispatch method finds that there are no threads left to schedule, then it should invoke std::exit(0) to terminate the program. Exiting the program makes sense because once there are no runnable threads, there is no way for a thread ever to become runnable (even if there are blocked threads, some other thread would have to run in order to unblock them). In a real operating system, device interrupts could cause blocked threads to become runnable, but there are no devices in this assignment.

Once you've written this code, you should be able to pass the ping_pong, round_robin, and block sanity tests.

Milestone 5: Thread Exit

In this milestone you will implement enough functionality for threads to exit cleanly. A Thread exits by calling Thread::exit: either the thread invokes Thread::exit directly, or it returns from its main function, in which case wrapper invokes Thread::exit. Implement Thread::exit now: it should delete the Thread and then call redispatch to run a different Thread.

Unfortunately the Stack creates complications for handling Thread exit. As currently implemented, when you delete a Thread, its Stack will also be deleted, and that's not safe. When the Thread object is deleted, the code that deletes it is actually running on the Stack, it will keep running on that Stack until Thread::redispatch changes to a different Stack. Thus, the stack cannot be freed immediately. However, it must eventually be freed, once you can be sure it's no longer in use. There are several ways to do this, but one simple way is to delay freeing a Thread's Stack until the next Thread is destroyed (at which point we know the old thread can't still be running). To do this, you'll need to implement a destructor for Thread objects. When the destructor is called, it deletes the Stack from the previous Thread and saves the Stack of the current Thread for the next exiting Thread to delete. This means there will always be one Stack that hasn't yet been deleted, but that's OK.

There are a couple of tricky aspects in implementing deferred stack deletion. First, we need to separate the lifetime of a Thread from that of its Stack (the Stack must live on, even after the Thread has been destroyed). As a result, you can't declare the stack instance variable as a Stack in the Thread class, since that would cause the Stack to be deleted when the Thread is deleted. Instead, you'll need to declare the instance variable as a Stack*, and then initialize it by calling new to dynamically allocate a Stack object. This object won't be freed until delete is called on it, so it can live on after the Thread has been destroyed.

You will also need to figure out where to store the old stack pointer so that it can be deleted when the next Thread is destroyed. Hint: this information will need to be accessible to all Threads, since we can't predict which one will be the next to exit.

Once you have written this code, you should be able to pass the exit and stack sanity tests; almost done!

Milestone 6: Enabling Interrupts

So far, your dispatcher has been non-preemptive: a thread can run as long as it likes. The final two milestones will add preemption, so that your dispatcher switches between threads using a round-robin scheduling mechanism with time slices. In this milestone you will add code to enable and disable interrupts properly, but no actual interrupts will occur. The next milestone will activate a timer and handle the interrupts that it generates.

Interrupts must be enabled in order to receive timer interrupts as part of round-robin scheduling. But interrupts must also be disabled at times in order for the thread dispatcher to operate safely. Recall from the lecture discussion of lock implementation that the lowest-level code that implements threads, locks, and condition variables cannot use locks for synchronization (locks don't exist at this level). In systems with only a single core, disabling interrupts provides protection equivalent to acquiring a lock, since the only way a conflicting thread could run is if an interrupt results in a context switch. This assignment is similar to a single-core system, since we are using a single system thread to implement multiple user-level threads: if interrupts are disabled, then we can safely execute code without possible interference from other threads.

Here are the rules for disabling and enabling interrupts:

• Interrupts must be disabled whenever modifying state that is shared among threads (such as the ready queue).
• Interrupts must be disabled whenever redispatch is invoked, and they must (eventually) be reenabled after redispatch returns.
• When a thread starts up for the first time in the wrapper function, it receives control from the dispatcher as if a call to redispatch returned, so interrupts will be disabled; wrapper must reenable interrupts.

We have provided two ways for you to disable interrupts. The best way is to create an IntrGuard object, which is analogous to std::lock_guard. When an IntrGuard object is constructed, it records whether interrupts are currently disabled, and in any case it disables interrupts. When an IntrGuard object is destroyed, it checks the recorded state: if interrupts were enabled when the object was created, then the destructor reenables interrupts; if interrupts were previously disabled, then it leaves them disabled. You should use IntrGuard objects whenever practical.

However, IntrGuard objects can't be used in situations where the code needs to either enable or disable interrupts, but not both. In these few cases you may use the following function:

intr_enable(bool on)

If the argument is false, it defers all interrupts. If the argument is true, it enables interrupts and immediately dispatches any deferred interrupts. Again, use intr_enable only where IntrGuard is not practical.

In section you will discuss two programs, interrupt.cc and interrupt2.cc, which illustrate interrupts and how to disable them. Your starter repo contains both of these files and make will build executables interrupt and interrupt2 from them. After section, but before completing this milestone, answer Question 2 in questions.txt.

Add code to thread.cc to enable and disable interrupts as needed. Once you have added this code, the enable_interrupts sanity test should pass. This test covers some, but not all, of the situations where interrupts must be enabled or disabled, so be sure to look through your code and reason about whether interrupts must be disabled.

Milestone 7: Preemption

To complete the implementation of preemption and round-Robin scheduling, you will need to implement two methods:

• Thread::preempt_init, which initializes preemption (see below).
• A timer interrupt handler.

The Thread::preempt_init method has the following interface:

Thread::preempt_init(std::uint64_t slice_usec)

The slice_usec argument indicates how long time slices should be, in microseconds. This method will be invoked once during initialization (by code outside the Thread class) if preemption is desired. If this method is not invoked, then your dispatcher should be non-preemptive, as it has been up until now.

Your implementation of preemption should use the following function, provided by the starter code in timer.cc, to generate timer interrupts:

void timer_init(std::uint64_t usec, std::function<void()> handler)

You only invoke this method once; after that, timer interrupts will occur every usec microseconds. During each timer interrupt, handler will be invoked.

If the timer fires at a time when interrupts have been disabled, this occurrence will be remembered and an interrupt will occur as soon as interrupts are re-enabled.

To complete this milestone, implement the preempt_init method and the timer handler function. Once you have made these changes, the preempt sanity test should pass. At this point, if you invoke tools/sanitycheck all the tests should pass.

Testing

The testing framework for this assignment is similar to what you have used in other assignments. When you invoke make, it will compile your code along with the file test.cc to form an executable program test. This program contains various tests that invoke your code to exercise features of the Thread class. The sanity checker will invoke test with different parameters to run a series of tests, and it will compare the output of those tests with the output produced by our sample solution, which is in samples/test_soln. The sanity checker prints out the command that it runs for each test, so if a test fails, you can run that command in isolation to repeat the test. You can also invoke samples/test_soln with the same parameter in order to see the expected output, and you can run test under gdb to study its behavior in more detail.

When you're debugging, it may be useful to look at the actual test code so you can see exactly what it's trying to do. All of the test code is in the file test.cc. Each test has a name such as enter, which is included on the command line to run the test, and the test is implemented by a function with the same name. Open test.cc and find the function enter, which is the first test that the sanity checker will run. As you can see, it is trying to create a new thread with enter_thread as the top-level method; then it invokes the dispatcher so the thread will run. You can find the code for enter_thread just above the code for enter; it prints a message and then exits the test application.

Note: we don't guarantee that the tests we have provided are exhaustive, so passing all of the tests is not necessarily sufficient to ensure a perfect score (CAs may discover other problems in reading through your code).

Miscellaneous Notes

• You may assume that your code will only run on single-core systems, so disabling interrupts is sufficient to ensure that no other thread will run.
• Do not use std::thread in this assignment; the Thread class is a replacement for std::thread.
• Don't worry about Valgrind errors (stack switching can cause false positives). Just try and make sure your code frees things it allocates.

Submitting

Once you are finished working and have saved all your changes, submit by running tools/submit.

We recommend you do a trial submission in advance of the deadline to allow time to work through any snags. You may submit as many times as you like; we will grade the latest submission. Submitting a stable but unpolished/unfinished version is like an insurance policy. If the unexpected happens and you miss the deadline to submit your final version, the earlier submit will earn points. Without a submission, we cannot grade your work. You can confirm the timestamp of your latest submission in your course gradebook.

Grading

Here is a recap of the work that will be graded on this assignment:

• questions.txt: answer all of the questions.
• thread.hh and thread.cc: flesh out the Thread class; it should work in both preemptive and non-preemptive modes.
• Code review for style.

We will grade your code using the provided sanity check tests and possible additional autograder tests. We will also review your code for other possible errors and for style. Check out our course style guide for tips and guidelines for writing code with good style!