Project 1: Leader Election using the Raft Consensus Protocol
In Projects 1 and 2 you and a teammate will implement the Raft
consensus protocol. In Project 1 you will implement enough of the
protocol to perform leader election (everything but the log); you
will also implement a simple remote shell. In
Project 2 you will add the log to create a replicated state machine
that executes shell commands.
For an introduction to Raft, please read the
extended version of the Raft paper. For this course you will need to
understand Sections 1-5 and the first part of Section 8 (finding the
leader); you do not need to understand Sections 6 and 7 in detail. Over
the course of Projects 1 and 2 you will implement all of the features
described in Section 5.
Features
For Project 1, you must implement enough of the Raft protocol to
start a cluster of servers, elect a leader, maintain leadership
with heartbeats, and elect a new leader if the current leader
fails or can't communicate with the other servers. Specifically,
you must:
- Create a mechanism for network communication between machines that is
sufficient for the needs of your Raft implementation.
The mechanism must support both communication between servers and
communication between clients and servers. You may not use any existing
library for network communication, such as gRPC.
You should build your mechanism directly on the basic socket system
calls such as
socket
, bind
,
accept
, and connect
.
- Implement the RequestVote and AppendEntries requests described in
the Raft paper. You won't implement the actual log until Project 2,
so for this project AppendEntries requests will not include any log
entries (they will be used only for heartbeats). Implement the requests as
if each server has a log that is empty.
- Implement the timeout-based leader election mechanism described
in the Raft paper. Once a leader gets elected, it must issue
AppendEntries requests to preserve its leadership.
If the current leader crashes or fails to send AppendEntries requests
for any reason, then other servers must start a new election.
The election mechanism must handle split votes as described in the
Raft paper. For this project you will not be able to incorporate log
information into the voting process as described in Section 5.4.1
of the Raft paper, but you should be able to implement all of the
other election features.
- Implement a simple persistent storage mechanism to preserve each
server's current term and vote for that term, if any. The current
values for this information must be stored safely in a file before
a server sends a request or response, and the server must recover
this information from the file when it restarts. You may not use
an existing database system or storage system for this: you must build
your functionality on the basic C/C++ file I/O facilities such as
fopen
, std::iostream
std::fstream. If
you are not sure whether it's OK to use a particular mechanism in
your implementation, check with me.
- Store the persistent data for each server in the current working
directory where that server is invoked. It must be possible to run
several servers simultaneously on a single laptop, each storing its
data in a different directory and listening on a different TCP port.
You will find this useful for testing.
- Ensure that your cluster can elect and maintain a stable leader
as long as a majority of the servers in the cluster are running and
respond promptly to requests. For example, if one server is down
or responds very
slowly to requests, it must still be possible for the other servers to
elect a leader. This means that each server must be capable of
issuing requests concurrently to all of the other servers in the cluster.
You cannot wait for a RequestVote request to one server to complete before
issuing a RequestVote request to another; if you do, a
slow response to the first request could prevent the election from completing
successfully. If a server crashes and restarts, the existing leader
must reconnect to that server before its election timeout expires, so
the restarting server doesn't trigger a new election.
- Write a simple client application that executes a loop where each
iteration (a) reads a one-line shell command from standard input, (b)
sends the command to the cluster leader, and (c) prints on standard
output the results returned by the leader.
When the leader receives a command from a client, it executes the
command as a shell command by invoking
bash
with the
-c
option. Once the command is completed, the server
returns to the client any output generated by the command.
If a follower receives a client command, it must reject it and the
client must retry with the leader (you must provide a mechanism that
allows clients to identify the leader). If the leader crashes, the client must
find the new leader, once it has been elected, and retry the request with
that leader. Note that leaders can crash in the middle of executing a client
request as well as between requests.
I recommend that you read through the Project 2 description before you
start on Project 1. Knowing what you will need to implement in
Project 2 may change how you implement Project 1,
which will save you time later.
Additional Notes and Requirements
- You must implement the project in C++.
- You must work in teams of two (one team of three is OK if we
end up with an odd number of students). This is important for three reasons:
first, by working in teams you can attack a larger project,
which leads to more interesting design issues; second, the
team approach means that you have someone with whom you can discuss
your designs; and third, it reduces the number of projects that I
have to read, which permits a better tradeoff between class size
and instructor sanity.
- Your most important goal is to create a clean, simple, and elegant
code structure. Although I expect your code to work, and it
must implement the features described above, I will be
judging it primarily on its structure; it's better to spend time
cleaning up the structure and documentation than fixing minor bugs.
- When designing your interfaces, you must work from scratch,
without using or consulting any existing code that offers similar
functionality, such as existing communication libraries or implementations
of Raft. This is important so that you can make design decisions on
your own. In addition, many existing packages have
bad interfaces, so they may not serve as good models. You may use
any of the C++ std:: classes, such as std::unordered_map.
If you have any questions about what existing packages it is OK to
use, please ask me.
- You may use an existing package for translating between in-memory
data structures and network messages, such as protocol buffers (I
recommend using protocol buffers).
- I will create a private GitHub repository for your team to use
and send you information about this repository. Create a branch in
this repo named
project1
and use this branch for all
of your work on the project.
- For at least one (non-trivial) source file, write the top-level
declarations and interface comments before you fill in any of the
method bodies. Create a commit on the
project1
branch
whose only change is the skeletal version of this file, and tag
that commit commentsBeforeCode1
. Make sure that the
commit message also includes the name of this file. Some of this initial
information will inevitably change in later commits; that's OK. I
recommend that you write comments before code for all your
files, but I will only require it for one file.
- Debugging distributed systems like this one is hard. For example,
you can't always use breakpoints to debug: if a server
enters the debugger, it no longer responds to requests, and this can
lead to timeouts elsewhere in the cluster. Good logging is crucial:
write out messages to the console or to a file that describe all of
the major actions of the system, such as receiving requests,
granting votes, becoming leader, etc. You can then compare the logs for
different servers to understand the system's behavior and track
down problems.
- Your implementation must allow sockets to be reused for multiple
requests. This is more difficult than discarding sockets after each
request, but it provides significantly better performance and would be
a requirement for any production implementation of Raft.
Sockets may still get closed, such as when machines crash; when this
happens you must reopen them.
- Overall, your implementation should provide "reasonable" performance,
meaning that it shouldn't be dramatically slower than the fastest possible
implementation.
- Be sure to think about memory management issues: who is responsible for
dynamically allocated memory, and when does it get freed?
You may find the classes
std::unique_ptr
and std::shared_ptr
helpful.
- Each client and server in a Raft cluster needs to know the network
addresses of the servers in the cluster. You do not need to implement the
cluster membership mechanism from the Raft paper. You can pick a simple
approach such as providing a list of server names on the command-line
when you invoke a client or server.
- Your implementation must allow servers to execute on different
machines, or all on a single machine.
- Your command execution mechanism must support multiple client
machines at once.
- I recommend that you choose an election timeout in the range
of 5-10 seconds; this is short enough that you won't have to wait
a long time during testing, but long enough so that you don't get
overwhelmed with log output from frequent elections.
- Figure 2 in the Raft paper provides a very precise specification
of the behavior of the system, and you should follow this religiously.
Even small deviations are likely to result in bugs. You might be
interested in reading the
advice
given to MIT students implementing Raft in a distributed systems class.
Development Environment
I strongly recommend using an integrated development environment (IDE)
for the projects in this class. For example, Eclipse
has good support for C++. Please configure your development
environment so that indent widths are 4 spaces, and only space
characters are stored in files, never tabs (this will make it easier for
me to review your code: tab characters and/or 8-space indents result
in very long lines in the code review tool).
I believe that you can use the following instructions to configure
Eclipse for this:
- Go to Window->Preferences->C++->Code Style->Formatter
- Select "Edit..."
- Set "Indentation size" to 4, "Tab size" to 4, and "Tab policy"
to "Spaces only", then save this (you may need to create a new
named profile to save this information).
- If you have already created some files using different
indentation, you can reformat them by selecting the files
in the Package Explorer, then right-clicking one of the
files and selecting Source->Format.
Please keep your lines no longer than 80 characters in length. Long lines
make code harder to read.
Submitting Your Project
To submit your project, push all of your changes to GitHub on a branch
named project1
.
Then create a pull request on GitHub. The base for the pull
request should be your
master
branch (which has nothing on it except your
initial commit) and the comparison branch should be the head of your
project1
branch. Use "Project 1" as the title for your pull
request. If your project is not completely functional at the time you submit,
describe what is and isn't working in the comments for the pull request.
If you are planning to use late days for this project (or any project) please
send me an email before the project deadline so that I know how many late
days you plan to use.
README file
I will download your GitHub repo and I may try running your code on
my laptop. Please create a README file in the top-level directory
of your project with instructions for how to run your server (e.g.,
how does each server know the names of the other servers?).