Unit Testing
Lecture Notes for CS 190
Spring 2016
John Ousterhout
- Two classes of tests:
- Unit tests: test individual pieces in isolation
- Focused, low-level; test individual methods or pieces of methods.
- Control: easier to ensure that each piece of code is tested
- Easier to write and run
- May not catch problems coming from interactions between
different pieces of code
- System tests (or integration tests): test the entire
system working together.
- Good for making sure that all of the pieces work together
- Can generate more complex interactions between pieces
- Harder to write and run.
- Why write tests?
- Bugs found earlier in the development cycle are much cheaper
to fix than those found later (e.g., in the field)
- Makes it easier to refactor code later
- Engineers should write their own unit tests (not separate QA
organization).
- Unit testing frameworks exist for most programming languages
- Junit for Java
- cppunit and gtest for C++
Structuring unit tests
- Test-driven development?
- Write tests first
- Tests are black-box (designed from interface, not implementation)
- Problem: can lead to poorly structured code
- Better for features, not for infrastructure
- My preference: white-box tests
- Design tests by looking at the implementation; make sure
every aspect is tested.
- Isomorphism: test structure matches code structure
- One test file per code file
- One group of tests per method, in the same order
- ...
- Makes it easy to find relevant tests after code modifications
- Include the name of the method and the thing being tested in
the test name: test_loadTweetFile_cantOpenFile
- May not need any other documentation in tests
- Each test case should be short and focused:
- Several small tests better than one long one
- Code coverage:
- Goal: test every piece of functionality
- Isomorphic structure helps to visualize coverage
- Use a test coverage tool if available
- Line-based coverage tools may not catch all the corner cases
(e.g., executing loops 0, 1, N times)
- Build infrastructure to make testing easier:
- Generate human-readable strings to compare against
- Mock out lower-level infrastructure (also called fixtures):
- Allows tests to be run without full system
- Easier to test exceptional cases
- Design for testability: small changes to the system design that make
it easier to write tests
- Move body of interrupt handler to a separate method
- Special flags set only during test (typically protected variables):
- Skip certain operations
- Use prespecified clock value
- Override configuration constants/limits (e.g. smaller cache size or
maximum argument length)
- Replace random numbers with predictable ones
- Keep additional metrics and statistics (count of total cache misses)
- Accessing private members during tests?
- In Java, no way: just use protected instead?
- In C++, #define PRIVATE public for testing
- Miscellaneous ideas:
- Write a test before fixing a bug
- Use tests to "document" tricky corner cases that can occur
- For complex data structures (e.g. trees), write a method to check
internal consistency
- Use during testing
- Can also use during production to hunt down non-reproducible problems