Unit Testing

Lecture Notes for CS 190
Spring 2015
John Ousterhout

  • Two classes of tests:
    • Unit tests:
      • Focused, low-level; test individual methods or pieces of methods.
      • Easier to ensure that each piece of code is tested
      • Easier to write and run
      • May not catch problems coming from interactions between different pieces of code
    • System tests (or integration tests): test the entire system working together.
      • Good for making sure that all of the pieces work together
      • Can generate more complex interactions between pieces
      • Harder to write and run.
  • Why write tests?
    • Find bugs in the initial version of the code
    • Bugs found earlier in the development cycle are much cheaper to fix than those found later (e.g., in the field)
    • Makes it easier to refactor code later
  • Engineers should write their own unit tests (not separate QA organization).
  • Unit testing frameworks exist for most programming languages
    • Junit for Java
    • cppunit and gtest for C++

Structuring unit tests

  • Test-driven development?
    • Write tests first
    • Tests are black-box (designed from interface, not implementation)
    • Problem: can lead to poorly structured code
  • My preference: white-box tests
    • Design tests by looking at the implementation; make sure every aspect is tested.
    • Isomorphism: test structure matches code structure
      • One test file per code file
      • One group of tests per method, in the same order
      • ...
      • Makes it easy to find relevant tests after code modifications
    • Include the name of the method and the thing being tested in the test name: test_loadTweetFile_cantOpenFile
      • May not need any other documentation in tests
  • Code coverage:
    • Goal: test every piece of functionality
    • Isomorphic structure helps to visualize coverage
    • Use a test coverage tool if available
      • E.g. EclEmma for Java
    • Line-based coverage tools may not catch all the corner cases (e.g., executing loops 0, 1, N times)
  • Each test case should be short and focused:
    • Several small tests better than one long one
  • Build infrastructure to make testing easier:
    • Helper methods
    • Mock out lower-level infrastructure (also called fixtures):
      • Allows tests to be run without full system
      • Easier to test exceptional cases
  • Design for testability: small changes to the system design that make testing much easier
    • Move body of interrupt handler to a separate method
    • Special flags set only during test (typically protected variables):
      • Skip certain operations
      • Use prespecified clock value
      • Override configuration constants/limits (e.g. smaller cache size or maximum argument length)
      • Replace random numbers with predictable ones
    • Keep additional metrics and statistics (count of total cache misses)
  • Test robustness:
    • Ideally, a system change should only break tests related to change
    • Test should not depend on parts of the system other than the specific features they are testing.
    • Examples:
      • Log messages including line numbers
      • Order of enumeration for hashes (sort results)
      • Avoid timing-sensitive tests
  • Accessing private members during tests?
    • In Java, no way: just use protected instead?
    • In C++, #define PRIVATE public for testing
  • Miscellaneous ideas:
    • Write a test before fixing a bug
    • Use tests to "document" tricky corner cases that can occur
    • For complex data structures (e.g. trees), write a method to check internal consistency
      • Use during testing
      • Can also use during production to hunt down non-reproducible problems