finish up doc/WritingTests.txt
This commit is contained in:
Nick Mathewson
parent
3780a6b439
commit
f3804a4d6f
|
|
@ -66,6 +66,8 @@ and stuff like that.
|
|||
|
||||
=== Finding test coverage
|
||||
|
||||
Test coverage is a measurement of which lines your tests actually visit.
|
||||
|
||||
When you configure Tor with the --enable-coverage option, it should
|
||||
build with support for coverage in the unit tests, and in a special
|
||||
"tor-cov" binary.
|
||||
|
|
@ -245,29 +247,155 @@ And later, you can restore the original function with:
|
|||
For more information, see the definitions of this mocking logic in
|
||||
testsupport.h.
|
||||
|
||||
=== Okay but what should my tests actually do?
|
||||
|
||||
We talk above about "test coverage" -- making sure that your tests visit
|
||||
every line of code, or every branch of code. But visiting the code isn't
|
||||
enough: we want to verify that it's correct.
|
||||
|
||||
So when writing tests, try to make tests that should pass with any correct
|
||||
implementation of the code, and that should fail if the code doesn't do what
|
||||
it's supposed to do.
|
||||
|
||||
You can write "black-box" tests or "glass-box" tests. A black-box test is
|
||||
one that you write without looking at the structure of the function. A
|
||||
glass-box one is one you implement while looking at how the function is
|
||||
implemented.
|
||||
|
||||
In either case, make sure to consider common cases *and* edge cases; success
|
||||
cases and failure csaes.
|
||||
|
||||
For example, consider testing this function:
|
||||
|
||||
/** Remove all elements E from sl such that E==element. Preserve
|
||||
* the order of any elements before E, but elements after E can be
|
||||
* rearranged.
|
||||
*/
|
||||
void smartlist_remove(smartlist_t *sl, const void *element);
|
||||
|
||||
In order to test it well, you should write tests for at least all of the
|
||||
following cases. (These would be black-box tests, since we're only looking
|
||||
at the declared behavior for the function:
|
||||
|
||||
* Remove an element that is in the smartlist.
|
||||
* Remove an element that is not in the smartlist.
|
||||
* Remove an element that appears in the smartlist more than once.
|
||||
|
||||
And your tests should verify that it behaves correct. At minimum, you should
|
||||
test:
|
||||
|
||||
* That other elements before E are in the same order after you call the
|
||||
functions.
|
||||
* That the target element is really removed.
|
||||
* That _only_ the target element is removed.
|
||||
|
||||
When you consider edge cases, you might try:
|
||||
|
||||
* Remove an element from an empty list.
|
||||
* Remove an element from a singleton list containing that element.
|
||||
* Remove an element for a list containing several instances of that
|
||||
element, and nothing else.
|
||||
|
||||
Now let's look at the implementation:
|
||||
|
||||
void
|
||||
smartlist_remove(smartlist_t *sl, const void *element)
|
||||
{
|
||||
int i;
|
||||
if (element == NULL)
|
||||
return;
|
||||
for (i=0; i < sl->num_used; i++)
|
||||
if (sl->list[i] == element) {
|
||||
sl->list[i] = sl->list[--sl->num_used]; /* swap with the end */
|
||||
i--; /* so we process the new i'th element */
|
||||
sl->list[sl->num_used] = NULL;
|
||||
}
|
||||
}
|
||||
|
||||
Based on the implementation, we now see three more edge cases to test:
|
||||
|
||||
* Removing NULL from the list.
|
||||
* Removing an element from the end of the list
|
||||
* Removing an element from a position other than the end of the list.
|
||||
|
||||
|
||||
=== What should my tests NOT do?
|
||||
|
||||
Tests shouldn't require a network connection.
|
||||
|
||||
Whenever possible, tests shouldn't take more than a second. Put the test
|
||||
into test/slow if it genuinely needs to be run.
|
||||
|
||||
Tests should not alter global state unless they run with TT_FORK: Tests
|
||||
should not require other tests to be run before or after them.
|
||||
|
||||
Tests should not leak memory or other resources.
|
||||
|
||||
When possible, tests should not be over-fit to the implementation. That is,
|
||||
the test should verify that the documented behavior is implemented, but
|
||||
should not break if other permissible behavior is later implemented.
|
||||
|
||||
|
||||
=== Advanced techniques: Namespaces
|
||||
|
||||
XXXX write this. danah boyd made us some really awesome stuff here.
|
||||
Sometimes, when you're doing a lot of mocking at once, it's convenient to
|
||||
isolate your identifiers within a single namespace. If this were C++, we'd
|
||||
already have namespaces, but for C, we do the best we can with macros and
|
||||
token-pasting.
|
||||
|
||||
We have some macros defined for this purpose in src/test/test.h. To use
|
||||
them, you define NS_MODULE to a prefix to be used for your identifiers, and
|
||||
then use other macros in place of identifier names. See src/test/test.h for
|
||||
more documentation.
|
||||
|
||||
|
||||
Integration tests: Calling Tor from the outside
|
||||
-----------------------------------------------
|
||||
|
||||
XXXX WRITEME
|
||||
Some tests need to invoke Tor from the outside, and shouldn't run from the
|
||||
same process as the Tor test program. Reasons for doing this might include:
|
||||
|
||||
* Testing the actual behavior of Tor when run from the command line
|
||||
* Testing that a crash-handler correctly logs a stack trace
|
||||
* Verifying that a violating a sandbox or capability requirement will
|
||||
actually crash the program.
|
||||
* Needing to run as root in order to test capability inheritance or
|
||||
user switching.
|
||||
|
||||
To add one of these, you generally want a new C program in src/test. Add it
|
||||
to TESTS and noinst_PROGRAMS if it can run on its own and return success or
|
||||
failure. If it needs to be invoked multiple times, or it needs to be
|
||||
wrapped, add a new shell script to TESTS, and the new program to
|
||||
noinst_PROGRAMS. If you need access to any environment variable from the
|
||||
makefile (eg ${PYTHON} for a python interpreter), then make sure that the
|
||||
makefile exports them.
|
||||
|
||||
Writing integration tests with Stem
|
||||
-----------------------------------
|
||||
|
||||
XXXX WRITEME
|
||||
The 'stem' library includes extensive unit tests for the Tor controller
|
||||
protocol.
|
||||
|
||||
For more information on writing new tests for stem, have a look around
|
||||
the tst/* directory in stem, and find a good example to emulate. You
|
||||
might want to start with
|
||||
https://gitweb.torproject.org/stem.git/tree/test/integ/control/controller.py
|
||||
to improve Tor's test coverage.
|
||||
|
||||
You can run stem tests from tor with "make test-stem", or see
|
||||
https://stem.torproject.org/faq.html#how-do-i-run-the-tests .
|
||||
|
||||
System testing with Chutney
|
||||
---------------------------
|
||||
|
||||
XXXX WRITEME
|
||||
The 'chutney' program configures and launches a set of Tor relays,
|
||||
authorities, and clients on your local host. It has a 'test network'
|
||||
functionality to send traffic through them and verify that the traffic
|
||||
arrives correctly.
|
||||
|
||||
Who knows what evil lurks in the timings of networks? The Shadow knows!
|
||||
-----------------------------------------------------------------------
|
||||
|
||||
XXXX WRITEME
|
||||
You can write new test networks by adding them to 'networks'. To add
|
||||
them to Tor's tests, add them to the test-network or test-network-all
|
||||
targets in Makefile.am.
|
||||
|
||||
(Adding new kinds of program to chutney will still require hacking the
|
||||
code.)
|
||||
|
|
|
|||
Loading…
Reference in New Issue