Getting Started Testing your Python

Ned Batchelder

@nedbat

http://nedbatchelder/text/starttest.html

Goals

Convince you to test
Show you a way to test your Python code
Remove the mystery

Why test?

The best way to know if your code works
Saves time in the long run
Produces better code
Removes fear from development

Common testing mindset

Most developers' #1 thought about testing:

I AM BAD

Reality

Developers aren't bad (most of them)
Testing is hard
It needs to be given its due
It's real engineering
It's worth it

Starting from first principles

Evolving tests

Stock portfolio class

A simple system under test:

01# portfolio1.py
02 
03class Portfolio(object):
04    """A simple stock portfolio"""
05    def __init__(self):
06        # stocks is a list of lists:
07        #   [[name, shares, price], ...]
08        self.stocks = []
09 
10    def buy(self, name, shares, price):
11        """Buy `name`: `shares` shares at `price`."""
12        self.stocks.append([name, shares, price])
13 
14    def cost(self):
15        """What was the total cost of this portfolio?"""
16        amt = 0.0
17        for name, shares, price in self.stocks:
18            amt += shares * price
19        return amt

First test: interactive

01$ python
02Python 2.7.2 (default, Jun 12 2011, 15:08:59) 
03>>> from portfolio1 import Portfolio
04>>> p = Portfolio()
05>>> p.cost()
060.0
07 
08>>> p.buy("IBM", 100, 176.48)
09>>> p.cost()
1017648.0
11 
12>>> p.buy("HPQ", 100, 36.15)
13>>> p.cost()
1421263.0

Good: testing the code
Bad: not repeatable
Bad: labor intensive
Bad: is it right?

Second test: standalone

1# porttest1.py
2from portfolio1 import Portfolio
3 
4p = Portfolio()
5print "Empty portfolio cost: %s" % p.cost()
6p.buy("IBM", 100, 176.48)
7print "With 100 IBM @ 176.48: %s" % p.cost()
8p.buy("HPQ", 100, 36.15)
9print "With 100 HPQ @ 36.15: %s" % p.cost()

1$ python porttest1.py
2Empty portfolio cost: 0.0
3With 100 IBM @ 176.48: 17648.0
4With 100 HPQ @ 36.15: 21263.0

Good: testing the code
Better: repeatable
Better: low effort
Still bad: is it right?

Third test: expected results

1# porttest2.py
2from portfolio1 import Portfolio
3 
4p = Portfolio()
5print "Empty portfolio cost: %s, should be 0.0" % p.cost()
6p.buy("IBM", 100, 176.48)
7print "With 100 IBM @ 176.48: %s, should be 17648.0" % p.cost()
8p.buy("HPQ", 100, 36.15)
9print "With 100 HPQ @ 36.15: %s, should be 21263.0" % p.cost()

1$ python porttest2.py
2Empty portfolio cost: 0.0, should be 0.0
3With 100 IBM @ 176.48: 17648.0, should be 17648.0
4With 100 HPQ @ 36.15: 21263.0, should be 21263.0

Good: testing the code repeatably with low effort
Better: explicit expected results
Bad: have to check the results yourself

Fourth test: check results automatically

01# porttest3.py
02from portfolio1 import Portfolio
03 
04p = Portfolio()
05print "Empty portfolio cost: %s, should be 0.0" % p.cost()
06assert p.cost() == 0.0
07p.buy("IBM", 100, 176.48)
08print "With 100 IBM @ 176.48: %s, should be 17648.0" % p.cost()
09assert p.cost() == 17648.0
10p.buy("HPQ", 100, 36.15)
11print "With 100 HPQ @ 36.15: %s, should be 21263.0" % p.cost()
12assert p.cost() == 21263.0

1$ python porttest3.py
2Empty portfolio cost: 0.0, should be 0.0
3With 100 IBM @ 176.48: 17648.0, should be 17648.0
4With 100 HPQ @ 36.15: 21263.0, should be 21263.0

Good: testing the code repeatably with low effort
Good: explicit expected results
Good: results checked automatically

Fourth test: what failure looks like

1$ python porttest3_broken.py
2Empty portfolio cost: 0.0, should be 0.0
3With 100 IBM @ 176.48: 17600.0, should be 17648.0
4Traceback (most recent call last):
5  File "porttest3_broken.py", line 9, in <module>
6    assert p.cost() == 17648.0
7AssertionError

Good: testing the code repeatably with low effort
Good: expected results checked automatically
So-so: failure is visible, but output is cluttered with success
Bad: failure prevents further tests

This is starting to get complicated!

Like any code, tests will grow
A test is a real program
Testing deserves real engineering
Common issues can be handled in standard ways
Unique issues can be handled in custom ways

unittest

Writing tests

unittest

Part of the Python standard library
Provides infrastructure for well-structured tests
Provides a test runner
Based on patterns used across many languages

Test classes

Tests are organized into classes
Tests are methods starting with test_

A simple unit test

01# test_port1.py
02 
03import unittest
04from portfolio1 import Portfolio
05 
06class PortfolioTest(unittest.TestCase):
07    def test_ibm(self):
08        p = Portfolio()
09        p.buy("IBM", 100, 176.48)
10        assert p.cost() == 17648.0
11 
12if __name__ == '__main__':
13    unittest.main()

1$ python test_port1.py
2.
3----------------------------------------------------------------------
4Ran 1 test in 0.000s
5 
6OK

Under the covers

1# unittest runs the tests as if I had written:
2testcase = PortfolioTest()
3try:
4    testcase.test_ibm()
5except:
6    [record failure]
7else:
8    [record success]

Add more tests

06class PortfolioTest(unittest.TestCase):
07    def test_empty(self):
08        p = Portfolio()
09        assert p.cost() == 0.0
10 
11    def test_ibm(self):
12        p = Portfolio()
13        p.buy("IBM", 100, 176.48)
14        assert p.cost() == 17648.0
15 
16    def test_ibm_hpq(self):
17        p = Portfolio()
18        p.buy("IBM", 100, 176.48)
19        p.buy("HPQ", 100, 36.15)
20        assert p.cost() == 21263.0

1$ python test_port2.py
2...
3----------------------------------------------------------------------
4Ran 3 tests in 0.000s
5 
6OK

A dot for every passed test

Under the covers

01# unittest runs the tests as if I had written:
02testcase = PortfolioTest()
03try:
04    testcase.test_empty()
05except:
06    [record failure]
07else:
08    [record success]
09 
10testcase = PortfolioTest()
11try:
12    testcase.test_ibm()
13except:
14    [record failure]
15else:
16    [record success]
17 
18testcase = PortfolioTest()
19try:
20    testcase.test_ibm_hpq()
21except:
22    [record failure]
23else:
24    [record success]

Test isolation

Every test is run with a new test object
Test isolation is very important
Tests are focused on a small amount of code
Tests can't affect each other
Failure doesn't prevent further tests

What failure looks like

01$ python test_port2_broken.py
02.F.
03======================================================================
04FAIL: test_ibm (__main__.PortfolioTest)
05----------------------------------------------------------------------
06Traceback (most recent call last):
07  File "test_port2_broken.py", line 14, in test_ibm
08    assert p.cost() == 17648.0
09AssertionError
10 
11----------------------------------------------------------------------
12Ran 3 tests in 0.001s
13 
14FAILED (failures=1)

Better: the failed test didn't keep others from running
Bad: what value was returned?

unittest assert helpers

Use self.assertEqual(x, y) instead of assert x == y
You get nicer messages

11def test_ibm(self):
12    p = Portfolio()
13    p.buy("IBM", 100, 176.48)
14    self.assertEqual(p.cost(), 17648.0)

01$ python test_port3_broken.py
02.F.
03======================================================================
04FAIL: test_ibm (__main__.PortfolioTest)
05----------------------------------------------------------------------
06Traceback (most recent call last):
07  File "test_port3_broken.py", line 14, in test_ibm
08    self.assertEqual(p.cost(), 17648.0)
09AssertionError: 17600.0 != 17648.0
10 
11----------------------------------------------------------------------
12Ran 3 tests in 0.001s
13 
14FAILED (failures=1)

unittest assert helpers

There are lots of specialized assert methods

01assertEqual(first, second)
02assertNotEqual(first, second)
03assertTrue(expr)
04assertFalse(expr)
05assertIn(first, second)
06assertNotIn(first, second)
07assertAlmostEqual(first, second)
08assertGreater(first, second)
09assertLess(first, second)
10assertRegexMatches(text, regexp)
11.. etc ..

Major enhancements in 2.7

unittest expanded significantly in 2.7 and 3.2
Some of what I show here is only available then
Test discovery
More asserters

Three possible outcomes

Test raises an exception other than AssertionError
Produces an E outcome

01$ python test_port3_broken2.py
02.E.
03======================================================================
04ERROR: test_ibm (__main__.PortfolioTest)
05----------------------------------------------------------------------
06Traceback (most recent call last):
07  File "test_port3_broken2.py", line 13, in test_ibm
08    p.buyxxxxx("IBM", 100, 176.48)
09AttributeError: 'Portfolio' object has no attribute 'buyxxxxx'
10 
11----------------------------------------------------------------------
12Ran 3 tests in 0.000s
13 
14FAILED (errors=1)

Under the covers

1testcase = PortfolioTest()
2try:
3    testcase.test_method()
4except AssertionError:
5    [record failure]
6except:
7    [record error]
8else:
9    [record success]

Testing for failure

What if your code is supposed to raise an exception?

1$ python
2Python 2.7.2 (default, Jun 12 2011, 15:08:59) 
3>>> from portfolio1 import Portfolio
4>>> p = Portfolio()
5>>> p.buy("IBM")
6Traceback (most recent call last):
7  File "<console>", line 1, in <module>
8TypeError: buy() takes exactly 4 arguments (2 given)

Can't just call the function

22def test_bad_input(self):
23    p = Portfolio()
24    p.buy("IBM")

01$ python test_port4_broken.py
02E...
03======================================================================
04ERROR: test_bad_input (__main__.PortfolioTest)
05----------------------------------------------------------------------
06Traceback (most recent call last):
07  File "test_port4_broken.py", line 24, in test_bad_input
08    p.buy("IBM")
09TypeError: buy() takes exactly 4 arguments (2 given)
10 
11----------------------------------------------------------------------
12Ran 4 tests in 0.001s
13 
14FAILED (errors=1)

assertRaises

assertRaises(exc, callable, arg1, arg2, ...)
Have to provide callable and args separately

22def test_bad_input(self):
23    p = Portfolio()
24    self.assertRaises(TypeError, p.buy, "IBM")

1$ python test_port4.py
2....
3----------------------------------------------------------------------
4Ran 4 tests in 0.000s
5 
6OK

Nicer in 2.7:

1def test_bad_input(self):
2    p = Portfolio()
3    with self.assertRaises(TypeError):
4        p.buy("IBM")

Add sell functionality

21def sell(self, name, shares):
22    """Sell some number of shares of `name`."""
23    for holding in self.stocks:
24        if holding[0] == name:
25            if holding[1] < shares:
26                raise ValueError("Not enough shares")
27            holding[1] -= shares
28            break
29    else:
30        raise ValueError("You don't own that stock")

Testing sell

26class PortfolioSellTest(unittest.TestCase):
27    def test_sell(self):
28        p = Portfolio()
29        p.buy("MSFT", 100, 27.0)
30        p.buy("DELL", 100, 17.0)
31        p.buy("ORCL", 100, 34.0)
32        p.sell("MSFT", 50)
33        self.assertEqual(p.cost(), 6450)
34 
35    def test_not_enough(self):
36        p = Portfolio()
37        p.buy("MSFT", 100, 27.0)
38        p.buy("DELL", 100, 17.0)
39        p.buy("ORCL", 100, 34.0)
40        with self.assertRaises(ValueError):
41            p.sell("MSFT", 200)
42 
43    def test_dont_own_it(self):
44        p = Portfolio()
45        p.buy("MSFT", 100, 27.0)
46        p.buy("DELL", 100, 17.0)
47        p.buy("ORCL", 100, 34.0)
48        with self.assertRaises(ValueError):
49            p.sell("IBM", 1)

Setting up a test

Tests can get repetitive
Many tests may need similar setup
A class can define setUp() to create initial state

26class PortfolioSellTest(unittest.TestCase):
27    def setUp(self):
28        self.p = Portfolio()
29        self.p.buy("MSFT", 100, 27.0)
30        self.p.buy("DELL", 100, 17.0)
31        self.p.buy("ORCL", 100, 34.0)
32 
33    def test_sell(self):
34        self.p.sell("MSFT", 50)
35        self.assertEqual(self.p.cost(), 6450)
36 
37    def test_not_enough(self):
38        with self.assertRaises(ValueError):
39            self.p.sell("MSFT", 200)
40 
41    def test_dont_own_it(self):
42        with self.assertRaises(ValueError):
43            self.p.sell("IBM", 1)

Under the covers

01testcase = PortfolioTest()
02try:
03    testcase.setUp()
04except:
05    [record error]
06else:
07    try:
08        testcase.test_method()
09    except AssertionError:
10        [record failure]
11    except:
12        [record error]
13    else:
14        [record success]
15    finally:
16        try:
17            testcase.tearDown()
18        except:
19            [record error]

setUp and tearDown: isolation!

Establish context for tests
Use them for common pre-test or post-test work
They ensure isolation, even in the face of failures

How NOT to do it

1class MyBadTestCase(unittest.TestCase):
2    def test_a_thing(self):
3        old_global = some_global_thing
4        some_global_thing = new_test_value
5 
6        do_my_test_stuff()
7         
8        some_global_thing = old_global

Common impulse
But if test fails, globals aren't restored

The right way

01class MyGoodTestCase(unittest.TestCase):
02    def setUp(self):
03        self.old_global = some_global_thing
04        some_global_thing = new_test_value
05 
06    def tearDown(self):
07        some_global_thing = self.old_global
08 
09    def test_a_thing(self):
10        do_my_test_stuff()

tearDown gets run no matter what *
Reuse: same setUp and tearDown apply to all the test methods
Test method is small and focused
* unless setUp fails

A better way

1def test_with_special_settings(self):
2    with patch_settings(SOMETHING='special', ANOTHER='weird'):
3        do_my_test_stuff()

01# From: http://stackoverflow.com/questions/913549
02from somewhere import MY_GLOBALS
03 
04NO_SUCH_SETTING = object()
05 
06@contextlib.contextmanager
07def patch_settings(**kwargs):
08    old_settings = []
09    for key, new_value in kwargs.items():
10        old_value = getattr(MY_GLOBALS, key, NO_SUCH_SETTING)
11        old_settings.append((key, old_value))
12        setattr(MY_GLOBALS, key, new_value)
13 
14    yield
15 
16    for key, old_value in old_settings:
17        if old_value is NO_SUCH_SETTING:
18            delattr(MY_GLOBALS, key)
19        else:
20            setattr(MY_GLOBALS, key, old_value)

Tests are real code

Test helpers can become significant
All the same techniques you use for product code, you can use for test code
Helper functions, classes, etc.
BTW: These helpers might need tests!

Approaches

crafting tests

What to test?

Any code that you want to work
Start with what worries you
Where are your bugs?

How to start?

Pick a chunk of code
What should it do?
Test that
What shouldn't it do?
Test that

Test granularity

How big is the code chunk?
Creates a hierarchy of tests:
- Unit
- Functional
- Integration
- System
Start small

What should the code do?

This can be difficult!
Tests force you to decide
Be as clear as you can
(Document it in your docstrings)

Test what the code should do

Typical values
Edge values
Combinations of values

Test what the code shouldn't do

Error conditions
Absurd values
Common mistakes in usage
Does it fail the way it should?

Where to put them?

Put tests in their own files
Name them test_*.py
Parallel to the code they're testing
Pick your granularity

Good tests should be...

Automated
Repeatable
Convenient: fast
Unambiguous

How much is enough?

Hard to know
Depends on your needs
Coverage measurement can help

Tests will fail

Could be the code is broken
Could also be the tests are broken
Tests have to be maintained
Refactoring code might mean refactoring tests too

Test-Driven Development

Write tests before code!
Ensures code is testable
Produces better code

Testability

The quality of being testable
Affects the design of your code
Mostly: improved modularity
See my PyCon 2010 talk: Tests & Testability

nose and py.test

Running tests

Test runners

Third-party test runners: nose, py.test, trial
Know how to run your unittest tests
Test discovery
Run tests better
Plugins provide lots of features

Test discovery

Running one file of tests is easy
Pre-2.7 unittest: hard to run many test files
New test runners provide test discovery
They sniff out your tests wherever they are

Test runners

unittest (2.7)
nose, py.test, trial
IDEs, Vim, Emacs

Lots of options

However you want to run your tests, you can
Choose the runner that suits you
Plugins allow further customization

Coverage

Testing tests

What code are you testing?

The goal: tests execute product code
But do they really?
How much of it?

Coverage measurement

Run your tests
Track what parts of product code are executed
Report on covered / not covered
You find code not being tested
Write more tests
The world is better!

HTML report

Coverage can only tell you a few things

What lines were executed
What branches were taken
100% coverage is difficult to reach
100% coverage doesn't tell you everything

What coverage can't tell you

Are you exercising all your data?
Are you exercising all your HTML templates?
Are you checking results properly?
Are you hitting all edge conditions?
Are you testing the right things?
Are you building the right product? ;-)

Test Doubles

Focusing tests

Testing small amounts of code

Systems are built in layers
Components depend on each other
How to test just one component without its dependencies?

Dependencies are bad

More suspect code in each test
Some components are slow
Some components are unpredictable

Test Doubles

Replace a component's dependencies
Focus on one component
Different kinds: Mocks, stubs, fakes, etc

Real-time data!

Hit the web to get current stock prices

51def current_prices(self):
52    """Return a dict mapping names to current prices."""
53    # http://download.finance.yahoo.com/d/quotes.csv?f=sl1&s=ibm,hpq
54    # returns comma-separated values:
55    #   "IBM",174.23
56    #   "HPQ",35.13
57    url = "http://download.finance.yahoo.com/d/quotes.csv?f=sl1&s="
58    names = [name for name, shares, price in self.stocks]
59    url += ",".join(sorted(names))
60    data = urllib.urlopen(url)
61    prices = dict((sym, float(last)) for sym, last in csv.reader(data))
62    return prices
63 
64def value(self):
65    """Return the current value of the portfolio."""
66    prices = self.current_prices()
67    total = 0.0
68    for name, shares, price in self.stocks:
69        total += shares * prices[name]
70    return total

It works great!

01$ python
02Python 2.7.2 (default, Jun 12 2011, 15:08:59) 
03>>> from portfolio3 import Portfolio
04>>> p = Portfolio()
05>>> p.buy("IBM", 100, 150.0)
06>>> p.buy("HPQ", 100, 30.0)
07 
08>>> p.current_prices()
09{'HPQ': 35.61, 'IBM': 185.21}
10 
11>>> p.value()
1222082.0

But how to test it?

Live data from the web is unpredictable
Could be slowish
Could be unavailable
Question should be:
- "Assuming yahoo.com is working,
- does my code work?"

Fake implementation of current_prices

45# Replace Portfolio.current_prices with a stub implementation.
46# This avoids the web, but also skips all our current_prices
47# code.
48class PortfolioValueTest(unittest.TestCase):
49    def fake_current_prices(self):
50        return {'IBM': 140.0, 'HPQ': 32.0}
51 
52    def setUp(self):
53        self.p = Portfolio()
54        self.p.buy("IBM", 100, 120.0)
55        self.p.buy("HPQ", 100, 30.0)
56        self.p.current_prices = self.fake_current_prices
57 
58    def test_value(self):
59        self.assertEqual(self.p.value(), 17200)

Good: test results are predictable

But some code isn't tested!

01$ coverage run test_port7.py
02........
03----------------------------------------------------------------------
04Ran 8 tests in 0.000s
05 
06OK
07$ coverage report -m
08Name         Stmts   Miss  Cover   Missing
09------------------------------------------
10portfolio3      33      6    82%   57-62
11test_port7      45      0   100%   
12------------------------------------------
13TOTAL           78      6    92%

51def current_prices(self):
52    """Return a dict mapping names to current prices."""
53    # http://download.finance.yahoo.com/d/quotes.csv?f=sl1&s=ibm,hpq
54    # returns comma-separated values:
55    #   "IBM",174.23
56    #   "HPQ",35.13
57    url = "http://download.finance.yahoo.com/d/quotes.csv?f=sl1&s="
58    names = [name for name, shares, price in self.stocks]
59    url += ",".join(sorted(names))
60    data = urllib.urlopen(url)
61    prices = dict((sym, float(last)) for sym, last in csv.reader(data))
62    return prices

Fake urllib.urlopen instead

48# A simple fake for urllib that implements only one method,
49# and is only good for one request.  You can make this much
50# more complex for your own needs.
51class FakeUrllib(object):
52    """An object that can stand in for the urllib module."""
53 
54    def urlopen(self, url):
55        """A stub urllib.urlopen() implementation."""
56        return StringIO('"IBM",140\n"HPQ",32\n')
57 
58 
59class PortfolioValueTest(unittest.TestCase):
60    def setUp(self):
61        self.old_urllib = portfolio3.urllib
62        portfolio3.urllib = FakeUrllib()
63 
64        self.p = Portfolio()
65        self.p.buy("IBM", 100, 120.0)
66        self.p.buy("HPQ", 100, 30.0)
67 
68    def tearDown(self):
69        portfolio3.urllib = self.old_urllib
70 
71    def test_value(self):
72        self.assertEqual(self.p.value(), 17200)

All of our code is executed

01$ coverage run test_port8.py
02........
03----------------------------------------------------------------------
04Ran 8 tests in 0.001s
05 
06OK
07$ coverage report -m
08Name         Stmts   Miss  Cover   Missing
09------------------------------------------
10portfolio3      33      0   100%   
11test_port8      51      0   100%   
12------------------------------------------
13TOTAL           84      0   100%

Stdlib is stubbed
All our code is run
No web access during tests

Even better: mock objects

Mock objects are automatic chameleons
Can act like any object
Also record what happened to them
You can make assertions after the fact

Automatic chameleons

49class PortfolioValueTest(unittest.TestCase):
50    def setUp(self):
51        self.p = Portfolio()
52        self.p.buy("IBM", 100, 120.0)
53        self.p.buy("HPQ", 100, 30.0)
54 
55    def test_value(self):
56        # Create a mock urllib.urlopen
57        with mock.patch('urllib.urlopen') as urlopen:
58 
59            # When called, it will return this value
60            urlopen.return_value = StringIO('"IBM",140\n"HPQ",32\n')
61 
62            # Run the test!
63            self.assertEqual(self.p.value(), 17200)
64 
65            # We can ask the mock what its arguments were
66            urlopen.assert_called_with(
67                "http://download.finance.yahoo.com/d/quotes.csv"
68                "?f=sl1&s=HPQ,IBM"
69                )

Using Foord's Mock: http://www.voidspace.org.uk/python/mock

Test doubles

A powerful way to isolate your code
Focuses tests on one thing at a time
Removes speed bumps and randomness
Lots of libraries to make it easier
Also called Dependency Injection