Testing with aspectlib.test

Spy & mock toolkit: record/mock decorators

Lightweight spies and mock responses

Example usage, suppose you want to test this class:

>>> class ProductionClass(object):
...     def method(self):
...         return 'stuff'
>>> real = ProductionClass()

With aspectlib.test.mock and aspectlib.test.record:

>>> from aspectlib import weave, test
>>> patch = weave(real.method, [test.mock(3), test.record])
>>> real.method(3, 4, 5, key='value')
3
>>> assert real.method.calls == [(real, (3, 4, 5), {'key': 'value'})]

As a bonus, you have an easy way to rollback all the mess:

>>> patch.rollback()
>>> real.method()
'stuff'

With mock:

>>> from mock import Mock
>>> real = ProductionClass()
>>> real.method = Mock(return_value=3)
>>> real.method(3, 4, 5, key='value')
3
>>> real.method.assert_called_with(3, 4, 5, key='value')

Capture-replay toolkit: Story and Replay

Elaborate tools for testing difficult code

Writing tests using the Story is viable when neither integration tests or unit tests seem adequate:

  • Integration tests are too difficult to integrate in your test harness due to automation issues, permissions or plain lack of performance.
  • Unit tests are too difficult to write due to design issues (like too many dependencies, dependency injects is too hard etc) or take too much time to write to get good code coverage.

The Story is the middle-ground, bringing those two types of testing closer. It allows you to start with integration tests and later mock/stub with great ease all the dependencies.

Warning

The Story is not intended to patch and mock complex libraries that keep state around. E.g.: requests keeps a connection pool around - there are better choices.

Note

Using the Story on imperative, stateless interfaces is best.

An example: mocking out an external system

Suppose we implement this simple GNU tree clone:

>>> import os
>>> def tree(root, prefix=''):
...     if not prefix:
...         print("%s%s" % (prefix, os.path.basename(root)))
...     for pos, name in reversed(list(enumerate(sorted(os.listdir(root), reverse=True)))):
...         print("%s%s%s" % (prefix, "|-- " if pos else "\-- ", name))
...         absname = os.path.join(root, name)
...         if os.path.isdir(absname):
...             tree(absname, prefix + ("|   " if pos else "    "))

Lets suppose we would make up some directories and files for our tests:

>>> if not os.path.exists('some/test/dir'):   os.makedirs('some/test/dir')
>>> if not os.path.exists('some/test/empty'): os.makedirs('some/test/empty')
>>> with open('some/test/dir/file.txt', 'w') as fh:
...     pass

And we’ll assert that tree has this output:

>>> tree('some')
some
\-- test
    |-- dir
    |   \-- file.txt
    \-- empty

But now we’re left with some garbage and have to clean it up:

>>> import shutil
>>> shutil.rmtree('some')

This is not very practical - we’ll need to create many scenarios, and some are not easy to create automatically (e.g: tests for permissions issues - not easy to change permissions from within a test).

Normally, to handle this we’d have have to manually monkey-patch the os module with various mocks or add dependency-injection in the tree function and inject mocks. Either approach we’ll leave us with very ugly code.

With dependency-injection tree would look like this:

def tree(root, prefix='', basename=os.path.basename, listdir=os.listdir, join=os.path.join, isdir=os.path.isdir):
    ...

One could argue that this is overly explicit, and the function’s design is damaged by testing concerns. What if we need to check for permissions ? We’d have to extend the signature. And what if we forget to do that ? In some situations one cannot afford all this (re-)engineering (e.g: legacy code, simplicity goals etc).

The aspectlib.test.Story is designed to solve this problem in a neat way.

We can start with some existing test data in the filesystem:

>>> os.makedirs('some/test/dir')
>>> os.makedirs('some/test/empty')
>>> with open('some/test/dir/file.txt', 'w') as fh:
...     pass

Write an empty story and examine the output:

>>> from aspectlib.test import Story
>>> with Story(['os.path.isdir', 'os.listdir']) as story:
...     pass
>>> with story.replay(strict=False) as replay:
...     tree('some')
some
\-- test
    |-- dir
    |   \-- file.txt
    \-- empty
STORY/REPLAY DIFF:
    --- expected...
    +++ actual...
    @@ ... @@
    +os.listdir('some') == ['test']  # returns
    +...isdir('some...test') == True  # returns
    +os.listdir('some...test') == [...'empty'...]  # returns
    +...isdir('some...test...dir') == True  # returns
    +os.listdir('some...test...dir') == ['file.txt']  # returns
    +...isdir('some...test...dir...file.txt') == False  # returns
    +...isdir('some...test...empty') == True  # returns
    +os.listdir('some...test...empty') == []  # returns
ACTUAL:
    os.listdir('some') == ['test']  # returns
    ...isdir('some...test') == True  # returns
    os.listdir('some...test') == [...'empty'...]  # returns
    ...isdir('some...test...dir') == True  # returns
    os.listdir('some...test...dir') == ['file.txt']  # returns
    ...isdir('some...test...dir...file.txt') == False  # returns
    ...isdir('some...test...empty') == True  # returns
    os.listdir('some...test...empty') == []  # returns

Now we can remove the test directories and fill the story:

>>> import shutil
>>> shutil.rmtree('some')

The story:

>>> with Story(['os.path.isdir', 'os.listdir']) as story:
...     os.listdir('some') == ['test']  # returns
...     os.path.isdir(os.path.join('some', 'test')) == True
...     os.listdir(os.path.join('some', 'test')) == ['dir', 'empty']
...     os.path.isdir(os.path.join('some', 'test', 'dir')) == True
...     os.listdir(os.path.join('some', 'test', 'dir')) == ['file.txt']
...     os.path.isdir(os.path.join('some', 'test', 'dir', 'file.txt')) == False
...     os.path.isdir(os.path.join('some', 'test', 'empty')) == True
...     os.listdir(os.path.join('some', 'test', 'empty')) == []

We can also disable proxying in replay so that the tested code can’t use the real functions:

>>> with story.replay(proxy=False) as replay:
...     tree('some')
some
\-- test
    |-- dir
    |   \-- file.txt
    \-- empty

>>> with story.replay(proxy=False, strict=False) as replay:
...     tree('missing-from-story')
Traceback (most recent call last):
...
AssertionError: Unexpected call to None/os.listdir with args:'missing-from-story' kwargs: