TESTING IN PYTHON

Automated vs. Manual Testing

Exploratory testing is a form of manual testing that is done without a plan. In an exploratory test, you’re just exploring the application.
To have a complete set of manual tests, all you need to do is make a list of all the features your application has, the different types of input it can accept, and the expected results. Now, every time you make a change to your code, you need to go through every single item on that list and check it.
Automated testing is the execution of your test plan (the parts of your application you want to test, the order in which you want to test them, and the expected responses) by a script instead of a human.
Python already comes with a set of tools and libraries to help you create automated tests for your application.

Unit Tests vs. Integration Tests

Testing multiple components is known as integration testing.
An integration test checks that components in your application operate with each other.
A major challenge with integration testing is when an integration test doesn’t give the right result. It’s very hard to diagnose the issue without being able to isolate which part of the system is failing.
A unit test is a smaller test, one that checks that a single component operates in the right way. A unit test helps you to isolate what is broken in your application and fix it faster.

Eample: To write a unit test for the built-in function sum(), you would check the output of sum() against a known output.

>>> assert sum([1, 2, 3]) == 7, "Should be 6"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError: Should be 6

If the result from sum() is incorrect, this will fail with an AssertionError and the message "Should be 6".
If the result is correct, nothing will be printed on the screen.
Instead of testing on the REPL, you’ll want to put this into a new Python file called test_sum.py and execute it again:

def test_sum():
    assert sum([1, 2, 3]) == 6, "Should be 6"

if __name__ == "__main__":
    test_sum()
    print("Everything passed")

Writing tests in this way is okay for a simple check, but what if more than one fails? This is where test runners come in. The test runner is a special application designed for running tests, checking the output, and giving you tools for debugging and diagnosing tests and applications.

Choosing a Test Runner

There are many test runners available for Python. The one built into the Python standard library is called unittest.
The three most popular test runners are:

• unittest
• nose or nose2
• pytest

unittest

contains both a testing framework and a test runner.
unittest requires that:

• You put your tests into classes as methods
• You use a series of special assertion methods in the unittest.TestCase class instead of the built-in assert statement

To convert the earlier example to a unittest test case, you would have to:

• Import unittest from the standard library
• Create a class called TestSum that inherits from the TestCase class
• Convert the test functions into methods by adding self as the first argument
• Change the assertions to use the self.assertEqual() method on the TestCase class
• Change the command-line entry point to call unittest.main()

import unittest

class TestSum(unittest.TestCase):

    def test_sum(self):
        self.assertEqual(sum([1, 2, 3]), 6, "Should be 6")

    def test_sum_tuple(self):
        self.assertEqual(sum((1, 2, 2)), 6, "Should be 6")

if __name__ == '__main__':
    unittest.main()

python test_sum_unittest.py
.F
======================================================================
FAIL: test_sum_tuple (__main__.TestSum)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_sum_unittest.py", line 9, in test_sum_tuple
    self.assertEqual(sum((1, 2, 2)), 6, "Should be 6")
AssertionError: Should be 6

----------------------------------------------------------------------
Ran 2 tests in 0.001s

FAILED (failures=1)

nose

You may find that over time, as you write hundreds or even thousands of tests for your application, it becomes increasingly hard to understand and use the output from unittest.
nose is compatible with any tests written using the unittest framework and can be used as a drop-in replacement for the unittest test runner.
The development of nose as an open-source application fell behind, and a fork called nose2 was created. If you’re starting from scratch, it is recommended that you use nose2 instead of nose.
To get started with nose2, install nose2 from PyPI and execute it on the command line. nose2 will try to discover all test scripts named test*.py and test cases inheriting from unittest.TestCase in your current directory:

$ pip install nose2
$ python -m nose2
.F
======================================================================
FAIL: test_sum_tuple (__main__.TestSum)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_sum_unittest.py", line 9, in test_sum_tuple
    self.assertEqual(sum((1, 2, 2)), 6, "Should be 6")
AssertionError: Should be 6

----------------------------------------------------------------------
Ran 2 tests in 0.001s

FAILED (failures=1)

Nose 2 documentation.

pytest

supports execution of unittest test cases.
The real advantage of pytest comes by writing pytest test cases. pytest test cases are a series of functions in a Python file starting with the name test_.
pytest has some other great features:

• Support for the built-in assert statement instead of using special self.assert*() methods
• Support for filtering for test cases
• Ability to rerun from the last failing test
• An ecosystem of hundreds of plugins to extend the functionality
  Writing the TestSum test case example for pytest would look like this:

def test_sum():
    assert sum([1, 2, 3]) == 6, "Should be 6"

def test_sum_tuple():
    assert sum((1, 2, 2)) == 6, "Should be 6"

You have dropped the TestCase, any use of classes, and the command-line entry point.
Pytest Documentation Website.

structure

project/
│
├── my_sum/
│   └── __init__.py
|
└── tests/
    └── test.py

Creating the __init__.py file means that the my_sum folder can be imported as a module from the parent directory.
The test file will need to be able to import your application to be able to test it, you want to place test.py above the package folder

my_sum/__init__.py

def sum(arg):
    total = 0
    for val in arg:
        total += val
    return total 

Create a folder called tests/ and split the tests into multiple files. It is convention to ensure each file starts with test_ so all test runners will assume that Python file contains tests to be executed.

You can import any attributes of the script, such as classes, functions, and variables by using the built-in __import__() function. Instead of from my_sum import sum, you can write the following:

target = __import__("my_sum.py")
sum = target.sum

The benefit of using __import__() is that you don’t have to turn your project folder into a package, and you can specify the file name. This is also useful if your filename collides with any standard library packages. For example, math.py would collide with the math module.

How to Structure a Simple Test

• Create your inputs
• Execute the code being tested, capturing the output
• Compare the output with an expected result

test.py

import unittest

from my_sum import sum


class TestSum(unittest.TestCase):
    def test_list_int(self):
        """
        Test that it can sum a list of integers
        """
        data = [1, 2, 3]
        result = sum(data)
        self.assertEqual(result, 6)

if __name__ == '__main__':
    unittest.main()

This code example:

• Imports sum() from the my_sum package you created
• Defines a new test case class called TestSum, which inherits from unittest.TestCase
• Defines a test method, test_list_int(), to test a list of integers. The method test_list_int() will:
  • Declare a variable data with a list of numbers (1, 2, 3)
  • Assign the result of my_sum.sum(data) to a result variable
  • Assert that the value of result equals 6 by using the assertEqual() method on the unittest.TestCase class
• Defines a command-line entry point, which runs the unittest test-runner main()

How to Write Assertions

The last step of writing a test is to validate the output against a known response. This is known as an assertion.
There are some general best practices around how to write assertions:

• Make sure tests are repeatable and run your test multiple times to make sure it gives the same result every time
• Try and assert results that relate to your input data, such as checking that the result is the actual sum of values in the sum() example
  unittest comes with lots of methods to assert on the values, types, and existence of variables. Here are some of the most commonly used methods:

Method	Equivalent to
assertEqual(a, b)	a == b
assertTrue(x)	bool(x) is True
assertFalse(x)	bool(x) is False
assertIs(a, b)	assertIs(a, b)
assertIsNone(x)	x is None
assertIn(a, b)	a in b
assertIsInstance(a, b)	isinstance(a, b)

assertIs(), assertIsNone(), assertIn(), and assertIsInstance() all have opposite methods, named assertIsNot(), and so forth.

Side Effects

When you’re writing tests, it’s often not as simple as looking at the return value of a function. Often, executing a piece of code will alter other things in the environment, such as the attribute of a class, a file on the filesystem, or a value in a database. These are known as side effects and are an important part of testing.
If you find that the unit of code you want to test has lots of side effects, you might be breaking the Single Responsibility Principle.
This means the piece of code is doing too many things and would be better off being refactored.

Executing Test Runners

The Python application that executes your test code, checks the assertions, and gives you test results in your console is called the test runner. At the bottom of test.py, you added this small snippet of code:

if __name__ == '__main__':
    unittest.main()

This is a command line entry point. It means that if you execute the script alone by running python test.py at the command line, it will call unittest.main(). This executes the test runner by discovering all classes in this file that inherit from unittest.TestCase.
Another way is using the unittest command line. Try this:

$ python -m unittest test

This will execute the same test module (called test) via the command line. You can provide additional options to change the output. One of those is -v for verbose.

$ python -m unittest -v test
test_list_int (test.TestSum) ... ok

----------------------------------------------------------------------
Ran 1 tests in 0.000s

Instead of providing the name of a module containing tests, you can request an auto-discovery using the following:

 python -m unittest discover

This will search the current directory for any files named test*.py and attempt to test them.
Once you have multiple test files, as long as you follow the test*.py naming pattern, you can provide the name of the directory instead by using the -s flag and the name of the directory:

python -m unittest discover -s tests

unittest will run all tests in a single test plan and give you the results.
Lastly, if your source code is not in the directory root and contained in a subdirectory, for example in a folder called src/, you can tell unittest where to execute the tests so that it can import the modules correctly with the -t flag:

python -m unittest discover -s tests -t src

unittest will change to the src/ directory, scan for all test*.py files inside the the tests directory, and execute them.