Introduction

Tests are an essential element to check if your written components in coala really do work like they should. Even when you think “I really looked over my code, no need for tests” you are wrong! Bugs introduced when not writing tests are often the most horrible ones, they have the characteristic to be undiscoverable (or only discoverable after dozens of hours of searching). Try to test as much as possible! The more tests you write the more you can be sure you did everything correct. Especially if someone else modifies your component, he can be sure with your tests that he doesn’t introduce a bug.

Actually Writing a Test

So how do you implement a test in coala? First up, tests are placed into the bears/tests (if you want to write a test for a bear) or tests (if you test a component written for the coalib) directory. They are also written in python (version 3) and get automatically executed by running:

$ py.test

There’s only one constraint: The name of the test file has to end with Test.py (for example MyCustomTest.py, but not MyCustomTestSuite.py).

Note

Often you don’t want to run all available tests. To run your specific one, type (in the coala root folder):

$ py.test -k <your-test>

You can even give partial names or queries like “not MyCustomTest” to not run a specific test. More information can be got with py.test -h

Coming to the test file structure. Every test script starts with your system imports, i.e.:

# Imports the python 're' package for regex processing.
import re
import unittest
# ...

As said before, in the next line your own imports follow. So just import what you need from coala:

import coalib.your_component.component1
import coalib.your_component.component2
# ...

Note

You can use system imports here also, but the coala codestyle suggests to place them above the three setup lines, like before.

Then the actual test suite class follows, that contains the tests. Each test suite is made up of test cases, where the test suite checks the overall functionality of your component by invoking each test case.

The basic declaration for a test suite class is as follows:

class YourComponentTest(unittest.TestCase):
    # Your test cases.
    pass

You should derive your test suite from unittest.TestCase to have access to the setUp() and tearDown() functions (covered in section below: ``setUp()`` and ``tearDown()``) and also to the assertion functions.

Now to the test cases: To implement a test case, just declare a class member function without parameters, starting with test_. Easy, isn’t it?

class YourComponentTest(unittest.TestCase):
    # Tests somethin'.
    def test_case1(self):
        pass

    # Doesn't test, this is just a member function, since the function name
    # does not start with 'test_'.
    def not_testing(self):
        pass

But how do you actually test if your component is correct? For that purpose you have asserts. Asserts check whether a condition is fulfilled and pass the result to the overall test-suite-invoking-instance, that manages all tests in coala. The result is processed and you get a message if something went wrong in your test.

Available assert functions are listed in the section Assertions below.

So an example test that succeeds would be:

# The sys import and setup is not needed here because this example doesn't
# use coala components.
import unittest


class YourComponentTest(unittest.TestCase):
    # Tests somethin'.
    def test_case1(self):
        # Does '1' equal '1'? Interestingly it does... mysterious...
        self.assertEqual(1, 1)
        # Hm yeah, True is True.
        self.assertTrue(True)

Note

Tests in coala are evaluated against their coverage, means how many statements will be executed from your component when invoking your test cases. A branch coverage of 100% is needed for any commit in order to be pushed to master - please ask us on gitter if you need help raising your coverage!

The branch coverage can be measured locally with the py.test --cov command.

See also

Module Executing Tests
Documentation of running Tests with coverage

As our coverage is measured across builds against several python versions (we need version specific branches here and there) you will not get the full coverage locally! Simply make a pull request to get the coverage measured automatically.

If some code is untestable, you need to mark your component code with # pragma: no cover. Important: Provide a reason why your code is untestable. Code coverage is measured using python 3.3 and 3.4 on linux.

# Reason why this function is untestable.
def untestable_func(): # pragma: no cover
    # Untestable code.
    pass

setUp() and tearDown()

Often you reuse components or need to make an inital setup for your tests. For that purpose the function setUp() exists. Just declare it inside your test suite and it is invoked automatically once at test suite startup:

class YourComponentTest(unittest.TestCase):
    def setUp(self):
        # Your initialization of constants, operating system API calls etc.
        pass

The opposite from this is the tearDown() function. It gets invoked when the test suite finished running all test cases. Declare it like setUp() before:

class YourComponentTest(unittest.TestCase):
    def tearDown(self):
        # Deinitialization, release calls etc.
        pass

Assertions

Here follows a list of all available assertion functions supported when inheriting from unittest.TestCase:

  • assertEqual(a, b)

Checks whether expression a equals expression b.

  • assertNotEqual(a, b)

Checks whether expression a not equals expression b.

  • assertTrue(a)

Checks whether expression a is True.

  • assertFalse(a)

Checks whether expression a is False.

  • assertIs(a, b)

Checks whether expression a is b.

  • assertIsNot(a, b)

Checks whether expression a is not b.

  • assertIsNone(a)

Checks whether expression a is None.

  • assertIsNotNone(a)

Checks whether expression a is not None.

  • assertIn(a, list)

Checks whether expression a is an element inside list.

  • assertNotIn(a, list)

Checks whether expression a is not an element inside list.

  • assertIsInstance(a, type)

Checks whether expression a is of type type.

  • assertNotIsInstance(a, type)

Checks whether expression a is not of type type.

  • assertRaises(error, function, *args, **kwargs)

Checks whether function throws the specific error. When calling this assert it invokes the function with the specified *args and **kwargs.

If you want more information about the python unittest-module, refer to the official documentation and for asserts the subsection [assert-methods] (https://docs.python.org/3/library/unittest.html#assert-methods).

Kickstart

This section contains a concluding and simple example that you can use as a kickstart for test-writing.

Put the code under the desired folder inside tests or bears/tests, modify it to let it test your stuff and run the test from the coala root folder py.test.

# Import here your needed system components.
import sys
import unittest

sys.path.insert(0, ".")
# Import here your needed coala components.


# Your test unit. The name of this class is displayed in the test
# evaluation.
class YourTest(unittest.TestCase):
    def setUp(self):
        # Here you can set up your stuff. For example constant values,
        # initializations etc.
        pass

    def tearDown(self):
        # Here you clean up your stuff initialized in setUp(). For example
        # deleting arrays, call operating system API etc.
        pass

    def test_case1(self):
        # A test method. Put your test code here.
        pass