Unit testing C code with CMocka

With mocking and code coverage tools inside!

In this article we are going to see why, and mostly how, test code written in C using modern tools.

Why testing C code?

Tests are good. Test Driven Development is even better. Writting tests along with source code allows to verify that what you wrote is behaving as you intented. Unit testing is very popular in languages such as Python, Ruby or even Java, from where the JUnit framework inspired a lot of testing libraries in other languages (unittest in Python for example).

However testing code written in C (and to some extent in C++) is a different kettle of fish. Indeed, these high-performance and embedded systems languages seems to be less testing-friendly than other higher-level ones. In C we often do very low level tasks, such as reading and writing device registers, or using Linux system calls. It seems hard to test this code that is very low level. Hard… but not impossible!

Today we are going to see how to test C code, because even if write inside registers and stuff most of the time, we still want to be sure that we read and write values in the good ones!

CMocka, a 100% C test framework (and more)

When I was looking for a C test framework, a good portion of people seemed to use the Google Test framework (written in C++) and to call C functions from C++ test code. I would rather use C only for my testing and not rely on a C++ framework, so I digged a little bit deeper.

And I found CMocka. This one is written in pure C and allows, in addition to write JUnit-like tests, to mock functions in order to verify that they are correctly called by other of our functions. This is its main advantage against other testing frameworks, hence the name CMocka. This is why you want to use this library instead of writing a main with some calls to assert.

It has a good Doxygen documentation if you want to explore the available APIs quickly.

Assert functions

Instead of using the classic assert() call from well-known assert.h, CMocka comes with a small set of practical assert functions. Among them, the ones that I use the most are:

void assert_int_equal(int a, int b);
void assert_string_equal(const char *a, const char *b);
void assert_memory_equal(const void *a, const void *b, size_t size);

Other functions seem useful too, such as assert_null and assert_in_range. You can find the full list of assert variants on the dedicated assert module documentation

Declaring tests

Test declaration is to be done in two steps. First we declare and implement the test function that must have the following type signature:

void test_do_something(void **state) {
    (void) state; /* unused */
}

The void **state pointer is used when you specify setup and teardown functions to the test, which allow to initialize and deconstruct a state for each test. Useful in order to avoid code duplication if you have to do these common steps for multiple independant tests. I do not use it at this moment so I add this little construction in order to avoid GCC to raise warnings about unused variables.

Function mocking with CMocka

Now that we have seen the basics of CMocka, lets dive in function mocking.

Here we come to the most interresting feature of CMocka: the possibility to create a function mock, to check that this function was called with the correct parameters, and to return whatever value you want. This feature is very useful for at least two test cases:

  • System call mocking, for simulating system failures
  • Custom function mocking, for decoupling tests

System call mocking with wrap

This allows us to verify that a system call like open() was correctly called, without really calling the system's open() function. So we can mock a lot of system calls like this, typically open(), write(), read() and close() functions. Once the mock is declared we can check that it was called with the good arguments and return success (0) or error (-1) in order to simulate system failures.

This even allows us to check that a function was not called if we did not expect it. If you call a mock without declaring that you expect it to be called, then you will get an error. This is useful if you want to check that you do not perform a read() if the open() function call has failed.

For this system call mocking we will use a GCC linker option named --wrap=.... Since you are more likely to call gcc instead of ld in your compilation process, you can tell gcc to pass the --wrap option to ld by passing the --Wl,--wrap=... argument to GCC.

The --wrap=open notation asks GCC to make every call to the open functon to reach the __wrap_open function instead. If for whatever reason you need to use the real open function after this option is passed, then you can call the __real_open function (after declaring its prototype).

Custom function mocking using weak symbol

We can even split our own tests in order to test only one function at a time, hence test decoupling. We can verify that a function correctly calls another function that we already tested before. If we do not perform this tests splitting, then we are not doing unit tests anymore but functionnal tests, since you will test a function that will call a function that will... You get it.

This splitting really depends on how you want to test your code and on your own taste. Typically, I use wrapping in order to test a function that uses system calls, and then mock this function inside another test unit in order to see that it is correctly called by other functions.

Here --wrap cannot work if the function you want to replace is in the same source file. Indeed, linking to this function is already done, because the compiler has found the function inside the same file. It does not mark it as "unresolved", which is a neccesary condition in order to make --wrap ld option work.

We then have to use another trick in order to redefine the function in order to mock it. For this, GCC provides us with a new bypass: weak symbols. By default, when we declare the implementation of a function in a C file, then the function symbol is defined as a strong symbol. Any attempt to redefine this strong symbol will lead to the well know error you may know: the multiple definiton of error.

libhello.a(hello.c.o): In function `open_i2c':
lib/hello.c:15: multiple definition of `open_i2c'
tests_do_something.c.o:tests_do_something.c:12: first defined here

Weak symbols, however, can be redefined. This is exactly what we need in order to test that our high-level function correctly calls our low-level function, while these to functions being implemented in the same source file.

There are two ways of declaring a weak symbol:

  • By passing an argument to GCC, telling it to export the symbol of this function as a weak symbol.
  • By putting a __attribute__((weak)) annotation before the function implementation.

First option is useful if you cannot modify source code of an exernal function you call in your program (for example from a third party library). However I prefer the second option that is quicker to set up, with only one line to add before the function declaration.

We may not be confortable with the idea that our users can now redefine our internal functions if we export them as weak symbols. A workaround I am using right now is to export functions as weak symbols only during a debug build, which is needed in order to build and run tests in my setup:

#ifdef DEBUG
#define WEAK_FOR_DEBUG __attr__((weak))
#else
#define WEAK_FOR_DEBUG

And we can declare our functions as weak only during debug build like this:

WEAK_FOR_DEBUG
int myfunction(int a, int b)
{
    return a + b;
}

Maybe I should use a TESTING define instead of DEBUG and define it only when building for tests but not for debugging.

Drawbacks: lots of executables

The downside of both of these methods is that once a function is redefined, it will be in the whole test executable. If you want to test a function and the functions it calls then you will need two different executables, and two different test files.

If we correctly test our program, then we are starting to see tens of test executables, with different wrapping and weak symbol redefinitions inside them. Hopefully we can rely on CMake in order to build and launch all of these test executables.

Building and running tests with CMake

CMake is the reference build tool for C and C++ projects. A lot of open-source projects already have dumped Makefiles for the CMake build system, offering a new world of possibilities. Here we are going to use it in order to compile our library and its unit tests which need different executables for each mocking/wrapping configuration.

CMake provides the ctest tool which, once called, will launch actions specified with the ADD_TEST function. It even allows us to run these tests in parallel on multiple cores, may this be needed. CMake coneniently adds a make target named test so we can call ctest from the makefile.

Here is the CMakeLists.txt that I use for building my tests.

# Find cmocka
INCLUDE_DIRECTORIES(${CMAKE_SOURCE_DIR}/../test/cmocka/include)
FIND_LIBRARY(CMOCKA_LIBRARY cmocka HINTS ../../test/cmocka/build/src)

# Include code coverage
set(CMAKE_MODULE_PATH ${CMAKE_SOURCE_DIR}/scripts/cmake)
INCLUDE(CodeCoverage)

# List of tests and their flags
LIST(APPEND tests_names "tests_without_flags")
LIST(APPEND tests_flags " ")

LIST(APPEND tests_names "tests_processes")
LIST(APPEND tests_flags "-Wl,--wrap,popen -Wl,--wrap,fgets -Wl,--wrap,pclose")

LIST(APPEND tests_names "tests_open_i2c")
LIST(APPEND tests_flags "-Wl,--wrap,open -Wl,--wrap,ioctl -Wl,--wrap,close")

LIST(APPEND tests_names "tests_do_something")
LIST(APPEND tests_flags "-Wl,--wrap,close")

# Declare all tests targets
LIST(LENGTH tests_names count)
MATH(EXPR count "${count} - 1")
FOREACH(i RANGE ${count})
    LIST(GET tests_names ${i} test_name)
    LIST(GET tests_flags ${i} test_flags)
    ADD_EXECUTABLE(${test_name} ${test_name}.c)
    TARGET_LINK_LIBRARIES(
        ${test_name}
        ${CMOCKA_LIBRARY}
        -fprofile-arcs -ftest-coverage
        mylib
    )
    IF(test_flags STREQUAL " ")
    ELSE()
        TARGET_LINK_LIBRARIES(
            ${test_name}
            ${test_flags}
        )
    ENDIF()
    ADD_TEST(${test_name} ${test_name})
ENDFOREACH()

# Coverage settings
SET(
    COVERAGE_EXCLUDES
    include/
)

SETUP_TARGET_FOR_COVERAGE(
    NAME test_coverage
    EXECUTABLE ctest
    DEPENDENCIES mylib
)

It is not perfect:

  • I use relative path for finding CMocka's include file and its dynamic library. I guess I should write and use a FindCMocka.cmake script.
  • I use a double list in order to create a kind of a dictionary with additional but optional extra flags to pass to the compiler. Maybe there is something more idiomatic, but this works.

As you can see I am using the CodeCoverage.cmake extension that allows to run a set of tests with code coverage support seamlessly. I have only tweaked it in order to activate branch coverage support.

So, lets see the results of this code coverage tool!

Code coverage with gcov and lcov

In order to activate code coverage support within GCC, we have to pass it the following flags:

-fprofile-arcs -ftest-coverage

When the program is run with these options enabled, then is will generate a code coverage report file which is a binary file. Lcov can help us translate this binary file into human-readable statistics.

Issues with gcov and system calls mocking

There is a big issue if you mock system calls and activate code coverage: once these compilation flags added, your program will try to write the code coverage report to a binary file when its execution has ended. However in order to write into this file it needs the open() function. If you have already mocked it, then the code coverage tool is going to have big trouble trying to write to its report file.

We can detect this problem by launching the test executable and by looking at its exit code. Sometimes all tests are green but the test program will return 255 (which is -1 mapped onto a uint8_t). This behavior is typical of a write error of code coverage results, that is called once your program has ended. If you observe this then there are good chances that you mocked a system call that gcov needed for its code coverage report.

The solution one proposed me on StackOverflow is to look if our open() mock is called on a file that looks like a code coverage report file. If it is the case then we really open the file in order for gcov to write to it. If the file does not look like a code coverage file then we can mock it for our testing needs.

int __real_open(const char *path, int flags, int mode);
int __wrap_open(const char *path, int flags, int mode)
{
    if (strlen(path) > 5 && !strcmp(path + strlen(path) - 5, ".gcda"))
        return __real_open(path, flags, mode);
    printf("hello from __wrap_open\n");
    return -1;
}

Complete example

Let's see a full example with wrapping of a system call and mocking of an internal function.

Source code we want to test

Here are the functions we want to test using CMocka.

/* hello.h */

#ifndef HELLO_H
#define HELLO_H

#include <stdint.h>

#define I2C_SLAVE_FORCE 0x0706

int open_i2c(uint8_t i2c_addr);
int do_something();

#endif /* HELLO_H */

And below is the trivial implementation.

/* hello.c */

#include "hello.h"

#include <stdio.h>

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

#define PATH_BUFFER_SIZE 32


__attribute__((weak))
int open_i2c(uint8_t i2c_bus)
{
    char path[PATH_BUFFER_SIZE];
    int fd;

    snprintf(path, PATH_BUFFER_SIZE, "/dev/i2c-%d", i2c_bus);
    fd = open(path, O_RDWR);

    if (fd < 0)
        return -1;

    return fd;
}

int do_something()
{
    int fd = open_i2c(42);

    if (fd < 0)
        return -1;

    return close(fd);
}

Testing the low-level open_i2c() function

First we want to test that the open_i2c() function we wrote opens the correct file given its i2c-bus argument. For this we create a new test file with the following code:

/* test_open_i2c.c */

#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>

#include <stdio.h>
#include <string.h>

#include "cmocka.h"
#include "hello.h"

/* redefinitons/wrapping */

int __real_open(const char *path, int flags, int mode);
int __wrap_open(const char *path, int flags, int mode)
{
    if (strlen(path) > 5 && !strcmp(path + strlen(path) - 5, ".gcda"))
            return __real_open(path, flags, mode);

    check_expected(path);
    return mock();
}

/* tests */

void test_open_i2c_failure(void **state)
{
    (void) state; /* unused */
    int ret;

    expect_string(__wrap_open, path, "/dev/i2c-99");
    will_return(__wrap_open, -1);

    ret = open_i2c(99);
    assert_int_equal(-1, ret);
}

void test_open_i2c_success(void **state)
{
    (void) state; /* unused */
    int ret;

    expect_string(__wrap_open, path, "/dev/i2c-99");
    will_return(__wrap_open, 42);

    ret = open_i2c(99);
    assert_int_equal(42, ret);
}


const struct CMUnitTest open_i2c_tests[] = {
    cmocka_unit_test(test_open_i2c_failure),
    cmocka_unit_test(test_open_i2c_success),
};


int main(void)
{
    return cmocka_run_group_tests(open_i2c_tests, NULL, NULL);
}

Can you see which one of the two GCC hacks we are going to use in order to redefine the open() system call in these tests? Since this function is undefined in our code we can freely use the -Wl,--wrap=open option! Make your eyes sharp and now read the two tests carefuly. See how we tell our open() mock to check its arguments and to return the value we want?

Testing the higher-level do_something() function

For the second test unit we want this time to check that our do_something() function is correctly calling our open_i2c() function and the system's close() function.

/* tests_do_something.c */

#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>

#include <stdio.h>

#include "cmocka.h"
#include "hello.h"

/* redefinitons/wrapping */

int open_i2c(uint8_t i2c_addr)
{
    check_expected(i2c_addr);
    return mock();
}

int __wrap_close(int fd)
{
    check_expected(fd);
    return mock();
}

/* tests */

void test_do_something_failure(void **state)
{
    (void) state; /* unused */
    int ret;

    expect_value(open_i2c, i2c_addr, 42);
    will_return(open_i2c, -1);

    ret = do_something();
    assert_int_equal(-1, ret);
}

void test_do_something_success(void **state)
{
    (void) state; /* unused */
    int ret;

    expect_value(open_i2c, i2c_addr, 42);
    will_return(open_i2c, 42);

    expect_value(__wrap_close, fd, 42);
    will_return(__wrap_close, 0);

    ret = do_something();
    assert_int_equal(0, ret);
}


const struct CMUnitTest do_something_tests[] = {
    cmocka_unit_test(test_do_something_failure),
    cmocka_unit_test(test_do_something_success),
};


int main(void)
{
    return cmocka_run_group_tests(do_something_tests, NULL, NULL);
}

This time we are redefining our open_i2c() function using the weak symbol redefinition, and still wrapping the external close() system function. See that using this trick we are not calling the real open_i2c() function, and thus we do not have to rewrap calls to open() again!

Finally, note that in the failure test case we do not specify that we expect __wrap_close to be called. If you remove the return in the C code and call close if open_i2c has failed, then you will get a CMocka error because you did not tell it what to verify for __wrap_close in this test case.

Bonus: code coverage results

Using the CMake file I presented you, CMake adds a test_coverage target to the generated makefile. It will clean the code coverage results, launch all the tests using ctest and then generate code coverage results as a set of HTML files.

LCOV code coverage result

Yay, this is a 100% code coverage report!

You can see that with these tests we have a full coverage of our code and all branches, since we wrote tests for expected failures too.

Conclusion

You now have no excuses to not test your C code.

Sometimes the number of wraps and mocks to test a single function can be prohibitive. One first step towards better code quality is to write tests that do not need a lot of mocks/wraps, typically algorithm-based functions such as sorting, etc. so you can test their behavior and border cases.

Testing system-intensive functions is possible using wrapping but can become cumbersome if you are using a lot of different system calls. But again not impossible (and then you are a lot more confident in your code for refactoring, such as function splitting, constants renaming, …).

I hope to see more tests in C-based projects from now! (but if you can, just go with Rust instead of C. This is hard for me in my current job where C is everywhere, but Rust should be used for new projects and has integrated unit tests support!)

Une question ou remarque ? N'hésitez pas à me contacter en envoyant un mail à microjoe, suivi d'un arobase, puis encore microjoe et enfin un « point org ».