Running C unit tests with pytest

In this post I will describe my approach to C unit testing using pytest. In particular, we get to see how to gracefully handle SIGSEGVs and prevent them from stopping the test runner abruptly. Furthermore, we shall try to write tests in a Pythonic way.

That’s probably what you might be asking right now. Why use a Python testing
framework to test C code? Why don’t just use C testing frameworks, like Google
Test, or Check. I can give you an answer with all the
reasons that led me to adopt pytest as the testing framework of choice
for one of my C projects, Austin. The first is that I spend most time
coding in Python these days and I have more familiarity with pytest than any
other testing framework. Secondly, whilst Austin is a C project, it actually
targets Python programs, so Python is already one of the testing dependencies
that ends up being installed in CI anyway. Hence, instead of spending time
learning an entire new testing framework, I could quickly write them in Python,
and leverage all the features of pytest, as well as all the packages that are
available for Python, should I ever need to. But these are not all the reasons
for adopting pytest for running C tests. If you keep reading you will discover a
few more that might convince you to use pytest for your C unit tests too!

Featured Content Ads

add advertising here

COGNAC Metaverse

Join World's first photorealistic metaverse, developed in India! GDWC Finalist.

https://thegdwc.com

Testing C code with Python would only make sense if it were easy to call native
code from the interpreter. Thankfully, the Python standard library comes with
the ctypes module that allows us to do just that! So let’s start
looking at some C code, for instance,

# file: fact.c

long fact(long n) {
    if (n < 1)
        return 1;
    return n * fact(n - 1);
}

which we want to compile as a shared object, e.g. with

gcc -shared -o fact.so fact.c

How do we test the fact function from Python? Easy peasy!

# file: fact.py

from ctypes import CDLL

libfact = CDLL("./fact.so")

assert libfact.fact(6) == 720
assert libfact.fact(0) == 1
assert libfact.fact(-42) == 1

Assuming we are in the directory where both fact.so and fact.py reside, we
can test the fact function inside fact.c simply with

Featured Content Ads

add advertising here

COGNAC Metaverse

Join World's first photorealistic metaverse, developed in India! GDWC Finalist.

https://thegdwc.com

If the test succeeds, the script’s return code will be 0.

Congratulations! You have now tested some C code with Python! 🎉

We are not here to just play around with bare asserts. I promised you the full
power of Python and pytest, so we can’t settle with just this simple
example. Let’s add pytest to our test dependencies and do this instead

# file: test_fact.py

from ctypes import CDLL

import pytest

@pytest.fixture
def libfact():
    yield CDLL("./fact.so")


def test_fact(libfact):
    assert libfact.fact(6) == 720
    assert libfact.fact(0) == 1
    assert libfact.fact(-42) == 1

Now run pytest to get

Featured Content Ads

add advertising here

COGNAC Metaverse

Join World's first photorealistic metaverse, developed in India! GDWC Finalist.

https://thegdwc.com

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.10.2, pytest-7.0.0, pluggy-1.0.0
rootdir: /tmp
collected 1 item

test_fact.py .                                                       [100%]

============================ 1 passed in 0.00s =============================

That’s some more informative output than what a plain Python test script would
give us! How about starting to leverage some of the other pytest features,
like parametrised tests? Let’s rewrite our test case like so

# file: test_fact.py

from ctypes import CDLL

import pytest


@pytest.fixture
def libfact():
    yield CDLL("./fact.so")


@pytest.mark.parametrize("n,e", [(6, 720), (0, 1), (-42, 1)])
def test_fact(libfact, n, e):
    assert libfact.fact(n) == e

Let’s run this again with pytest, this time with a more verbose output:

pytest -vv
=========================== test session starts ============================
platform linux -- Python 3.10.2, pytest-7.0.0, pluggy-1.0.0 -- /tmp/.venv/bin/python3.10
cachedir: .pytest_cache
rootdir: /tmp
collected 3 items

test_fact.py::test_fact[6-720] PASSED                                [ 33%]
test_fact.py::test_fact[0-1] PASSED                                  [ 66%]
test_fact.py::test_fact[-42-1] PASSED                                [100%]

============================ 3 passed in 0.01s =============================

Sweet! 🍯

Thus far we’ve got an idea of how to invoke C from Python and how to write some
simple tests that we can run with pytest while also leveraging features like
fixtures and parametrised tests. Let us now step this up a notch and consider
the organisation of sources within an actual C project, for instance

my-c-project/
├── docs/
├── src/    <- All *.c and *.h sources, perhaps organised into sub-folders
├── tests/  <- Our test sources, obviously!
├── ChangeLog
├── configure.ac
├── LICENCE
├── Makefile.am
├── README
...

In the previous example, we built the shared object fact.so by hand, but in a
CI environment we would probably want to automate that step too. What should we
use for that? A bash script? A makefile? Python, of course! What else?!? 😀

Let's make our sample C sources slightly more interesting. For example, we could
borrow a few parts of the cache.c and cache.h sources from Austin,
which implement a simple LRU cache. This is part of the spec

// file: src/cache.h

#ifndef CACHE_H
#define CACHE_H

#include 
#include 

typedef uintptr_t key_dt;
typedef void *value_t;

typedef struct queue_item_t
{
    struct queue_item_t *prev, *next;
    key_dt key;
    value_t value; // Takes ownership of a free-able object
} queue_item_t;

typedef struct queue_t
{
    unsigned count;
    unsigned capacity;
    queue_item_t *front, *rear;
    void (*deallocator)(value_t);
} queue_t;

queue_item_t *
queue_item_new(value_t, key_dt);

void
queue_item__destroy(queue_item_t *, void (*)(value_t));

queue_t *
queue_new(int, void (*)(value_t));

int
queue__is_full(queue_t *);

int
queue__is_empty(queue_t *);

value_t
queue__dequeue(queue_t *);

queue_item_t *
queue__enqueue(queue_t *, value_t, key_dt);

void
queue__destroy(queue_t *);

and this is the corresponding part of the implementation

// file: src/cache.c

#include 
#include 

#include "cache.h"

#define isvalid(x) ((x) != NULL)

// ----------------------------------------------------------------------------
queue_item_t *
queue_item_new(value_t value, key_dt key)
{
    queue_item_t *item = (queue_item_t *)calloc(1, sizeof(queue_item_t));

    item->value = value;
    item->key = key;

    return item;
}

// ----------------------------------------------------------------------------
void
queue_item__destroy(queue_item_t *self, void (*deallocator)(value_t))
{
    if (!isvalid(self))
        return;

    deallocator(self->value);

    free(self);
}

// ----------------------------------------------------------------------------
queue_t *
queue_new(int capacity, void (*deallocator)(value_t))
{
    queue_t *queue = (queue_t *)calloc(1, sizeof(queue_t));

    queue->capacity = capacity;
    queue->deallocator = deallocator;

    return queue;
}

// ----------------------------------------------------------------------------
int
queue__is_full(queue_t *queue)
{
    return queue->count == queue->capacity;
}

// ----------------------------------------------------------------------------
int
queue__is_empty(queue_t *queue)
{
    return queue->rear == NULL;
}

// ----------------------------------------------------------------------------
value_t
queue__dequeue(queue_t *queue)
{
    if (queue__is_empty(queue))
        return NULL;

    if (queue->front == queue->rear)
        queue->front = NULL;

    queue_item_t *temp = queue->rear;
    queue->rear = queue->rear->prev;

    if (queue->rear)
        queue->rear->next = NULL;

    void *value = temp->value;
    free(temp);

    queue->count--;

    return value;
}

// ----------------------------------------------------------------------------
queue_item_t *
queue__enqueue(queue_t *self, value_t value, key_dt key)
{
    if (queue__is_full(self))
        return NULL;

    queue_item_t *temp = queue_item_new(value, key);
    temp->next = self->front;

    if (queue__is_empty(self))
        self->rear = self->front = temp;
    else
    {
        self->front->prev = temp;
        self->front = temp;
    }

    self->count++;

    return temp;
}

// ----------------------------------------------------------------------------
void
queue__destroy(queue_t *self)
{
    if (!isvalid(self))
        return;

    queue_item_t *next = NULL;
    for (queue_item_t *item = self->front; isvalid(item); item = next)
    {
        next = item->next;
        queue_item__destroy(item, self->deallocator);
    }

    free(self);
}

It's quite a fair bit of code; however, we are not interested in how the data
structures are implemented, but rather to what it actually implements. This
already gives us plenty to play with.

The important detail here is that our C application has a component implemented
in cache.c and we want to unit-test it. Before we can run any actual tests, we
need to build a binary object that we can invoke from Python. So let's put this
code in tests/cunit/__init__.py

from pathlib import Path
from subprocess import PIPE, STDOUT, run

HERE = Path(__file__).resolve().parent
TEST = HERE.parent
ROOT = TEST.parent
SRC = ROOT / "src"


class CompilationError(Exception):
    pass


def compile(source: Path, cflags=[], ldadd=[]):
    binary = source.with_suffix(".so")

    result = run(
        ["gcc", "-shared", *cflags, "-o", str(binary), str(source), *ldadd],
        stdout=PIPE,
        stderr=STDOUT,
        cwd=SRC,
    )

    if result.returncode == 0:
        return

    raise CompilationError(result.stdout.decode())

This simply defines the compile utility that allows us to invoke gcc to
compile a source and generate the .so shared object. We can then use it in our
test source this way

from ctypes import CDLL

import pytest
from tests.cunit import SRC, compile

C = CDLL("libc.so.6")


@pytest.fixture
def cache():
    source = SRC / "cache.c"
    compile(source)
    yield CDLL(str(source.with_suffix(".so")))


def test_cache(cache):
    lru_cache = cache.lru_cache_new(10, C.free)
    assert lru_cache
    cache.lru_cache__destroy(lru_cache)

At this point, your project folder should have the following structure

my-c-project/
...
├── src/
|   ├── cache.c
|   └── cache.h
├── tests/
|   ├── cunit/
|   |   ├── __init__.py
|   |   └── test_cache.py
|   └── __init__.py
...

and when you run pytest again, this time the C source would be compiled at
runtime using gcc. The tests then run as before, which should produce the same
output we saw earlier.

If you're still with me, then things are probably looking interesting to you
too. So let's test a bit more of the functions exported by the caching
component. Let's make a test case for the queue_item_t and queue_t objects,
like so

from ctypes import CDLL

import pytest
from tests.cunit import SRC, compile

C = CDLL("libc.so.6")


@pytest.fixture
def cache():
    source = SRC / "cache.c"
    compile(source)
    yield CDLL(str(source.with_suffix(".so")))


NULL = 0


def test_queue_item(cache):
    value = 1
    queue_item = cache.queue_item_new(value, 42)
    assert queue_item

    cache.queue_item__destroy(queue_item, C.free)


@pytest.mark.parametrize("qsize", [0, 10, 100, 1000])
def test_queue(cache, qsize):
    q = cache.queue_new(qsize, C.free)

    assert cache.queue__is_empty(q)
    assert qsize == 0 or not cache.queue__is_full(q)

    assert cache.queue__dequeue(q) is NULL

    values = [C.malloc(16) for _ in range(qsize)]
    assert all(values)

    for k, v in enumerate(values):
        assert cache.queue__enqueue(q, v, k)

    assert qsize == 0 or not cache.queue__is_empty(q)
    assert cache.queue__is_full(q)
    assert cache.queue__enqueue(q, 42, 42) is NULL

    assert values == [cache.queue__dequeue(q) for _ in range(qsize)]

Let's run the new tests with pytest -vv and

=============================== test session starts ===============================
platform linux -- Python 3.10.2, pytest-7.0.0, pluggy-1.0.0 -- /home/gabriele/Projects/cunit/.venv/bin/python3.10
cachedir: .pytest_cache
rootdir: /home/gabriele/Projects/cunit
collected 5 items

tests/cunit/test_cache.py::test_queue_item Fatal Python error: Segmentation fault

Current thread 0x00007f4016e4f740 (most recent call first):
  File "/home/gabriele/Projects/cunit/tests/cunit/test_cache.py", line 24 in test_queue_item
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/python.py", line 192 in pytest_pyfunc_call
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/python.py", line 1718 in runtest
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 168 in pytest_runtest_call
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 261 in 
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 340 in from_call
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 260 in call_runtest_hook
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 221 in call_and_report
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 132 in runtestprotocol
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/main.py", line 347 in pytest_runtestloop
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/main.py", line 322 in _main
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/main.py", line 268 in wrap_session
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 165 in main
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 188 in console_main
  File "/home/gabriele/Projects/cunit/.venv/bin/pytest", line 8 in 
[1]    337951 segmentation fault  .venv/bin/pytest -vv

Wait, what?! Where are our tests? A segmentation fault?!? Where did that come
from? Well, there goes all this pytest hype! 😡

Now, do you think I would have written this post if this was really the end of
the story?

If you want to figure out for yourself where the problem is, pause here. When
you are ready to carry on, change line 20 to

and now the tests will all be happy! However, we really want to avoid crashing
the pytest process when we run into a segmentation fault, which is not so rare
when running arbitrary C code. Not only that, but we would like to get some
useful information, like a traceback, that could give us insight as to where the
problem might be! One of the many strengths of pytest is its extensive
configuration API. How do we use it to not crash the test runner?
The idea is to spawn another pytest process that runs just a test. Now, if
that test causes a segmentation fault, the parent process will keep running the
other tests. Let's put this into tests/cunit/conftest.py

# file: tests/cunit/conftest.py

import os
import sys
from subprocess import PIPE, run
from types import FunctionType


class SegmentationFault(Exception):
    pass


class CUnitTestFailure(Exception):
    pass


def pytest_pycollect_makeitem(collector, name, obj):
    if (
        not os.getenv("PYTEST_CUNIT")
        and isinstance(obj, FunctionType)
        and name.startswith("test_")
    ):
        obj.__cunit__ = (str(collector.fspath), name)


def cunit(module: str, name: str, full_name: str):
    def _(*_, **__):
        test = f"{module}::{name}"
        env = os.environ.copy()
        env["PYTEST_CUNIT"] = full_name

        result = run([sys.argv[0], "-svv", test], stdout=PIPE, stderr=PIPE, env=env)

        if result.returncode == 0:
            return

        raise CUnitTestFailure("n" + result.stdout.decode())

    return _


def pytest_collection_modifyitems(session, config, items) -> None:
    if test_name := os.getenv("PYTEST_CUNIT"):
        # We are inside the sandbox process. We select the only test we care
        items[:] = [_ for _ in items if _.name == test_name]
        return

    for item in items:
        if hasattr(item._obj, "__cunit__"):
            item._obj = cunit(*item._obj.__cunit__, full_name=item.name)

Let's re-run our broken test suite and see what happens this time:

================================ test session starts =================================
platform linux -- Python 3.10.2, pytest-7.0.0, pluggy-1.0.0 -- /home/gabriele/Projects/cunit/.venv/bin/python3.10
cachedir: .pytest_cache
rootdir: /home/gabriele/Projects/cunit
collected 5 items

tests/cunit/test_cache.py::test_queue_item <- tests/cunit/conftest.py FAILED   [ 20%]
tests/cunit/test_cache.py::test_queue[0] <- tests/cunit/conftest.py PASSED     [ 40%]
tests/cunit/test_cache.py::test_queue[10] <- tests/cunit/conftest.py PASSED    [ 60%]
tests/cunit/test_cache.py::test_queue[100] <- tests/cunit/conftest.py PASSED   [ 80%]
tests/cunit/test_cache.py::test_queue[1000] <- tests/cunit/conftest.py PASSED  [100%]

====================================== FAILURES ======================================
__________________________________ test_queue_item ___________________________________

_ = ()
__ = {'cache': }
test = '/home/gabriele/Projects/cunit/tests/cunit/test_cache.py::test_queue_item'
env = {'ANDROID_HOME': '/home/gabriele/.android/sdk', 'COLORTERM': 'truecolor', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'DEFAULTS_PATH': '/usr/share/gconf/ubuntu.default.path', ...}
result = CompletedProcess(args=['.venv/bin/pytest', '-svv', '/home/gabriele/Projects/cunit/tests/cunit/test_cache.py::test_queu...__init__.py", line 188 in console_mainn  File "/home/gabriele/Projects/cunit/.venv/bin/pytest", line 8 in n')

    def _(*_, **__):
        test = f"{module}::{name}"
        env = os.environ.copy()
        env["PYTEST_CUNIT"] = full_name

        result = run([sys.argv[0], "-svv", test], stdout=PIPE, stderr=PIPE, env=env)

        if result.returncode == 0:
            return

>       raise CUnitTestFailure("n" + result.stdout.decode())
E       tests.cunit.conftest.CUnitTestFailure: 
E       ============================= test session starts ==============================
E       platform linux -- Python 3.10.2, pytest-7.0.0, pluggy-1.0.0 -- /home/gabriele/Projects/cunit/.venv/bin/python3.10
E       cachedir: .pytest_cache
E       rootdir: /home/gabriele/Projects/cunit
E       collecting ... collected 1 item
E       
E       tests/cunit/test_cache.py::test_queue_item

tests/cunit/conftest.py:49: CUnitTestFailure
============================== short test summary info ===============================
FAILED tests/cunit/test_cache.py::test_queue_item - tests.cunit.conftest.CUnitTestF...
============================ 1 failed, 4 passed in 1.21s =============================

How do we like this better? Now the first test fails with the segmentation
fault, but the rest of the test suite still runs and we can see the reason of
the failure for the first test in the report, i.e. the segmentation fault.

But what is the conftest.py code actually doing? Let's have a look. The
pytest_pycollect_makeitem hook gets invoked when the tests inside
testscunit are being collected by pytest. At this stage we "mark" them as C
unit tests by giving each collected item the __cunit__ attribute. The value
is a tuple containing the information of where the item came from
(collection.fspath is the path of the module that defined the test, e.g.
tests/cunit/test_cache.py) and the test name (e.g. test_queue_item). In our
case we only care about items that are of FunctionType type and that start
with test_. The environment variable PYTEST_CUNIT is used to detect whether
we are running in the parent pytest process or the "sandbox" child. In the
latter case we don't care of marking tests because we know exactly what we want
to run.

Once all the tests have been collected, we use the
pytest_collection_modifyitems hook to actually modify the tests that we
previously marked as C unit tests. Again, the behaviour depends on whether we
are in the parent pytest process, or in the sandbox. If PYTEST_CUNIT is set,
that's the signal that we are in the child pytest process. The value, as we
shall see shortly, contains the information needed to pick the test that we want
to run. So we use it to modify the list items to just the test that matches
the information stored in PYTEST_CUNIT. In the parent pytest process we
actually modify what the items that we marked as C unit tests do. Obviously, we
don't want them to run the actual test, but rather a new instance of pytest
that will then run the test on behalf of the parent process. The magic happens
inside the "decorator" cunit, which we use to build a closure around the test
that we want to run. As you can see, it returns a function (with a bit of a
funny and unusual signature) which, when called, will set the PYTEST_CUNIT
variable with the full name of the test (this is to support parametrised tests)
and then run sys.argv[0] (which should be pytest if we run the test suite
with pytest), followed by the switches -svv and the path of the module that
provides the test we are wrapping around. This way, when the process terminates,
either because the test passed or something really bad happened, we can inspect
the return code and the streams, and act accordingly.

So now we have a test runner that can handle segmentation faults gracefully but
still doesn't tell us where they actually happened. Can we get some more
detailed information in the output? When a Python test fails we get a nice
traceback that tells us where things went wrong. Can we do the same with C unit
tests? The answer is yes, provided we collect core dumps while tests run. If
you are running on Ubuntu, you can do ulimit -c unlimited and a core dump
will be generated in the working directory every time a segmentation fault
occurs. We can then run gdb in batch mode to print a nice traceback that will
hopefully help us investigate the problem. S

Knowasiak