How to write a simple pytest hook function

A conftest plugin for retrying failed tests

AI-generated image

Introduction 🧾

Hooks are functions that pytest provides which are triggered in a specific order throughout a test’s different phases (e.g., setup, call, teardown). These functions can be overridden to provide your own implementation or you can wrap functionality around the default one.

A pytest plugin basically consists of one or more of these hook functions. There are:

• builtin plugins: loaded from pytest’s internal _pytest directory
• external plugins: modules discovered through setuptools entry points
• conftest.py plugins: modules auto-discovered in test directories

In our example, we are going to create a local conftest plugin that can be used for our own project.

Goal 🎯

Tests, especially ones that communicate with 3rd parties, are occasionally flaky. Thus, it would be useful to be able to:

• retry running failed tests marked with a specific flag
• define the retry strategy (max calls, timeout)

So, this is what we will achieve with this example, while also gaining a better understanding of pytest’s plugins.

We draw inspiration from the pytest-rerunfailures plugin.

Code 🐍

Retry logic

• From pytest’s API reference, we can see that one hook we could implement to retry a failed test is the pytest_runtest_call function, which is Called to run the test for test item (the call phase).
  Note that implementing this hook will only retry tests that fail on the call phase, not on setup or teardown.
• We will use a hookwrapper, to give us the flexibility to run the original hook implementation and be able to do some pre or post-processing of the result and to make our code more reusable (see GitHub).
• It’s important to keep in mind that tests will invoke all hooks defined in conftest.py files closer to the root of the filesystem (see here).
• Our conftest.py should include something like:

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(item: Item) -> Iterator[Any]:
    """
    Implemented hook to:
    - retry running failing tests

    Reference:
    https://docs.pytest.org/en/latest/reference/reference.html#pytest.hookspec.pytest_runtest_call
    """

    # First run of the test with `yield` (no exceptions are raised in any case).
    outcome: Result[Any] = yield  # type: ignore[assignment]
    
    # If no markers are provided (`None`), a test will be covered by the retry
    # mechanism for any marker.
    pytest_markers = {"retry_failed"}
    retry_on_exceptions = Exception
    max_calls = 3
    retry_timeout_in_seconds = 3.0

    item_markers = {marker.name for marker in item.iter_markers()}

    if pytest_markers is not None:
        common_markers = item_markers & pytest_markers

    # This filtering applies to a single test item
    if pytest_markers is None or len(common_markers) > 0:
        # These attributes can be used to make a retry report (`pytest_runtest_makereport` hook).
        item._max_calls = max_calls  # type: ignore[attr-defined]
        item._calls = 1  # type: ignore[attr-defined]

        exc_type = None
        if outcome.excinfo is not None:
            exc_type = outcome.excinfo[0]
            exc = outcome.excinfo[1]

        if exc_type is not None and issubclass(exc_type, retry_on_exceptions):
            failure = str(exc) + "\n"
            item.add_report_section(when="call", key="failure", content=failure)

            # First call happens when `outcome` is yielded, so we need one
            # less than `max_calls`.
            while max_calls - 1 > 0:
                time.sleep(retry_timeout_in_seconds)

                try:
                    item._calls += 1  # type: ignore[attr-defined]
                    item.runtest()
                    return
                except retry_on_exceptions as e:
                    failure = str(e) + "\n"
                    item.add_report_section(when="call", key="failure", content=failure)

                    max_calls -= 1

Another way to go, that would allow us even more control of each phase of the test item, would be to implement the pytest_runtest_protocol function, but we choose a simpler approach for our example. To see how that can be done check out pytest-rerunfailures.

Retry Summary Report

As an extra, we add the tests that were retried and the reason they failed in previous attempts to pytest’s report, by implementing the pytest_runtest_makereport and pytest_terminal_summary hooks.

For the details see the conftest.py.

Source Code💻

To see a sample project structure and run this hook see the following repo, https://github.com/KAUTH/pytest-retry-conftest-plugin.