> ## Content Index
> Fetch the complete content index at: https://www.smallstepsystems.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# QTest: Data-Driven Unit Tests Are Hard To Understand
- URL: https://www.smallstepsystems.com/qtest-data-driven-unit-tests-hard-to-understand/
- Published: 2019-08-19T08:00:41.000Z
- Updated: 2026-01-25T13:42:58.000Z
- Author: Burkhard Stubert
- Tags: Continuous Delivery

Today, I looked at the data-driven unit tests I had written nearly four weeks ago. It took me a couple of minutes to understand the tests again. Understanding my own tests should have been much easier.

[Data-driven unit tests in QTest](https://doc.qt.io/qt-5/qttestlib-tutorial2-example.html?ref=smallstepsystems.com) have a fundamental problem. The data goes in one function, say `testWriteFrame_data`, and the test case working on the data goes in another function, say `testWriteFrame`. I must go back and forth to understand the test case. While going back and forth, I typically forget one piece of information. So, I must do another round trip.

So, I sat down and converted each row of the table created by `testWriteFrame_data` into a test function of its own. The resulting test cases are much easier to understand. They have about the same code size as the original solution. But see for yourself.

Here are the original data-driven test cases - split up in the data part `testWriteFrame_data` and the execution part `testWriteFrame`.

```
void TestReadWriteOnMockCanBus::testWriteFrame_data()
{
    QTest::addColumn<MockCanFrameCollection>("outgoingFrames");
    QTest::addColumn<MockCanFrameCollection>("expectedCanFrames");
    QTest::addColumn<bool>("isCanIoOk");
    auto frame1 = MockCanFrame{MockCanFrame::Type::Outgoing, 
        0x18ef0201U, "018A010000000000"};
    auto frame2 = MockCanFrame{MockCanFrame::Type::Outgoing, 
        0x18ef0301U, "01B5010000000000"};

    QTest::newRow("Write two frames in expected order")
            << MockCanFrameCollection{frame1, frame2}
            << MockCanFrameCollection{frame1, frame2}
            << true;
    QTest::newRow("Write more frames than expected")
            << MockCanFrameCollection{frame1, frame2}
            << MockCanFrameCollection{frame1}
            << false;
    // 4 more rows ...
}

void TestReadWriteOnMockCanBus::testWriteFrame()
{
    QFETCH(MockCanFrameCollection, outgoingFrames);
    QFETCH(MockCanFrameCollection, expectedCanFrames);
    QFETCH(bool, isCanIoOk);

    setExpectedCanFrames(m_device, expectedCanFrames);

    for (const auto &frame : outgoingFrames) {
        QVERIFY(m_device->writeFrame(frame));
    }
    QCOMPARE(actualCanFrames(m_device) == expectedCanFrames, 
             isCanIoOk);
    QCOMPARE(m_writtenSpy->size(), outgoingFrames.size());
}
```

The function `testWriteFrame_data` creates a table with the columns `outgoingFrames`, `expectedCanFrames` and `isCanIoOk` and six rows (only two of them shown). `QTest::newRow` creates a row with a tag (e.g., "Write one expected frame") and a value for each column.

The test runner calls the function `testWriteFrame` for each row. The `QFETCH` macros extract the data from a row and assign them to the respective variables.

The function `testWriteFrame` specifies with the call to `setExpectedCanFrames` that it expects the test case to produce the `expectedCanFrames` on the CAN bus. It then writes all the `outgoingFrames` to the CAN bus and finally checks the results.

So, what is my problem with data-driven tests? I must look at two places - `testWriteFrame_data` and `testWriteFrame` \- at the "same" time to understand the test case. Typically, I go back and forth between the two places a couple of time, before I understand why the test fails. This may even involve scrolling up and down through the code. I easily forget a piece of information and must take another round trip.

I want to understand the test function with one glance. All the required information must be in one place. I achieved this by converting each row into a test function of its own. Here are the test functions for the two rows shown above.

```
void TestReadWriteOnMockCanBus::testWriteTwoFramesInExpectedOrder()
{
    expectWriteFrame(c_frame1);
    expectWriteFrame(c_frame2);

    m_device->writeFrame(c_frame1);
    m_device->writeFrame(c_frame2);

    CHECK_FRAMES_EXPECTED_EQUAL;
}

void TestReadWriteOnMockCanBus::testWriteMoreFramesThanExpected()
{
    expectWriteFrame(c_frame1);

    m_device->writeFrame(c_frame1);
    m_device->writeFrame(c_frame2);

    CHECK_FRAMES_EXPECTED_NOT_EQUAL;
}
```

The first block with the calls to `expectWriteFrame` specifies which CAN messages the second block with the calls to `writeFrame` must produce. The third block is a macro to check the actual result of the second block against the expected result of the first block. The two `CHECK_FRAMES` macros give better diagnostic output than the two `QCOMPARE` macros in the original version, because they don't have to be the same for an expected failure and for an expected success.

The intent of the rewritten tests is instantly clear. One look at the tests is enough to understand them.

This simplicity is not possible with the data-driven approach of QTest. The required information is always in two places. The actual test function tends to have a higher complexity, because it must take care of multiple execution paths (the `for` loop and the `isCanIoOk` flag in this example).

My conclusion is clear: I won't use data-driven unit tests much in the future.

**Acknowledgement**. I want to thank James W. Grenning for pointing out the same problem in my solution to one of the exercises in his course [Test-Driven Development for Embedded C/C++](https://wingman-sw.com/training/tdd-cpp?ref=smallstepsystems.com).