Skip to content

Catch2 integration does not handle Generators inside sections #205

@Manu343726

Description

@Manu343726

Per Catch2 documentation:

GENERATE can be seen as an implicit SECTION, that goes from the place GENERATE is used, to the end of the scope.
[...]
The fact that GENERATE introduces a virtual SECTION can also be used to make a generator replay only some SECTIONs, without having to explicitly add a SECTION

I looked into the Catch2 integration and it seems like the Catch2 API does not include generator information as part of the sector:

// Approvals/integrations/catch/Cathc2Approvals.h

struct Catch2TestCommitRevert : Catch::TestEventListenerBase
{
    ...
    virtual void sectionStarting(Catch::SectionInfo const& sectionInfo) override
    {
        currentTest.sections.push_back(sectionInfo.name); // Name here does not include the generator value
    }
    ...
};

In fact, if you look at the Catch::SectionInfo type it only includes name and source location. I'm no expert on the Catch2 internals but at first glance, I would say Catch2 generators are handled more like part of the body of a section rather than variants of the parent section (Like generated tests cases in other frameworks).

The result of this behavior is that if you run an approval inside a section with a generator, the approval is uniquely registered for the section rather than each generated variant of the section, resulting in approval failures following the execution of the different variants of the section:

enum class Keyword
{
    Unknown, Foo, Bar, Quux
};

Keyword parseKeyword(const std::string_view word);

SECTION("Verify our parsing algorithm works with the most relevant words in the vocabulary of a programmer")
{
     const std::string_view word = GENERATE("foo", "bar", "quux");
     Approvals::verify(parseKeyword(word)); // If we approve "foo", it fails in the next iteration of the generator
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions