Added support for test hierarchies

I finally got the time to get into the details of how to use the unit testing part of PlatformIO. First, many thanks for a super nice feature 😃 .

A few comments below. Sorry for the length, this is really because I love this feature and would love to see it be even better if it can be that I write something so long, so please take it as compliment not criticism 😃 .

the “what is included for building tests” question

It was quite confusing to me at first that plaformio unit testing does not use any of the files / headers in src and include. After a bit of confusion, I found the platformio web page documenting that unit testing by default only uses the lib content, and that a (discouraged) platformio.ini flag is needed to also use what comes from the src. This is the kind of thing that I was very confused about - I know “a posteriori” that I should have read the documentation in more details, but just wondering if this information may be added to the readmes of the src and include folders of each project 😃 .

the “tests folders need to be at the root of test” question

Another thing I was a bit confused about is that the tests always need to live in the “root” of the test folder. What I mean is that it is ok to have a structure:

- test
   | - test_embedded_test_1
   |     |- main.cpp
   |     |- util.cpp
   | - test_embedded_test_2
   |     |- main.cpp
   |     |- another_util.cpp

but that the following structure fails to run tests, complains about multiple setup and loop functions etc, and that was confusing for me at first:

- test
   | embedded
         | - test_1
         |     |- main.cpp
         |     |- util.cpp
         | - test_2
         |     |- main.cpp
         |     |- another_util.cpp

Having a possibility to have a bit of hierarchy would maybe be convenient though - for grouping stuff by category etc. Would it maybe be an option to allow the user to put a .test_hierarchy file or similar in folders that are not the root of a test case, but an “organization level” for a set of tests? Let me know if this is unclear. Maybe there is a better strategy, too.

the “bad tests setup” question

I first set up my tests wrongly, and then the test suite was not failing, just freezing. This was also very confusing. The issue was: tests need to be within a “RUN_TEST()” command and not just “natively”, i.e.:

• this works:

void setup() {
    UNITY_BEGIN();

    RUN_TEST(some_test);

    UNITY_END();
}

• this does not work (freeze), without error messages, which is confusing to me:

    UNITY_BEGIN();

    some_test;

    UNITY_END();
}

This is the kind of stupid mistake that happens, and is was very confusing to me that stuff just froze, without any meaningful error message or else… Wondering if there would be a way to catch this at compile time.

the caching question

Would it be possible to perform caching of all tests with all their dependencies etc individually and in “parallel” of the src, so that it is fast to run and re run tests and project building / upload back and forth? I think I observed the following behavior:

• if I build / upload several times in a row this is very fast, thanks to src build caching
• if I test several times in a row this is very fast, thanks to test build caching
• if I build then test then build, the second build is slow and need to recompile everything
• if I test then build then test, the second test is slow and need to recompile everything

Can you confirm that this is well the behavior that is happening at the moment and that I am not missing something? 😃

What about / would it be possible to set things so that, for example, both the src build and test(s) build(s) get cached “side by side” of each other, so that one can have fast builds in all cases even switching back and forth? I (and most people today 😃 ) have plenty of disk space, so I would not mind if the platformio build cache for a project is real 1 + N platformio build cache folders, with 1 src cache and N individual test caches (1 for each test). Of course if there can be an even better strategy that would be great too 😃 .

[ivankravets]

Guys, thanks for your feedback on PlatformIO Unit Testing. We refactored our unit testing engine from scratch. A lot of sweet things are coming soon. See https://github.com/platformio/platformio-core/milestone/94 and https://github.com/platformio/platformio-core/blob/develop/HISTORY.rst

Summary of what has been done:

1. Added support for build_type = test
2. Added support for test hierarchies (nested test suites). A test suite (folder) MUST start from test_ prefix. You can now write complex tests that include test-code related folders.

Please upgrade to the latest PIO Core via pio upgrade --dev and re-test these features. Does it work now?

Please also review https://github.com/platformio/platformio-core/milestone/94 , did we miss something?

[ivankravets]

Guys, thank you so much for your feedback and comments. Let us summarize your requests:

1. Allow nested folder structure in the “test” folder
2. Add support for build_type = test, the firmware will be built together with a testing framework
3. Update test/README with information that source files from the src folder are not included in the test build by default.

Did I miss something?