Testing UGens / c++ code during development

Development Server plugins
1

@nathan mentioned the use of google’s test suite to ensure mathematical correctness in c++ code for server plugins in another thread, and I thought it deserved it’s own here.

This article was mentioned as a good intro to using Google’s test suite: Testing, for people who hate testing / fuzzy notepad

And also I realised that this talk just came out from the Audio Developer Conference, might be relevant too:

2 Likes
2

i don’t think i have much knowledge to share on testing UGens specifically, other than that it’s probably easiest to separate out the actual mathy algorithmy parts so you can test those with your own test data free from any SC/plugin baggage.

for specific testing frameworks, two alternatives i have used are Catch2 and Boost.Test (which is what we use in SuperCollider itself). i would personally recommend Catch2 over Boost.Test; it’s very easy to integrate, it’s got better support for modern C++ features, and I think it offers a better user experience overall. i’d be curious to know if anyone has used it for audio work.

3

thanks @VIRTUALDOG you’ve convinced me to try out Catch2 :slight_smile:

Costas Calamvokis’ talk above is really great (and also mentions Catch2 as an easier and better alternative to Google’s test suite)

I have actually done what you’ve recommended. My plugin collection has most of the math and algorithms seperated into header files that then contain the building blocks for the rest of the plugins so implementing automated testing here should actually be pretty easy.

Thanks for the inspiration!

1 Like
4

This talk is also related, even though I haven’t seen all of it yet:

5

Alright, I’m starting to get setup and things are going relatively well, except I am running in to a problem which is probably due to my lacking cmake skills. I decided to go for catch2.

First of all, I installed catch2 systemwide using my package manager (sudo pacman -S catch2 on arch linux).

Then, I added this section to my UGen CMakeLists.txt file. I added an option TESTING which may be set to turn on/off building the tests binary. Not sure if this is the smartest way of doing such a thing but seems the easiest for me at least.

####################################################################################################
# Testing
option(TESTING "Build tests binary using catch2" ON)

# Tests
if (TESTING)
	# Requires that catch2 is installed on the system
	find_package(Catch2 REQUIRED)
	add_executable(tests test/main.cpp)
	target_link_libraries(tests Catch2::Catch2)
endif()

And created a test file in a subdirectory from the root of my project:
test/main.cpp which looks like this (a silly non-test for testing that it works):

#define CATCH_CONFIG_MAIN // This tells Catch to provide a main() - only do this
                          // once
#include "catch2/catch.hpp"

unsigned int Factorial(unsigned int number) {
  return number <= 1 ? number : Factorial(number - 1) * number;
}

TEST_CASE("Factorials are computed", "[factorial]") {
  REQUIRE(Factorial(1) == 1);
  REQUIRE(Factorial(3) == 2);
  REQUIRE(Factorial(3) == 6);
  REQUIRE(Factorial(10) == 3628800);
}

Then running the usual make/cmake commands, I get the tests binary in my build directory and running that runs the tests. Hurray!

The problem: Including stuff that includes SC’s plugin api

So, then my problem is this: When I add my header file allpass.hpp to this test file mentioned above, it cannot find the sc api stuff.
At the top of test/main.cpp I add this header:

#include "../plugins/Common/allpass.hpp"

It seems to find the header and include it, but compilation of the tests binary is terminated with this error message:

In file included from /home/mads/code/notamplugins/test/main.cpp:3:
/home/mads/code/notamplugins/test/../plugins/Common/allpass.hpp:11:10: fatal error: SC_PlugIn.hpp: No such file or directory
   11 | #include "SC_PlugIn.hpp"
      |          ^~~~~~~~~~~~~~~

This of course is caused by my allpass.hpp relying on the SC_PlugIn.hpp include.

I guess I need to do some cmake trick here. Does anyone have any pointers/tips?

6

Aha, I think I figured out how to do this now.

The problem was that the test folder did not have all of the SuperCollider stuff in it’s header search path.

The steps to adding Catch2 as a testing framework in my project were these:

Add Catch2 as submodule

git submodule add https://github.com/catchorg/Catch2

This downloads Catch2 to a folder of the same name at the root of my project.
Then mkdir test to create a seperate folder for my tests and add the file test/main.cpp there:

#define CATCH_CONFIG_MAIN // This tells Catch to provide a main() - only do this
                          // once
#include "SC_Constants.h"
#include "catch2/catch.hpp"
//
// A silly test to see if the plugin's pi is equivalent to M_PI
TEST_CASE("See if Pi is okay", "[pi check]") { REQUIRE(pi == M_PI); }

And then I added these lines to my CMakeLists.txt:


# Add seperate tests executable for running catch2 tests
add_executable(tests test/main.cpp)

# Add catch2 library
add_subdirectory(Catch2)
include_directories(test Catch2)

# Add plugins folder to search path of tests binary
include_directories(test plugins)

# Add SuperCollider files to header search path for the tests binary
include_directories(test ${SC_PATH}/include/plugin_interface)
include_directories(test ${SC_PATH}/include/common)
include_directories(test ${SC_PATH}/include/server)

And now I can test things in files that include the SC Plugin header.

7

Made a small repository here with a simple example of this GitHub - madskjeldgaard/supercolliderplugin-catch2: An example plugin for SuperCollider with the Catch2 testing framework setup for automatic testing

8

Late to the party, but CMake includes functionality for running tests, see Testing · Modern CMake or Testing With CTest · Wiki · CMake / Community · GitLab

This allows you to run all or a subset of your tests simply with cmake --build <folder> --target test

Disclaimer: I’ve only discovered this part of CMake recently and haven’t really used it myself (yet).

1 Like