Writing benchmarks

This document describes everything you need to know to write your own benchmarks.

Basics

The benchmark itself can use the the same C++ language features as the function to measure against. Within the benchmark editor the following structure is required:

#include <Headers> // You do not need to include the test function but all required for its 

bb_register(<Function signature>);

void bb_entry() {
    // optionally custom test data initialization
    
    // one or more calls to bb_invoke
    bb_invoke(<Function name>, arg0, arg1, ...);
}

A benchmark usually tests a single function. There can be multiple implementations for this function given in the source editors, but they all have to match exactly in their interface. To make the function known to the benchmark, you need to register it by using the bb_register(<Signature>); helper with the full signature.

// Example for function: int add(int a, int b);
bb_register(int add(int,int));

// Example for function: bool does_contain(std::vector<float> const& vec, float f);
bb_register(bool does_contain(std::vector<float> const&, float));

In some cases, you may want to compare different function signatures. This can by done by registering multiple signatures.

Only after registration, the function names can be used within the benchmark code.

Measure a function call

The benchmark will execute all statements inside the  bb_entry() handler. This happens independently for all configurations given.

To measure the runtime of a function you use the bb_invoke(<function name>, arg0[, arg1, ...]); helper. This helper internally

• starts a clock
• sets up the given arguments (stack, registers)
• calls the function
• stops the clock
• append gathered data to list of results (runtime, arg0, return value)

You may call bb_invoke for all arguments of interest. The list within the result set is ordered by the sequence of the calls.

Note: Invoking a function without any parameters is not supported.

Examples

bb_register(int square(int));

bb_entry() {
    bb_invoke(square, -5); // calls square(-5)
    bb_invoke(square, 5);
    bb_invoke(square, 99);
}

bb_register(int square(int));

bb_entry() {
    for (int arg=0; arg<10; arg++) {
        bb_invoke(square, arg);
    }
}

bb_register(int add(int,int));

bb_entry() {
    bb_invoke(add, 0, 1); // calls add(0,1)
}

Custom serialization

The result data contains a serialized view about the parameters passed and the function return value for each invocation. For full flexibility, this serialization takes place on the target itself. There are pre-defined conversions for the following types:

• bool
• char, short, int, long
• unsigned char, unsigned short, unsigned int, unsigned long
• ‘-‘ for unknown types

To make other types known to the system, you can provide a custom serialization by specialising the bb_serialize<>(); function template and write the desired output to bb_print(char const*);

Assuming, we are benchmarking a function with an optional parameter.

#include <optional>

int func(std::optional<int> const& x) {
    // do something
}

There exists no pre-defined serialization for the optional type. Optional can either contain an int, or nothing at all. In the first case, we want to display optional(<Value>), in the second case an optional(). The function specialization has to take place within the benchmark code. The following example yields the the desired output.

#include <optional>

bb_register(int func(std::optional<int> const&));

// Provide custom result serialization for optional type
template <>
void bb_serialize(std::optional<int> const& x) {
    if (x.has_value()) {
        char buf[20];
        std::snprintf(buf, 20, "optional(%i)", x.value());
        bb_print(buf);
    } else {
        bb_print("optional(-)");
    }
}

void bb_entry() {
    bb_invoke(func, std::optional());
    bb_invoke(func, std::optional(1));
    bb_invoke(func, std::optional(-5));
}

Start benchmark

A click on Start Benchmark will compile your benchmark for all given configurations.  The request is then queued. Once a slot on the chosen MCU(s) is available, it is executed.

The results are gathered and shown within the result view once available.

Help with errors

I cannot start the benchmark

The Start Benchmark is disabled, when you have errors within your code editors or the benchmark code itself. Within the editors, the line causing an error will get highlighted. Hover over the error to get more information.

I get no results

A benchmark may (partially) fail for several reasons. Hover over the error-icon  to get more details.

• Compilation error: The linking failed. Double check your code function name and signature with the name and signature given within the benchmark. They need to match exactly for each code.
• Target currently not available: If there is no free slot within 60s on your chosen MCU, the request is discarded. Retry later.
• Maximal executions per day exeeded. The number of daily benchmarks is limited. See Plan details for more information.
• Benchmark execution took too long or yielded an error: A function call did either consume too much runtime or caused a hardware interrupt. The specific limitations depend on the chosen target.
• Too many iterations. Decrease iterations in benchmark: The number of invocations (via bb_invoke) per benchmark is limited. Restart with less calls. See Plan details for more information.