Functional testing of C++ code
tutorial objectives: Use Octave to test your C++ code. Octave is a free software similar to Matlab (same syntaxes). The detailed method is only useful if your C++ algorithms to be tested are already available in Octave. The field covered is quite vast: statistics, signal processing (sounds, images), matrix calculation, optimizations, specific mathematical functions, interpolations, etc.
Level: Intermediate.
Prerequisites :
- Have a Linux OS (or Cygwin for Windows users…).
- Have Octave (I recommend at least Octave 3.2) and packages that contain mkoctfile in particular.
- Have g++ (gcc suite)
- Know a minimum of C++ and have some knowledge of the Octave language (or Matlab since the languages are almost identical).
Introduction
In this tutorial, we will see how to test your C++ code with Octave. This is of course not a method to be applied systematically but it can prove to be valuable when you have to code, for example, an fft, matrix operations, image or sound processing algorithms, etc. In short, as soon as you have scientific algorithms, this method will be very useful to you.
Octave will therefore allow you to compare the results from your C++ code with its own results. It is then very easy to do Monte Carlo (consists of making thousands of draws to be able to make statistics) to carry out tests of robustness and coverage. Octave will even allow you to analyze your errors (bad conditioning of matrices for example) thanks to its primitive numbers and to the fact that Octave is an inline scripting language.
Principle of the tutorial
Brief description of Octave
Octave is an Open Source software written in C++ which is composed of a set of libraries (static .a and dynamic .so) and an interpreter of the Octave language (by the way which is very close to the Matlab language).
So you can use Octave scientific computing features in C++ directly in your C++ code. Well I digress, this is not the subject of the tutorial…
In this tutorial, we will test a matrix addition. In practice, it would probably be too cumbersome to use this method for a simple addition, but the approach will thus be simple and easier to understand.
The steps of the tutorial
The objective of the tutorial is to perform the following steps:
1. We suppose to have a C++ class MyMatrix having elements and an operation of addition of 2 matrices of identical sizes.
2. We are going to create a C++ wrapper of the functionality we want to test (the addition in this case). In other words, we are going to create a new matrix addition function for Octave that will call our code to test. This piece of code will be produced in C++ respecting a well-defined interface. Octave will then be able to invoke this C++ function from its own interpreter. The diagram below will no doubt help to clarify my obscure remarks…
Limits and caveats
You're going to tell me (and you'll be right!): it's all very well, but what credit should we give to this test? If I find differences on some elements of the matrix of the order of 10 -15 , what to think?
That's a good point! First, it is imperative to type the Octave elements. Indeed, this language (like many scripting languages) is very flexible because it types inline, on the fly. By default, if an array element is decimal, it will use double precision (64 bits). But if your matrix is typed float (32 bits), the comparison is no longer relevant… You must therefore use the Octave types. They have been introduced since the last versions of Octave.
Feel free to check out the documentation for matrix types under Octave .
Second, you have to set a tolerance. If your application is critical, set yourself a very low tolerance for each item. Otherwise you can set an average tolerance on all the elements of the matrix.
Good as I feel you impatient, let us return in the heart of the technique!
Realization of the test code
C++ code to test
We want to test a C++ class named MyMatrix.
RQ1: This class is really a simple case study for the tutorial. I went to the fastest for this tutorial, the object not being to develop a matrix calculation library ;-)! I, therefore, do not recommend that you code it like this for a real application!
RQ2: Warning Octave already has a class called Matrix, that's why I gave my class a great name… lol To avoid collisions, you can use namespaces, it will be a better solution than changing the name of your class! Here for the tutorial, whatever...
#ifndef _MATRIX_H_
#define _MATRIX_H_
#include <iostream>
using namespace std;
/**
* Matrix class with float elements (32 bit precision)
*
* This class is very, very, very bad designed... It's only to
* do the tutorial...
*/
class MyMatrix {
private:
/**
* number of rows
*/
int mNbRows;
/**
* number of columns
*/
int mNbColumns;
/**
* Elements
*/
float mElements[255][255];
audience:
/**
* Builder
*
* @param nbRows: number of rows
* @param nbColumns: number of columns
* @param initValue: Value which be used to initialize all elements
*/
MyMatrix(int nbRows, int nbColumns, float initValue) {
mNbRows = nbRows;
mNbColumns = nbColumns;
for (int i=0; i<mNbRows; i++)
for (int j=0; j<mNbColumns; j++)
mElements[i][j] = initValue;
}
/**
* Builder
*
* @param nbRows: number of rows
* @param nbColumns: number of columns
*/
MyMatrix(int nbRows, int nbColumns) {
mNbRows = nbRows;
mNbColumns = nbColumns;
}
/**
* Destroyer
*/
virtual ~MyMatrix() {
mNbRows = 0;
mNbColumns = 0;
}
// Inline getters and setters...
inline float getElement(int indexRow, int indexColumn) {
return mElements[indexRow][indexColumn];
}
inline void setElement(int indexRow, int indexColumn, float newElement) {
mElements[indexRow][indexColumn] = newElement;
}
inline int getNbRows() {
return this->mNbRows;
}
inline int getNbColumns() {
return this->mNbColumns;
}
/**
* MyMatrix addition to test with Octave
*/
static MyMatrix addition(MyMatrix A, MyMatrix B) {
MyMatrix C(A.getNbRows(), B.getNbColumns(), 0.f);
for (int i=0; i<A.getNbRows(); i++)
for (int j=0; j<A.getNbColumns(); j++)
C.setElement(i, j, A.getElement(i, j) + B.getElement(i, j));
return C;
}
/**
* To print matrix on terminal
*/
static void print(MyMatrix A) {
for (int i=0; i<A.getNbRows(); i++) {
for (int j=0; j<A.getNbColumns(); j++) {
cout << A.getElement(i, j) << "\t";
}
cout<<endl;
}
cout<<endl;
}
}; // end of Class definition
#endif /* _MATRIX_H_ */
(code-box)
To summarize, this minimalist C++ class has the following characteristics:
- Elements (a 2D array of 32-bit floating point numbers).
- The dimension of the matrix.
- Getters/setters…
- A static method of addition which receives 2 matrices as argument and returns the result in a 3rd matrix .
- A static method of displaying a matrix.
Realization of the wrapping for Octave
We are going to create the file for this: wrapperAdditionCppATester.cpp.
Analysis of Octave documentation
Let's analyze the corresponding Octave doc .
For this wrapping, here is a very simple example:
#include <octave/oct.h>
DEFUN_DLD (helloworld, args, nargout, "Hello World Help String") {
int nargin = args. length();
octave_stdout << "Hello World has " << nargin
<< " input arguments and "
<< nargout << " output arguments.\n";
return octave_value_list();
}
(code-box)
You have to include oct.h, so far, no problem;-)
DEFUN is a macro whose arguments are as follows:
- helloworld: the name of the function that we are going to create and which will be callable from the Octave interpreter
- args: the list of arguments we want to pass to our function (for us: 2 matrices)
- nargout: the number of arguments at the output of our function
- "Hello World Help String": a character string that will be displayed when you type: "help myOctaveFunction" in the Octave interpreter.
In the body of this function, we start by initializing an integer variable nargin which will contain the number of arguments that must enter our function.
Then we implement the wrapping in the body…
Finally, we return
octave_value_list();(code-box)
Adaptation to our needs
Here we are…
1. First of all, you need the right headers:
octave/oct.h(code-box)
but also
octave/Array.h(code-box)
in order to be able to manipulate typed Octave arrays.
2. We must retrieve the 2 matrices that will be in the argument of our new Octave function :
ArrayN<float> A_octave = args(0).array_value();
ArrayN<float> B_octave = args(1).array_value();
(code-box)
The ArrayN template must receive the same type as the one used in MyMatrix.hpp, otherwise the comparison of the results will be meaningless.
3. We then define our own matrices which will be exact copies of A_octave and B_octave but in our format:
MyMatrix A_cpp(nbRows, nbCols, 0.f);
MyMatrix B_cpp(nbRows, nbCols, 0.f);
MyMatrix C_cpp(nbRows, nbCols, 0.f);
for (int i=0; i<nbRows; i++) {
for (int j=0; j<nbCols; j++) {
A_cpp.setElement(i, j, A_octave(i, j));
B_cpp.setElement(i, j, B_octave(i, j));
}
}
(code-box)
4. We call our method to be tested:
C_cpp = MyMatrix::addition(A_cpp, B_cpp);(code-box)
5. Then we copy this result into a matrix in Octave format:
const dim_vector dimC(nbRows, nbCols);
ArrayN<float> C_octave(dimC);
for (int i=0; i<nbRows; i++) {
for (int j=0; j<nbCols; j++) {
C_octave(i, j) = C_cpp.getElement(i, j);
}
}
(code-box)
6. We return the result:
return octave_value(C_octave);(code-box)
Here is the final source code:
#include <octave/oct.h> // Must be always include: contains the DEFUN_DLD macro.
#include <octave/Array.h> // To be able to use typed arrays from Octave
#include "./MyMatrix.hpp"
DEFUN_DLD(wrapperAdditionCppATester, args, nargout, "Sum of two float32 matrix") {
// Number of arguments
int nargin = args.length();
// Valid case: 2 matrix to sum as following: C = A + B
if (nargin == 2) {
ArrayN<float> A_octave = args(0).array_value();
ArrayN<float> B_octave = args(1).array_value();
// first matrix dimensions
const int nbRows = A_octave.rows();
const int nbCols = A_octave.cols();
// computation possible if only the 2 matrix have same dimensions
if ((! error_state) && (nbRows == B_octave.rows()) && (nbCols == B_octave.cols())) {
// matrix copies: Octave --> C++
MyMatrix A_cpp(nbRows, nbCols, 0.f);
MyMatrix B_cpp(nbRows, nbCols, 0.f);
MyMatrix C_cpp(nbRows, nbCols, 0.f);
for (int i=0; i<nbRows; i++) {
for (int j=0; j<nbCols; j++) {
A_cpp.setElement(i, j, A_octave(i, j));
B_cpp.setElement(i, j, B_octave(i, j));
}
}
// We call our addition C++ function
C_cpp = MyMatrix::addition(A_cpp, B_cpp);
// result storage in an Octave matrix
const dim_vector dimC(nbRows, nbCols);
ArrayN<float> C_octave(dimC);
// matrix copy: C++ --> Octave
for (int i=0; i<nbRows; i++) {
for (int j=0; j<nbCols; j++) {
C_octave(i, j) = C_cpp.getElement(i, j);
}
}
// We return the result and stop this function here
return octave_value(C_octave);
}
}
// We print how to use this function
else {
print_usage();
}
// It's an obligation to return this following...
return octave_value_list();
}
(code-box)
Building the dynamic .oct library for Octave
Now, this file will be used to build a dynamic library for Octave. It will therefore not have the extension .so but .oct!
You have to launch a terminal and go to its working directory, where the sources are. We then invoke the mkoctfile command applied to our wrapping cpp file:
We can then see, if there is no error of course ;-) that 2 files have been created. The .o was used to make the .oct. You can purge it, it is useless… The .oct is your dynamic Octave library. Congratulations!
You may get a message like this:
No command mkoctfile …(code-box)
. In this case, you are probably missing packages containing the required Octave libraries in your Linux distribution. On Ubuntu, in Package Manager, you can search by keyword "octave" and add packages, then try mkoctfile again...
By the way, note that it is possible to use a static or dynamic library instead of my MyMatrix.h file. In this case, the handling is a little more difficult because you have to modify the environment variable LD_LIBRARY_PATH beforehand (for the "ld" linker) and indicate the location of the includes... Classic when using libraries. Do not hesitate to ask me questions if necessary on this point if you want to go further.
Call of our addition function under Octave
We take our terminal and launch Octave with the command: octave
You can do an "ls" under Octave to make sure you are in the right place. We should always see the .oct file
We can try the interpretation of our new function:
Test script under Octave
- Vary the dimensions of the dies.
- Vary the dynamic noise used to generate random numbers in matrices.
- etc
function[numberOfSuccess] = test_MyMatrix_addition(numberOfRealization, acceptanceThreshold)
# Out argument initialization
numberOfSuccess = 0;
# Monte Carlo random sampling on:
# - Matrix number of rows
# - Matrix number of columns
# - Matrix elements
for i = 1: numberOfRealization,
# Matrix dimension sampling
numRows = round(rand*100) + 1;
numCols = round(rand*100) + 1;
# Matrix multiplier sampling
multiplier = rand*100;
# Matrix allocations and elements sampling
A = single( rand(nbRows, nbCols) * multiplier ); # cast in 32bit floating point
B = single( rand(nbRows, nbCols) * multiplier ); # cast in 32bit floating point
# Reference Matrix initialized with Octave computation
C_octave = A + B;
# Matrix to test with our computation
C_MyMatrix = wrapperAdditionCppATester(A, B);
# Analysis of 2 results differences
C_diff = abs(C_octave - C_MyMatrix);
diff_max = max(max(C_diff));
# If we consider a success test, we increment the 'numberOfSucess' memory
if (diff_max < acceptanceThreshold)
numberOfSuccess = numberOfSuccess + 1;
end
end
end function
(code-box)
Here is his call:
Result: 100 runs played, 100 runs validated!
We have just validated our addition matrix operation in 32 bits! Not so complex in the end… I let you imagine that in the case of an FFT, it is better to avoid unrolling examples on paper. This method therefore makes life easier ;-).
RQ : In this example, we cannot really talk about Monte Carlo. You would have to do tens of thousands of runs... It's up to you to see the criticality of your application and size the tests accordingly. Nights are quite practical for running this kind of tests!
Conclusion
As you have seen, this method is quite greedy in terms of calculation time because the copies and wrappings go in all directions. That said, our goal is to be able to do functional testing and not optimal embedded code.
This method has the advantage of being able to test your own code very easily with Octave's libraries, and there are many of them. You can very easily adapt this tutorial to test C code or Fortran code. Consult the online help on Octave. Everything is explained.
In this tutorial, the operations presented are very manual and therefore impractical. Rest assured, it is possible to automate all this! Maybe a next tutorial?
Tutorial perspective
I hope this tutorial will be useful to you and that my explanations were clear enough…
In a future tutorial, we will see how to configure Eclipse CDT to automate everything we have implemented in this tutorial. For this, we will make a homemade Builder for Eclipse to build all our files (.o, .oct, etc.).
A bit later I will also post a tutorial based on these 2 previous ones which will add reverse wrapping which will allow us to call the test Octave script from C++. This may seem twisted and useless, but you will see that we can then perform unit testing (with a test framework similar to JUnit for Java). We can thus add to our classic unit test sets, Monte Carlo functional tests and trace all this in test reports! :O
Source code to download: Click Here