Description

A test bench to conduct performance measurement series.

Outfitted especially to determine runtime behaviour of the Scheduler and associated parts of the Lumiera Engine through systematic execution of load scenarios.

Scheduler Stress Testing

The point of departure for any stress testing is to show that the subject will break in controlled ways only. For the Scheduler this can easily be achieved by overloading until job deadlines are broken. Much more challenging however is the task to find out about the boundary of regular scheduler operation. This realm can be defined by the ability of the scheduler to follow and conform to the timings set out explicitly in the schedule. Obviously, short and localised load peaks can be accommodated, yet once a persistent backlog builds up, the schedule starts to slip and the calculation process will flounder.

A method to determine such a »breaking point« in a systematic way is based on building a synthetic calculation load and establish the timings of a test schedule based on a simplified model of expected computation expense. By scaling and condensing these schedule timings, a loss of control can be provoked, and determined by statistical observation: since the process of scheduling contains an essentially random component, persistent overload will be indicated by an increasing variance of the overall runtime, and a departure from the nominal runtime of the executed schedule.

Another, complimentary observation method is to inject a defined and homogeneous load peak into the scheduler and then watch the time it takes to process, the processing overhead and achieved degree of concurrency. The actual observation using this measurement setup attempts to establish a single control parameter as free variable, allowing to look for correlations and to build a linear regression model to characterise a supposed functional dependency. Simply put, given a number of fixed sizes jobs (not further correlated) as input, this approach yields a »number of jobs per time unit« and »socked overhead« — thereby distilling a behaviour model to describe the actual stochastic data.

Setup

To perform this test scheme, an operational Scheduler is required, and an instance of the TestChainLoad must be provided, configured with desired load properties. Moreover, the actual measurement setup requires to perform several test executions, controlling some parameters in accordance to the observation scheme. The control parameters and the specifics of the actual setup should be clearly visible, while hiding the complexities of measurement execution.

This can be achieved by a »Toolbench«, which is a framework with building blocks, providing a pre-arranged measurement rig for the various kinds of measurement setup. The implementation code is arranged as a »sandwich« structure...

StressTestRig, which is also the framework class, acts as bottom layer to provide an anchor point, some common definitions implying an invocation scheme
- first a TestChainLoad topology is constructed, based on test parameters
- this is used to create a TestChainLoad::SchedulerCtx, which is then outfitted specifically for each test run
the middle layer is a custom Setup class, which inherits from the bottom layer and fills in the actual topology and configuration for the desired test
the test performance is then initiated by layering a specific test tool on top of the compound, which in turn picks up the parametrisation from the Setup and base configuration, visible as base class (template param) CONF Together, this leads to the following code scheme, which aims to simplify experimentation:
using StressRig = StressTestRig<16>;
struct Setup : StressRig
{
uint CONCURRENCY = 4;

auto testLoad()
{....define a Test-Chain-Load topology....}

auto testSetup (TestLoad& testLoad)
{ return StressRig::testSetup(testLoad)
.withLoadTimeBase(500us)
// ....more customisation here
}
};
auto result = StressRig::with<Setup>()
.perform<bench::SpecialToolClass>();

Breaking Point search

The bench::BreakingPoint tool typically uses a complex interwoven job plan, which is tightened until the timing breaks. The stressFactor of the generated schedule will be the active parameter of this test, performing a binary search for the breaking point. The Measurement attempts to narrow down to the point of massive failure, when the ability to somehow cope with the schedule completely break down. Based on watching the Scheduler in operation, the detection was linked to three conditions, which typically will be triggered together, and within a narrow and reproducible parameter range:

an individual run counts as accidentally failed when the execution slips away by more than 2ms with respect to the defined overall schedule. When more than 55% of all observed runs are considered as failed, the first condition is met
moreover, the observed ''standard derivation'' must also surpass the same limit of > 2ms, which indicates that the Scheduling mechanism is under substantial strain; in regular operation, the slip is rather ~ 200µs.
the third condition is that the ''averaged delta'' has surpassed 4ms, which is 2 times the basic failure indicator.

Parameter Correlation

As a complement, the bench::ParameterRange tool is provided to run a specific Scheduler setup while varying a single control parameter within defined limits. This produces a set of (x,y) data, which can be used to search for correlations or build a linear regression model to describe the Scheduler's behaviour as function of the control parameter. The typical use case would be to use the input length (number of Jobs) as control parameter, leading to a model for Scheduling expense.

Observation tools

The TestChainLoad, together with its helpers and framework, already offers some tools to visualise the generated topology and to calculate statistics, and to watch an performance with instrumentation. In addition, the individual tools provide some debugging output to watch the measurement scheme. Result data is either a tuple of values (in case of bench::BreakingPoint), or a table of result data as function of the control parameter (for bench::ParameterRange). Result data, when converted to CSV, can be visualised as Gnuplot diagram.

See also: TestChainLoad_test; SchedulerStress_test; binary-search.hpp; gnuplot-gen.hpp

Definition in file stress-test-rig.hpp.

#include "test-chain-load.hpp"
#include "lib/binary-search.hpp"
#include "lib/test/transiently.hpp"
#include "vault/gear/scheduler.hpp"
#include "lib/time/timevalue.hpp"
#include "lib/meta/function.hpp"
#include "lib/format-string.hpp"
#include "lib/format-cout.hpp"
#include "lib/gnuplot-gen.hpp"
#include "lib/stat/statistic.hpp"
#include "lib/stat/data.hpp"
#include "lib/random.hpp"
#include "lib/util.hpp"
#include <algorithm>
#include <utility>
#include <vector>
#include <tuple>
#include <array>

Classes
class	BreakingPoint< CONF >
	Specific stress test scheme to determine the »breaking point« where the Scheduler starts to slip. More...

struct	LoadPeak_ParamRange_Evaluation::DataRow

struct	StressTestRig< maxFan >::Launcher< CONF >

struct	LoadPeak_ParamRange_Evaluation
	Mix-in for setup of a #ParameterRange evaluation to watch the processing of a single load peak, using the number of added job as independent parameter. More...

class	ParameterRange< CONF >
	Specific test scheme to perform a Scheduler setup over a given control parameter range to determine correlations. More...

struct	BreakingPoint< CONF >::Res

class	StressTestRig< maxFan >
	Configurable template framework for running Scheduler Stress tests Use to build a custom setup class, which is then injected to perform a specific measurement tool. Several tools and detailed customisations are available in `namespace bench` More...

Typedefs
using	IncidenceStat = lib::IncidenceCount::Statistic

Functions
template<typename F , typename G >
auto	linearRegression (Column< F > const &x, Column< G > const &y)
	Calculate a linear regression model for two table columns. More...

Namespaces
	vault
	Vault-Layer implementation namespace root.

	vault::gear
	Active working gear and plumbing.

Class Documentation

◆ vault::gear::test::bench::BreakingPoint::Res

struct vault::gear::test::bench::BreakingPoint::Res

Class Members
double	stressFac
double	percentOff
double	stdDev
double	avgDelta
double	avgTime
double	expTime

Collaboration diagram for BreakingPoint< CONF >::Res:

Function Documentation

◆ linearRegression()

auto vault::gear::test::bench::linearRegression	(	Column< F > const &	x,
		Column< G > const &	y
	)

inline

Calculate a linear regression model for two table columns.

Returns: a tuple (socket,gradient,Vector(predicted),Vector(deltas),correlation,maxDelta,stdev)

Definition at line 557 of file stress-test-rig.hpp.

References vault::gear::test::bench::linearRegression().

Referenced by vault::gear::test::bench::linearRegression().

Here is the call graph for this function:

Here is the caller graph for this function: