`experiment_matrix`#

benchmarks.experiment_matrix._config_generator(local_config, min_iter=None, max_iter=None, warmup=False, callback=None)[source]#

Configuration generator.

Turns the iterable config into a generator of config dictionaries.

Parameters

local_config – The configuration dictionary. All values must be iterable.
min_iter – The minimum number of iterations per config.
max_iter – The maximum number of iterations per config.
warmup – Whether to run the experiment once as a warmup.
callback – Boolean function to determine whether the experiment should continue.

Raises

StopIteration – Raised when the list of configs is exhausted. The value of the StopIteration exception contains the accumulated results.

Yields

A configuration dictionary.

Returns

Accumulated results for all experiments. This is returned via the value of the StopIteration exception.

benchmarks.experiment_matrix._result_accumulator(config, min_iter=None, max_iter=None, warmup=False, callback=None)[source]#

Result accumulator.

Maintains a list of results from the current experiment.

Parameters

config – The configuration dictionary.
min_iter – The minimum number of iterations per config.
max_iter – The maximum number of iterations per config.
warmup – Whether to run the experiment once as a warmup.
callback – Boolean function to determine whether the experiment should continue.

Yields

The current configuration for the experiment.

Returns

The accumulated results for the experiment.

benchmarks.experiment_matrix.experiment_matrix(ex, min_iter=3, max_iter=32, warmup=True, callback=None, same_seed=True)[source]#

Experiment matrix factory.

Creates an experiment matrix, which is a typical Sacred Experiment.

Parameters

ex – The Sacred Experiment to iterate over.
min_iter (int) – The minimum number of iterations per config.
max_iter (int) – The maximum number of iterations per config.
warmup (bool) – Whether to run the experiment once as a warmup.
callback – Boolean function to determine whether the experiment should continue.
same_seed (bool) – Whether all iterations should use the same seed.

Returns

A Sacred Experiment object.

Example

ex = Experiment()

@ex.config
def ex_config():
    value1 = 1
    value2 = 2

@ex.main
def main(value1, value2, _run):
    print(f"value1: {value1}, value2: {value2}")
    print(_run.info)
    return 1

mx = experiment_matrix(ex, warmup=True)

@mx.config
def mx_config():
    value1 = [1, 2]
    value2 = [3, 4]

if __name__ == "__main__":
    r = mx.run_commandline()

benchmarks.experiment_matrix.sample_mean(results, alpha=0.95)[source]#

Callback to determine when the results are within a confidence interval.

The sample mean function computes the relative error of the results and returns True if the last entry of the results list is within the interval. Returns False otherwise.

Parameters

results (list) – list of values returned from a sequence of experiments.
alpha (float) – The probability between 0 and 1 that a value is within the interval.

Returns

True if the last result is within the interval. False otherwise.

Return type

bool

Example

>>> config = {"value1": [1, 2], "value2": [3, 4]}
>>> mx = ExperimentMatrix(config, callback=sample_mean)

experiment_matrix#

`experiment_matrix`#