Benchmark

This class provides methods for benchmarking different
optimization studies against each other
Example:

   benchmark = Benchmark('num_average',6);
   benchmark.add_study(study1);
   benchmark.add_study(study2);
   studies = benchmark.studies();
   for i_study = 1:length(studies)
       study = studies{i_study};
       while(not(study.is_done))
           suggestion = study.get_suggestion();
           observation = evaluate(suggestion.sample);
           study.add_observation(observation,suggestion.id);
        end
        benchmark.add_study_results(study);
   end
   data = benchmark.get_data('x_type','num_evaluations','y_type','objective',...
                 'average_type','mean');
   plots = [];
   for ii=1:length(data.names)
        X = data.X(ii,:);
        Y = data.Y(ii,:);
        plots(end+1) = plot(X,Y,'LineWidth',1.5);
   end
   legend(plots, data.names{:});


Methods:

    add_study

        Purpose: Adds a study to the benchmark.
        Usage: benchmark.add_study(study1);
        Input:
           study: A study object.

    add_study_results

        Purpose: Adds the results of a benchmark study at the end
            of an optimization run
        Usage: benchmark.add_study_results(study1);
        Input: A study object after the study was run

    studies

        Purpose: Returns a list of studies to be run for the benchmark
        Usage: studies = benchmark.studies()

    get_data

        Purpose: Get benchmark data
        Usage: data = benchmark.get_data('x_type','num_evaluations',...
                      'y_type','objective','average_type','mean');
        Input:
           x_type: Data on x-axis. Can be either 'num_evaluations' or 'time'
           y_type: Data type on y-axis. Can be either 'objective', 'distance',
               (i.e. accumulated minimum distance off all samples to overall
               minimum), or 'min_distance' (i.e. distance of current minimum
               to overall minimum).
           y_type: Data type on y-axis. Can be either 'objective' or
               'distance', i.e. distance to minimum.
           average_type: Type of averaging over study runs. Can be either
               'mean' w.r.t. x-axis data or 'median' w.r.t. y-axis data
           invert: If True, the objective is multiplied by -1.
               (Parameter not available for distance average types)
           log_scale: If True, the ouput of Y and sdev are determined as
               mean and standard deviations of the natural logarithm of the
               considered y_type.
           minimum: Vector with minimum position. (Only available for
                distance average types)
           scales: Vector with positive weights for scaling distance in
                different directions. (Only available for distance average types)
           norm: Order of distance norm as defined in
                numpy.linalg.norm. (Only available for distance average types)
           num_samples: Number of samples on y-axis. (Only available for
                median average type or time on x-axis)