A typical use of Bayesian optimization is in sequential mode, where one does only one experiment (function evaluation) at a time. However, one could hope to reduce the overall wall clock time by running a few function evaluations / experiments at a time.

Approaches that try such parallelisation typically yield speed up of convergence wrt wall clock time somewhere proportional to the number of experiments run in parallel, eg:

https://arxiv.org/pdf/1602.05149.pdf (see page 13) https://people.orie.cornell.edu/pfrazier/Presentations/2015.02.CMU.pdf (see slide 32)

Currently such feature is not supported by skopt explicitly (correct me if I am wrong), but one could nonetheless attempt some simple approaches already. One such approach is to sample multiple points that minimize current surrogate, more precisely something like calling the below function multiple times, and evaluating results in parallel:

def get_point(X,Y):
    """
    :param X: All points evaluated so far
    :param Y: Objective values at corresponding points
    :return: next point to evaluate
    """
    opt = Optimizer(dimensions=dimensions,base_estimator=some_surrogate)
    opt.tell(X, Y)
    return opt.ask()

Assume for simplicity that at a single sequential step I can run multiple experiments in parallel. Even with simplistic approach above, I seem to be able to achieve speedups proportional to what is shown in the literature when I run multiple experiments in parallel. Consider example of optimizing 2 parameters of decision tree, averaged over 64 repeats (green: 1 experiment / step, blue 2, red: 4):

You can reproduce these results with /examples/skopt_parallel_example.py.

I have more examples with similar results, for more dimensions (up to 11), other surrogates, other toy and “real” optimization problems etc. I also tried a bit more complicated approaches for obtaining points to evaluate in parallel, but they did not perform much better.

In light of this, I propose to make a minimalistic ipynb example describing such approach. While I cannot make theoretical argument that it should always yield speedup (probably there are counterexamples), this seems to be useful in practice and at least running multiple experiments instead of one should not do harm. Let me know if you would be interested or if someone is already working on something like this.

Fixes https://github.com/scikit-optimize/scikit-optimize/issues/978 Fixes https://github.com/scikit-optimize/scikit-optimize/issues/987 Fixes https://github.com/scikit-optimize/scikit-optimize/issues/927 Fixes https://github.com/scikit-optimize/scikit-optimize/issues/990 Fixes https://github.com/scikit-optimize/scikit-optimize/issues/981 via updated scikit-learn

Fixes https://github.com/scikit-optimize/scikit-optimize/issues/1006

Fixes https://github.com/scikit-optimize/scikit-optimize/issues/718

Closes https://github.com/scikit-optimize/scikit-optimize/pull/951

This pull-request partially solves issue #576.

I have added some more support for named dimensions in the search-space, as well as two plotting functions. I don't have time to fix your existing plotting functions but I have made TODO comments for what needs to be done in plots.py

Before I start, let me repeat that I really like your library! So please take the following as friendly and constructive feedback. I spent the whole Saturday from morning to evening working on this and half of this Sunday. A large part of yesterday was just trying to understand your code because of the severe lack of comments. I spent this much effort because I think your library is important and I hope you will polish it some more.

Please note that I have to travel tomorrow so it might be a while before I can respond if you have questions / comments to this PR.

How to test this pull-request

The following code uses the additions of this PR by simulating an objective function for tuning the hyper-parameters of a simple neural network. It will save a few plots to disk and print various info. It should work if you copy this to a python-file and run it.

(And yes! My code is also ugly when I'm writing it. I polish it when it is done and works. But this code is not for release :-)

import numpy as np
from math import exp

from skopt import gp_minimize
from skopt.space import Real, Categorical, Integer
from skopt.plots import plot_histogram, plot_contour, plot_objective

dim_learning_rate = Real(name='learning_rate', low=1e-6, high=1e-2, prior='log-uniform')
dim_num_dense_layers = Integer(name='num_dense_layers', low=1, high=5)
dim_num_dense_nodes = Integer(name='num_dense_nodes', low=5, high=512)
dim_activation = Categorical(name='activation', categories=['relu', 'sigmoid'])

dimensions = [dim_learning_rate,
              dim_num_dense_layers,
              dim_num_dense_nodes,
              dim_activation]

default_parameters = [1e-4, 1, 64, 'relu']


def model_fitness(x):
    learning_rate, num_dense_layers, num_dense_nodes, activation = x

    fitness = ((exp(learning_rate) - 1.0) * 1000) ** 2 + \
              (num_dense_layers) ** 2 + \
              (num_dense_nodes/100) ** 2

    fitness *= 1.0 + 0.1 * np.random.rand()

    if activation == 'sigmoid':
        fitness += 10

    return fitness

print(model_fitness(x=default_parameters))

search_result = gp_minimize(func=model_fitness,
                            dimensions=dimensions,
                            n_calls=40,
                            x0=default_parameters)

print(search_result.x)
print(search_result.fun)

for fitness, x in sorted(zip(search_result.func_vals, search_result.x_iters)):
    print(fitness, x)

space = search_result.space

print(search_result.x_iters)

print(space.to_dict(x=default_parameters))

print("Plotting now ...")

# fig = plot_histogram(result=search_result, dimension_id='activation')
fig = plot_histogram(result=search_result, dimension_id='learning_rate', bins=20)
# fig = plot_histogram(result=search_result, dimension_id='num_dense_layers', bins=20)
# fig = plot_histogram(result=search_result, dimension_id='num_dense_nodes', bins=20)
fig.savefig('histogram.png')

fig = plot_contour(result=search_result,
                     dimension_id1='learning_rate',
                     dimension_id2='num_dense_nodes')
fig.savefig('contour_learning_rate_vs_num_dense_nodes.png')

fig = plot_contour(result=search_result,
                     dimension_id1='num_dense_layers',
                     dimension_id2='num_dense_nodes')
fig.savefig('contour_num_dense_layers_vs_num_dense_nodes.png')

print("Done plotting!")

Comments about this PR

• I am completely new to your library so it is possible I have misunderstood how something works. Please feel free to make changes to the code I have added. But please keep the good commenting and coding style as it will help people in the future who have to understand and maintain this code.

• I am not familiar with the syntax you use for doc-strings. I have tried to imitate it. I normally use PyCharm which has a different syntax for doc-strings. It is one of the things I really don't like about Python, that you don't have to specify data-types in the function-declaration, but then you often have to do it in the doc-strings using a strange syntax. It's a really poor language-design. But anyway, please check if my docstrings are correct.

• Some of the functionality I have added such as space.get_dimensions() is not actually used by my code. It is intended to be helpful for when you fix the other plotting functions to support named dimensions and search-spaces with categories, etc.

• I added the function space.to_dict(). There is a somewhat related function in utils.point_asdict(). But I prefer the name to_dict() which is also used by Pandas. I realize it is bad to have two naming conventions in one library. Could I convince you to change the name of asdict()? :-)

Comments about your existing code

You REALLY need to make better comments in your code! It is very difficult for others to understand what you want to do and how everything fits together. When there is a bug we have to guess what your intentions are with the code. If you don't have time to maintain this library in the future, it will probably become abandoned because others will find it easier to just start a new project rather than to try and understand and build on yours. That would be a pity, because you have some outstanding ideas in your library!

For example, this is one of the few comments in plots.py:

# plots on the diagonal are special, like Texas. They have
# their own range so do not mess with them.

I had to read this numerous times to finally understand that it was probably a joke about Texas USA, although I'm still not 100% sure. You really have to be a master at writing comments in order to pull off jokes, otherwise it will be confusing and harmful to the people who try to understand your code. I personally don't write jokes in code and I am very reluctant to make jokes in my video tutorials as well, for the same reason.

Please look at the coding and commenting style in the functions I have added. You may think that I write too many comments, but what I am aiming for is to explain as much as possible in plain English. Every time you have a function call into skopt or matplotlib or some other package, the reader needs to look up the semantics of that function. If that is also poorly documented, reading your code becomes like solving a giant puzzle. It takes immense effort and bugs are allowed to propagate very easily. When everything is commented in plain English, you can read and understand the code very quickly.

Another issue is your actual coding style. I found bugs as well as code that was correct but not very good. For example, these nested loops are typical in plots.py:

for i in range(space.n_dims):  # rows
    for j in range(space.n_dims):  # columns
        # ...
        if i != j:
            # ...
            if j > 0:
                # ...

Firstly, you should have a local variable called n_dims instead of always referring to space.n_dims. Secondly, your nesting depth is sometimes 4 or 5 inside these for-loops, because you haven't been very precise in your loop-conditions. You can rewrite the for-loops to something like the following. It drastically reduces the nesting-depth and also only executes the loops that are necessary:

for i in range(n_dims):
    # Do something for the diagonal case.

    for j in range(i):
        # Do something for the case where j<i.

    for j in range(i+1, n_dims):
        # Do something for the case where j>i.

You also have this code in plots.py which is incredibly difficult to understand because of the multiple nestings and list-comprehension:

diagonal_ylim = (np.min([ax[i, i].get_ylim()[0]
                         for i in range(space.n_dims)]),
                 np.max([ax[i, i].get_ylim()[1]
                         for i in range(space.n_dims)]))

It is better to split code like this into several lines to avoid the complicated nesting and also make it easier to step-debug. I added a revised version to the code, although I've commented it out for now so you can test it properly.

Another thing is that it is quite confusing how the Space and Dimension objects interact and wrap each other, e.g. regarding the transformation. It appears that the transformation and its inverse may be carried out repeatedly to nullify each other, although I'm not sure.

Doing anything in the __init__ method of a sklearn estimator will most of the time break his compatibility with sklearn's clone, get_params and set_params, or the ability to fit it several times. It is in fact the case here, it would break if used in a Pipeline or other meta-objects in sklearn.

This PR is a quick (for now, dirty) proposal to fix the compatibility issues.

I've basically moved every lines of code that are not self.foo = foo, where foo is an arg, inside the fit() and add_space() methods.

Fixes #550 #551

Added docs and some minor cosmetics. Removed prior for now so that we can compare our results with a randomized search where we have some prior knowledge about the candidates. (I understand it might be useful but YAGNI)

Fixes #428

Getting started with a documentation marathon.

• [ ] module doc strings
• [ ] adapt doc building so sphinx docs are pushed to github.io
• [ ] structure the API docs
• [ ] integrate example notebooks
• [ ] extend quickstart material
• [ ] add two paragraph summary of when skopt is the right tool

I would like to get the 0.1 release out before school starts again (i.e September). This is just a parent issue to track the blockers.

• [x] Consistent and backward-compatible API. Addressed by https://github.com/scikit-optimize/scikit-optimize/pull/75
• [x] SMAC https://github.com/scikit-optimize/scikit-optimize/issues/57
• (Local search technique that performs better than random sampling on piecewise constant predict functions (https://github.com/scikit-optimize/scikit-optimize/issues/74), postponed till we have a conclusion in #109)
• [x] Examples (https://github.com/scikit-optimize/scikit-optimize/issues/65)
• [x] Support for Python 2.7 (#87)
• [x] Consistent return types #86
• [x] Name collision #76 (punting for now)
• [x] Need a logo #107 (code speaks louder than images, no logo required)
• [ ] release mechanics #133
• [x] better defaults #166
• [x] merge #145
• [x] merge #169
• [x] maybe merge #162 (nice to have but don't hold the 🚄 for it)
• [x] stop this list from getting ever longer 📋

Is there anything else?

I didn't see an example of optimizing over the hyperparams of several sklearn objects assembled into a pipeline (i.e. a pipeline generalization examples/hyperparameter-optimization.py). I took a couple hours and made a notebook showing how to do this, if that example seems useful.

It's kinda drafty and terse compared to the other (really nice!) example notebooks. I'm happy to add/modify if you can think of more stuff that might be interesting to show here.

Create a scatter plot from a OptimizeResult.

Do you know how to make the top triangle vanish or something useful we could display there?

This is the output from a run on hart6:

First part of #640.

Scope

• [x] Tooling and prerequisites: Cherry pick as much as possible from betatim's existing setup.
  • [x] Building the documentation and creating a html version should behave correctly.
  • [x] Include the napoleon plugin so that we can stay with the Numpy docstring style.
  • [x] Have the first examples show up correctly in the new sphinx docs (to verify that everything is set up correctly). (*)

(*): Restrict this to the section Top level minimization functions, mainly dummy_minimize.

Update: Almost finished. I need to look over all the changes again, then i set it to MRG. All aesthetical, content-related or structural changes are not yet done and will be processed in further PRs.

A class as discussed in #78 under the tentative name of SkoptSearchCV (discussion on what might be a better name is welcome :)

Minimalist usage example as of right now:

from skopt.wrappers import SkoptSearchCV
from skopt.space import Real, Categorical, Integer

opt = SkoptSearchCV(
    SVC(),
    [{
        'C': Real(1e-6, 1e+6, prior='log-uniform'),
        'gamma': Real(1e-6, 1e+1, prior='log-uniform'),
        'degree': Integer(1, 8),
        'kernel': Categorical(['linear', 'poly', 'rbf']),
    }],
    n_jobs=1, n_iter=32,
)

opt.fit(X_train, y_train)
print(opt.score(X_test, y_test))

If something is missing in todos let me know.

• [x] add skopt.wrappers to setup.py
• [x] implement basic wrapper similar to RandomizedSearchCV using BaseSearchCV
• [x] support multiple search spaces as list of dicts, similar to GridSearchCV
• [x] support parallel search
• [x] add draft of example usage in sklearn-wrapper
• [x] add all necessary data to cv_results_
• [x] add tests
• [x] fix the docstrings and add comments
• [x] review and improve example usage

I frequently use skopt.gp_minimize. But, since numpy version 1.24.0, it raises an exception

AttributeError: module 'numpy' has no attribute 'int'

This issue has been anticipated for some time, in the form of a deprecation warning, and it is targeted by PR #1093 and #1123. I think it is redundant for me to contribute with my own PR, since #1123 seems to solve the problem, but fails validation because of seemingly independent issues with the circleci automation.

In the meantime I will just have to manually add a constraint on the numpy version for my own pipelines, so I have no personal urgency. But I guess it could be a problem for new users, trying to figure out why the function is not working. I hope this issue will help them fix it.

As per the title, when attempting to use the ExtraTreesRegressor class an sklearn.utils._param_validation.InvalidParameterError exception is raised as below.

sklearn.utils._param_validation.InvalidParameterError: The 'criterion' parameter of ExtraTreesRegressor must be a str among {'squared_error', 'poisson', 'friedman_mse', 'absolute_error'}. Got 'mse' instead.

The ExtraTreesRegressor.criterion attribute is set using a default optional argument to the constructor with the value mse. It looks like this criterion value was removed in sklearn 1.2 as per this commit.

It should be an easy fix. Just requires changing the default argument for criterion to either friedman_mse or squared_error I guess... Not sure which one of these would be most appropriate as the default...

Hi everyone.

I would like to apply the command "skopt.optimizer()" to minimize a function with given initial points. However, I could not find any option for inserting given initial values for parameters as can be seen in the following: class skopt.optimizer.Optimizer(dimensions, base_estimator='gp', n_random_starts=None, n_initial_points=10, initial_point_generator='random', n_jobs=1, acq_func='gp_hedge', acq_optimizer='auto', random_state=None, model_queue_size=None, acq_func_kwargs=None, acq_optimizer_kwargs=None) Would you please let me know how to insert initial values using the command. Thanks

According to the documentation, setting n_jobs should set the number of cores which are used by gp_minimize. I was trying to do that with the following code:

from skopt import gp_minimize

def J(paramlist):
 	x = paramlist[0]
 	z=0.0
 	for i in range(1,500000000):
 		z+=1.0/i
 	print(z)
 	return z*x**2
  
 res = gp_minimize(J,                 
                   [(-1.0, 1.0)],
                   acq_func="EI",    
                   n_calls=1000,        
                   n_random_starts=1,  
                   random_state=1234,
                   acq_optimizer="lbfgs",
                   n_jobs=5,
                   )

However, regardless of the value of n_jobs, it only runs on a single core.

scikit-optimize version: 0.8.0, 0.8.1, 0.9.0

For all samplers in skopt/sampler/, includes Lhs, Sobol, Halton, Hammersly, Grid, the return type of sampler.generate() is list, but should it be numpy.ndarray according to the class description?

The reason is that h = space.inverse_transform(h) is called, and inverse_transform converts np.ndarray to list in the end. Add code like h = np.asarray(h) will help.

Reproduction Code:

from skopt.sampler import Lhs, Sobol, Halton, Hammersly, Grid

dimensions = [(0.0, 1.0) for i in range(2)]
size = 4
for sampler in [Lhs, Sobol, Halton, Hammersly, Grid]:
    print(sampler)
    X = sampler().generate(dimensions, size)
    print(type(X), X)

I am seeing very slow performance when using skopt.Optimizer. Here is a reproducible example, that takes ~22.5s on my laptop:

os.environ["OMP_NUM_THREADS"] = "1"
import skopt
import numpy as np
import time
noise_level =0.1
def f(x, noise_level=noise_level):
    return np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2))\
           + np.random.randn() * noise_level

def optimize(n_trials=16, n_batches=3):
    opt = skopt.Optimizer([(-2.0, 2.0)], "GP", n_initial_points=n_trials)
    for b in range(n_batches):
        xs = opt.ask(n_trials)
        ys = [f(x) for x in xs]
        opt.tell(xs, ys)

def timeit(fn):
    start = time.time()
    fn()
    end = time.time()
    print(f"Took {end-start:.2f}s")


timeit(optimize)

I understand that GPs time complexity is O(n^3), but, this still seems too slow for such a small number of trials (my real problem involves a 6 dimensional space and hundreds of trials). Note that changing the base_estimator from GP to GBRT cuts the excution time to ~6s for this toy example. What am I missing?