Generalized and Efficient Blackbox Optimization System.

我想使参数C1 在集合 {3.6, 4.0, 4.4, 4.8, 5.2, 5.6, 6.0, 7.0, 8.0}中任取一个，我尝试这样写：C1 = sp.Categorical("C1", [3.6, 4.0, 4.4, 4.8, 5.2, 5.6, 6.0, 7.0, 8.0])，但应该出问题了，想请问下应该如何书写

I can see there's a page in the docs about the transfer learning feature, however, I wasn't able to find a clear example of how to do this with OpenBox.

import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision import datasets
from torchvision.transforms import ToTensor

class NeuralNetwork(nn.Module):
    def __init__(self, feature_dims1=512, feature_dims2=256, feature_dims3=128):
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(28*28, feature_dims1),
            nn.ReLU(),
            nn.Linear(feature_dims1, feature_dims2),
            nn.ReLU(),
            nn.Linear(feature_dims2, feature_dims3),
            nn.ReLU(),
            nn.Linear(feature_dims3, 10),
        )

    def forward(self, x):
        x = self.flatten(x)
        logits = self.linear_relu_stack(x)
        return logits

def train_loop(dataloader, model, loss_fn, optimizer, device):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        X = X.to(device)
        y = y.to(device)

        # Compute prediction and loss
        pred = model(X)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")

def test_loop(dataloader, model, loss_fn, device):
    size = len(dataloader.dataset)
    num_batches = len(dataloader)
    test_loss, correct = 0, 0

    with torch.no_grad():
        for X, y in dataloader:
            X = X.to(device)
            y = y.to(device)

            pred = model(X)
            test_loss += loss_fn(pred, y).item()
            correct += (pred.argmax(1) == y).type(torch.float).sum().item()

    test_loss /= num_batches
    correct /= size
    print(f"Test Error: \n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \n")
    
    return correct

training_data = datasets.FashionMNIST(
    root="data",
    train=True,
    download=True,
    transform=ToTensor()
)

test_data = datasets.FashionMNIST(
    root="data",
    train=False,
    download=True,
    transform=ToTensor()
)

from openbox import sp

def get_configspace():
    space = sp.Space()
    learning_rate = sp.Real("learning_rate", 1e-3, 0.3, default_value=0.1, log=True)
    batch_size = sp.Int("batch_size", 32, 64)
    feature_dims1 = sp.Int("feature_dims1", 256, 512)
    feature_dims2 = sp.Int("feature_dims2", 256, 512)
    feature_dims3 = sp.Int("feature_dims3", 256, 512)
    space.add_variables([learning_rate, batch_size, feature_dims1, feature_dims2, feature_dims3])    
    return space

def objective_function(config: sp.Configuration):
    params = config.get_dictionary()
    params['epochs'] = 10
    
    train_dataloader = DataLoader(training_data, batch_size=params['batch_size'])
    test_dataloader = DataLoader(test_data, batch_size=params['batch_size'])
    
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")    
    model = NeuralNetwork(
        feature_dims1=params['feature_dims1'], 
        feature_dims2=params['feature_dims2'],
        feature_dims3=params['feature_dims3']
    )
    model = model.to(device)

    loss_fn = nn.CrossEntropyLoss()

    optimizer = torch.optim.SGD(model.parameters(), lr=params['learning_rate'])
    
    for epoch in range(params['epochs']):
        print(f"Epoch {epoch+1}\n-------------------------------")
        train_loop(train_dataloader, model, loss_fn, optimizer, device)
        correct = test_loop(test_dataloader, model, loss_fn, device)
    
    result = dict()
    result['objs'] = [1-correct, ]
    result['constraints'] = [
        params['feature_dims2'] - params['feature_dims1'], 
        params['feature_dims3'] - params['feature_dims2'], 
    ]
    
    return result

from openbox import Optimizer

# Run
opt = Optimizer(
    objective_function,
    get_configspace(),
    num_objs=1,
    num_constraints=2,
    max_runs=10,
    surrogate_type='prf',
    time_limit_per_trial=180,
    task_id='hpo',
)
history = opt.run()

history = opt.get_history()
print(history)

history.plot_convergence()

history.visualize_jupyter()

print(history.get_importance())

I add 2 constrains in results from objective_function, hope "feature_dims1 > feature_dims2 > feature_dims3", but it seems not to work as expect.

Hi there,

I was trying to plot the gaussian process model using the function advisor.surrogate_model.predict(). But it gives predictions far away from the sample points and objective function.

The advisor itself seems to be working fine with output close to the minimum, but the fit surface doesn't look right. What's the proper way of plotting the gaussian process fit?

Cheers

Hi there,

I was trying to pass all the parameters to the objective function, this includes both varying parameters to be determined and fixed parameters. I tried to use Latin Hypercube for initial sampling, and it reminds me that only int and float can be used in Latin Hypercube (constant and categorical are not accepted). So I tried to split the parameters into two parts and pass them into the objective function separately. Then I found in optimizer.parallel_smbo.wrapper() the kwargs is set to dict() and I can't pass my fixed parameters into the objective function.

Was I missing something? How do I pass additional kwargs to the objective function? I would appreciate a relevant example.

Cheers!

Describe the bug A clear and concise description of what the bug is. e.g., on which version, xxx fails. 在使用OpenBox对参数重要性排序这个功能时，我输入了一列类别型参数，程序在完成调优后无法输出参数重要性排序，会报错。具体来说，我编写了一个测试压缩算法效率的demo。参数为COMPRESS_LEVEL（整数型），COMPRESS_METHOD（类别型）。其中整数型参数取值为1~9，类别型参数取值为["bz2", "zlib", "gzip"]。具体做法是测试不同COMPRESS_LEVEL和COMPRESS_METHOD组合下，对文件的压缩效率。评价指标分别为压缩时间和压缩率（压缩后文件大小与压缩前文件大小之比）。优化器参数设置如下：
opt = Optimizer( compress, space, num_objs=2, num_constraints=0, max_runs=50, surrogate_type='auto', acq_type='ehvi', time_limit_per_trial=30, task_id='compress', ref_point=[6.97, 2.74],） 使用OpenBox时可以正常调优，但是在使用history.get_importance()函数时，无法对参数重要性进行评估，会报类型错误！

To Reproduce Steps to reproduce the behavior: 具体错误为：numpy.core._exceptions._UFuncNoLoopError: ufunc 'maximum' did not contain a loop with signature matching types (dtype('<U11'), dtype('<U11')) -> None

Expected behavior 可以正常输出特征重要性排名

Outputs and Logs If applicable, add outputs and logs to help explain your problem. 具体调优日志如下： OpenBox-compress-logs.log

Additional context 当我对类型行特征中的字符串数据转换为数字类型后，该功能又可以正常使用。所以OpenBox是否不支持对类别型参数如字符串等数据进行重要性排序呢？

您好，我启动了多个进程openbox实例，总进程数量是小于cpu核心数量的，每个openbox都是单进程模式Optimizer(....)，max_runs 设置的是500次，运行时发现，在前几十次时速度还是比较快的，但到后面会越来越慢，运行到100-300轮的时候，每次获取参数都要几十秒的时间，请问这是什么原因，怎么解决，多谢？ Hello, I have started several OPENBOX instances. The total number of processes is less than the number of CPU cores. Each OPENBOX is a single process mode optimizer (...), max_ Runs is set to 500 times. It is found that the speed is still relatively fast in the first dozens of times, but it will be slower and slower later. When running to 100-300 rounds, it takes tens of seconds to obtain parameters every time. What is the reason and how to solve it? Thank you?

Hi, if there are hundreds of hyperparams, can I use open-box to get a good result? And is there a empirical relationship between run times and number of hyperparams?

Collecting pyrfr==0.7.0 Downloading pyrfr-0.7.0.tar.gz (290 kB) -------------------------------------- 290.7/290.7 kB 1.8 MB/s eta 0:00:00 Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Building wheels for collected packages: pyrfr Building wheel for pyrfr (setup.py): started Building wheel for pyrfr (setup.py): finished with status 'error' Running setup.py clean for pyrfr Failed to build pyrfr Installing collected packages: pyrfr Running setup.py install for pyrfr: started Running setup.py install for pyrfr: finished with status 'error'

error: subprocess-exited-with-error

python setup.py bdist_wheel did not run successfully. exit code: 1

[6 lines of output] warning: build_py: byte-compiling is disabled, skipping.

cl: 命令行 warning D9002 :忽略未知选项“-std=c++11” regression_wrap.cpp C:\Users\16406\AppData\Local\Programs\Python\Python310\include\pyconfig.h(59): fatal error C1083: 无法打开包括文件: “io.h”: No such file or directory error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.34.31933\bin\HostX86\x64\cl.exe' failed with exit code 2 [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for pyrfr error: subprocess-exited-with-error

Running setup.py install for pyrfr did not run successfully. exit code: 1

[6 lines of output] C:\Users\16406\AppData\Local\Programs\Python\Python310\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( cl: 命令行 warning D9002 :忽略未知选项“-std=c++11” regression_wrap.cpp C:\Users\16406\AppData\Local\Programs\Python\Python310\include\pyconfig.h(59): fatal error C1083: 无法打开包括文件: “io.h”: No such file or directory error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.34.31933\bin\HostX86\x64\cl.exe' failed with exit code 2 [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure

Encountered error while trying to install package.

pyrfr

note: This is an issue with the package mentioned above, not pip. hint: See above for output from the failure.

Describe the bug

import math
import numpy as np
from math import prod, fabs
from openbox import sp


def mishra(config: sp.Configuration):
    config_dict = config.get_dictionary()
    ws = np.array([config_dict['w%d' % i] for i in range(3)])
    # if fabs(sum(ws)-1)>1e-6:
    #     return {
    #         "objs":[math.inf,],
    #     }

    # if sum(ws) > 1:
    #     return {
    #         "objs": [math.inf, ]
    #     }

    result = dict()
    result['objs'] = [-1 * (ws[0] * 2 + ws[1] * 3 + ws[2]), ]
    # ?
    result["constraints"] = [fabs(sum(ws) - 1) - 1e-6, ]
    return result


N = 3
df = 1 / N
params = {
    'float': {
        f"w{i}": (0, 1, df) for i in range(N)
    }
}
space = sp.Space()
space.add_variables([
    sp.Real(name, *para) for name, para in params['float'].items()
])

from openbox import Optimizer

opt = Optimizer(
    mishra,
    space,
    num_objs=1,
    num_constraints=1,
    surrogate_type='gp',
    acq_optimizer_type='random_scipy',
    max_runs=50,
    time_limit_per_trial=10,
    task_id='soc',
)
history = opt.run()

print(history)

Here is my code, I want the sum of these three parameters to be 1, I tried adding constraints of both 1-sum(ws) and sum(ws)-1 but it doesn't work

At the beginning, I only launch 12 parallel workers. Now for getting result faster，I launch 30 parallel workers. But I find that the result is getting worse in same searching times. Is this as expected?

By the way, the performance of different workers is different. Some runs very faster, maybe triple faster than the slower workers.

E.g., is the resource dimension (e.g., the number of epochs or the number of samples to train) treated as just another hyperparameter? How can one do cross-task transfer learning with multi-fidelity, e.g., how does one report xval accuracy at every epoch for previous trials?

Does OpenBox support sampling some parameters only when a certain value has been sampled by some other parameter?

E.g., for a neural net, don't sample layer3_n_filters, layer3_filter_w, layer3_filter_h if we only have 2 layers in the net (e.g., n_layers == 2).

Or is there another way to address this, e.g., can one just signal that a particular parameter combination is invalid and quit early? Does OpenBox train a separate classification model that tries to predict feasibility for each combination? Should one use unequality constraints? What is the impact on the efficiency of exploring the search space? E.g., does it assign a high cost to unfeasible combinations, and, if so, does this mean it will also assign a high prior cost to combinations on the edge of feasibility?

In short, how can one treat conditional search spaces?

The project is wonderful! It helps me a lot! But I don't know how to disable the function to print information, i.e.

I don't want to print the information which makes my results hard to observe. So is there any method to solve it?