Automated Hyperparameter Optimization Competition

Related tags

Deep LearningTHPO_Kit_2021
Overview

QQ浏览器2021AI算法大赛 - 自动超参数优化竞赛

ACM CIKM 2021 AnalyticCup

在信息流推荐业务场景中普遍存在模型或策略效果依赖于“超参数”的问题，而“超参数"的设定往往依赖人工经验调参，不仅效率低下维护成本高，而且难以实现更优效果。因此，本次赛题以超参数优化为主题，从真实业务场景问题出发，并基于脱敏后的数据集来评测各个参赛队伍的超参数优化算法。本赛题为超参数优化问题或黑盒优化问题：给定超参数的取值空间，每一轮可以获取一组超参数对应的Reward，要求超参数优化算法在限定的迭代轮次内找到Reward尽可能大的一组超参数，最终按照找到的最大Reward来计算排名。

1. 重要资源

2.代码结构

|--example_random_searcher  随机算法代码提交示例
|  `--searcher.py
|
|--example_bayesian_optimization 贝叶斯优化算法提交示例
|  |--requirements.txt     提交附加程序包示例
|  `--searcher.py
|
|--input                   测试评估函数数据
|  |--data-2
|  `--data-30
|
|--thpo                    thpo比赛工具包
|  |--__init__.py
|  |--abstract_searcher.py
|  |--common.py
|  |--evaluate_function.py
|  |--reward_calculation.py
|  |--run_search_one_time.py
|  `--run_search.py
|
|--main.py                 测试主程序文件
|--local_test.sh           本地测试脚本
|--prepare_submission.sh   提交代码前打包脚本
|--environments.txt        评测环境已经安装的包
`--requirements.txt        demo程序依赖的包环境

3. 快速入门

3.1 环境搭建

THPO-Kit程序工具包使用python3编写，程序依赖包在requirements.txt中，需要安装依赖包才能执行，使用pip3安装依赖包：

pip3 install -r requirements.txt

3.2 算法创建

  1. 参照 example_randon_searcher，新建一个自己算法的目录my_algo
  2. 在my_algo目录下新建searcher.py文件
  3. 在searcher.py文件里实现自己的Searcher类（文件名和类名不允许自定义）
  4. 实现 __init__ 和 suggest 函数
  5. 修改 local_test.sh，将SEARCHER修改为my_algo
  6. 执行 local_test.sh 脚本，将得到算法的执行结果

Step 1 - Step 2:[root folder]

|--my_algo
|  |--requirements.txt
|  `--searcher.py 
|--local_test.sh

Step 3 - Step 4:[searcher.py]

# 必须引入searcher抽象类，必不可少
from thpo.abstract_searcher import AbstractSearcher
from random import randint

class Searcher(AbstractSearcher):
    searcher_name = "RandomSearcher"

    def __init__(self, parameters_config, n_iter, n_suggestion):
        AbstractSearcher.__init__(self, 
                                  parameters_config, 
                                  n_iter,
                                  n_suggestion)

    def suggest(self, suggestion_history, n_suggestions=1):
        next_suggestions = []
        for i in range(n_suggestions):
            next_suggest = {
                name: 
                conf["coords"][randint(0,len(conf["coords"])-1)]
                for name, conf in self.parameters_config.items()
            }
            next_suggestions.append(next_suggest)
        return next_suggestions

Step 5:[local_test.sh]

SEARCHER="my_algo"

3.3 本地运行

执行脚本local_test.sh进行本地评测

./local_test.sh

执行结果：

====================== run search result ========================
 err_code:  0  err_msg:  
========================= iters means ===========================
func: data-2 iteration best: [25.24271821 26.36435157 12.77928619 10.19180929 11.3147711  10.17430656
 12.77928619 27.79752169 26.36793589 11.12007615]
func: data-30 iteration best: [-0.95264345 -0.27725879 -0.36873091 -0.68088963 -0.28840479 -0.50006427
 -0.32088949 -0.78627201 -0.53204227 -0.98427191]
========================= fianl score ============================
example_bayesian_optimization final score:  0.47173337831255463
==================================================================

3.4 提交比赛代码

使用prepare_submission.sh 脚本打包，提交打包后的searcher程序包到比赛代码提交入口。

./prepare_submission.sh example_random_searcher

执行结果：

upload_example_random_searcher_08131917
  adding: requirements.txt (stored 0%)
  adding: searcher.py (deflated 66%)
----------------------------------------------------------------
Built achive for upload
Archive:  ./upload_example_random_searcher_08131917.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  08-13-2021 19:17   requirements.txt
     3767  08-13-2021 19:17   searcher.py
---------                     -------
     3767                     2 files
For scoring, upload upload_example_random_searcher_08131917.zip at address:
https://algo.browser.qq.com/

QQ Browser 2021 AI Algorithm Competiton - Automated Hyperparameter Optimization Contest

ACM CIKM 2021 AnalyticCup

The choices of hyperparameters have critical effects on models or strategies in recommendation systems. But the hyperparameters are mostly chosen based on experience, which brings high maintenance costs and sub-optimal results. Thus, this track aims at automated hyperparameters optimization based on anonymized realistic industrial tasks and datasets. Given the space of all possible hyperparameters' values, a reward could be achieved with a set of hyperparameters in each iteration. The participants are asked to maximize the reward within a given limit of iterations with a hyperparameters optimization algorithm. The final rank of the participants will be the rank of their maximum reward.

1.Resource

2.Repo structure

|--example_random_searcher   	    # example of random search
|  `--searcher.py
|
|--example_bayesian_optimization    # example of bayesian optimization
|  |--requirements.txt              # extra paackge requirement
|  `--searcher.py
|
|--input                            # testcases
|  |--data-2
|  `--data-30
|
|--thpo                             # thpo-kit
|  |--__init__.py
|  |--abstract_searcher.py
|  |--common.py
|  |--evaluate_function.py
|  |--reward_calculation.py
|  |--run_search_one_time.py
|  `--run_search.py
|
|--main.py                          # main
|--local_test.sh                    # script for local test
|--prepare_submission.sh            # script for submission
|--environments.txt                 # packages installed in remote envrionment
`--requirements.txt                 # demo requirements

3. Quick start

3.1 Environment setup

The THPO-Kit program toolkit is written in python3. The program dependency packages are in requirements.txt, and the dependency packages needs to be installed to execute scripts. Use pip3 to install the dependency package:

pip3 install -r requirements.txt

3.2 Create a searcher

  1. Refer to example_randon_searcher, create a new directory my_algo for your algorithm
  2. Create a new searcher.py file in the my_algo directory
  3. Implement your own Searcher class in the searcher.py file (the file name and class name are not allowed to be customized)
  4. Implement __init__ and suggest functions
  5. Modify local_test.sh and change SEARCHER to my_algo
  6. Execute the local_test.sh script to get the results of the algorithm

Step 1 - Step 2:[root folder]

|--my_algo
|  |--requirements.txt
|  `--searcher.py 
|--local_test.sh

Step 3 - Step 4:[searcher.py]

# MUST import AbstractSearcher from thpo.abstract_searcher
from thpo.abstract_searcher import AbstractSearcher
from random import randint

class Searcher(AbstractSearcher):
    searcher_name = "RandomSearcher"

    def __init__(self, parameters_config, n_iter, n_suggestion):
        AbstractSearcher.__init__(self, 
                                  parameters_config, 
                                  n_iter,
                                  n_suggestion)

    def suggest(self, suggestion_history, n_suggestions=1):
        next_suggestions = []
        for i in range(n_suggestions):
            next_suggest = {
                name: 
                conf["coords"][randint(0,len(conf["coords"])-1)]
                for name, conf in self.parameters_config.items()
            }
            next_suggestions.append(next_suggest)
        return next_suggestions

Step 5:[local_test.sh]

SEARCHER="my_algo"

3.3 Local test

Execute the script local_test.sh for local evaluation

./local_test.sh

Execution output：

====================== run search result ========================
 err_code:  0  err_msg:  
========================= iters means ===========================
func: data-2 iteration best: [25.24271821 26.36435157 12.77928619 10.19180929 11.3147711  10.17430656
 12.77928619 27.79752169 26.36793589 11.12007615]
func: data-30 iteration best: [-0.95264345 -0.27725879 -0.36873091 -0.68088963 -0.28840479 -0.50006427
 -0.32088949 -0.78627201 -0.53204227 -0.98427191]
========================= fianl score ============================
example_bayesian_optimization final score:  0.47173337831255463
==================================================================

3.4 Submission

Use prepare_submission.sh script to create a zip file, and submit the zip file to competition website Code submission entry.

./prepare_submission.sh example_random_searcher

Execution output:

upload_example_random_searcher_08131917
  adding: requirements.txt (stored 0%)
  adding: searcher.py (deflated 66%)
----------------------------------------------------------------
Built achive for upload
Archive:  ./upload_example_random_searcher_08131917.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  08-13-2021 19:17   requirements.txt
     3767  08-13-2021 19:17   searcher.py
---------                     -------
     3767                     2 files
For scoring, upload upload_example_random_searcher_08131917.zip at address:
https://algo.browser.qq.com/
Owner
GitHub Repository
Pure python implementation reverse-mode automatic differentiation

MiniGrad A minimal implementation of reverse-mode automatic differentiation (a.k.a. autograd / backpropagation) in pure Python. Inspired by Andrej Kar

Kenny Song 76 Sep 12, 2022
A Haskell kernel for IPython.

IHaskell You can now try IHaskell directly in your browser at CoCalc or mybinder.org. Alternatively, watch a talk and demo showing off IHaskell featur

Andrew Gibiansky 2.4k Dec 29, 2022
A Pytorch implementation of "Manifold Matching via Deep Metric Learning for Generative Modeling" (ICCV 2021)

Manifold Matching via Deep Metric Learning for Generative Modeling A Pytorch implementation of "Manifold Matching via Deep Metric Learning for Generat

69 Dec 10, 2022
MonoScene: Monocular 3D Semantic Scene Completion

MonoScene: Monocular 3D Semantic Scene Completion MonoScene: Monocular 3D Semantic Scene Completion] [arXiv + supp] | [Project page] Anh-Quan Cao, Rao

298 Jan 08, 2023
Codeflare - Scale complex AI/ML pipelines anywhere

Scale complex AI/ML pipelines anywhere CodeFlare is a framework to simplify the integration, scaling and acceleration of complex multi-step analytics

CodeFlare 169 Nov 29, 2022
Official implementation of CVPR2020 paper "Deep Generative Model for Robust Imbalance Classification"

Deep Generative Model for Robust Imbalance Classification Deep Generative Model for Robust Imbalance Classification Xinyue Wang, Yilin Lyu, Liping Jin

9 Nov 01, 2022
GuideDog is an AI/ML-based mobile app designed to assist the lives of the visually impaired, 100% voice-controlled

Guidedog Authors: Kyuhee Jo, Steven Gunarso, Jacky Wang, Raghav Sharma GuideDog is an AI/ML-based mobile app designed to assist the lives of the visua

Kyuhee Jo 5 Nov 24, 2021
Ladder Variational Autoencoders (LVAE) in PyTorch

Ladder Variational Autoencoders (LVAE) PyTorch implementation of Ladder Variational Autoencoders (LVAE) [1]: where the variational distributions q at

Andrea Dittadi 63 Dec 22, 2022
Compositional and Parameter-Efficient Representations for Large Knowledge Graphs

NodePiece - Compositional and Parameter-Efficient Representations for Large Knowledge Graphs NodePiece is a "tokenizer" for reducing entity vocabulary

Michael Galkin 107 Jan 04, 2023
PyTorch implementation of popular datasets and models in remote sensing

PyTorch Remote Sensing (torchrs) (WIP) PyTorch implementation of popular datasets and models in remote sensing tasks (Change Detection, Image Super Re

isaac 222 Dec 28, 2022
This project deploys a yolo fastest model in the form of tflite on raspberry 3b+. The model is from another repository of mine called -Trash-Classification-Car

Deploy-yolo-fastest-tflite-on-raspberry 觉得有用的话可以顺手点个star嗷 这个项目将垃圾分类小车中的tflite模型移植到了树莓派3b+上面。 该项目主要是为了记录在树莓派部署yolo fastest tflite的流程 (之后有时间会尝试用C++部署来提升

7 Aug 16, 2022
From the basics to slightly more interesting applications of Tensorflow

TensorFlow Tutorials You can find python source code under the python directory, and associated notebooks under notebooks. Source code Description 1 b

Parag K Mital 5.6k Jan 09, 2023
[ArXiv 2021] Data-Efficient Instance Generation from Instance Discrimination

InsGen - Data-Efficient Instance Generation from Instance Discrimination Data-Efficient Instance Generation from Instance Discrimination Ceyuan Yang,

GenForce: May Generative Force Be with You 93 Dec 25, 2022
HistoKT: Cross Knowledge Transfer in Computational Pathology

HistoKT: Cross Knowledge Transfer in Computational Pathology Exciting News! HistoKT has been accepted to ICASSP 2022. HistoKT: Cross Knowledge Transfe

Mahdi S. Hosseini 5 Jan 05, 2023
MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks

MEAL-V2 This is the official pytorch implementation of our paper: "MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tric

Zhiqiang Shen 653 Dec 19, 2022
Fedlearn支持前沿算法研发的Python工具库 | Fedlearn algorithm toolkit for researchers

FedLearn-algo Installation Development Environment Checklist python3 (3.6 or 3.7) is required. To configure and check the development environment is c

89 Nov 14, 2022
Pytorch implementation for "Adversarial Robustness under Long-Tailed Distribution" (CVPR 2021 Oral)

Adversarial Long-Tail This repository contains the PyTorch implementation of the paper: Adversarial Robustness under Long-Tailed Distribution, CVPR 20

Tong WU 89 Dec 15, 2022
This is a collection of simple PyTorch implementations of neural networks and related algorithms. These implementations are documented with explanations,

labml.ai Deep Learning Paper Implementations This is a collection of simple PyTorch implementations of neural networks and related algorithms. These i

labml.ai 16.4k Jan 09, 2023
Image morphing without reference points by applying warp maps and optimizing over them.

Differentiable Morphing Image morphing without reference points by applying warp maps and optimizing over them. Differentiable Morphing is machine lea

Alex K 380 Dec 19, 2022
Segmentation Training Pipeline

Segmentation Training Pipeline This package is a part of Musket ML framework. Reasons to use Segmentation Pipeline Segmentation Pipeline was developed

Musket ML 52 Dec 12, 2022