An experimental technique for efficiently exploring neural architectures.

Overview

SMASH: One-Shot Model Architecture Search through HyperNetworks

An experimental technique for efficiently exploring neural architectures.

SMASHGIF

This repository contains code for the SMASH paper and video.

SMASH bypasses the need for fully training candidate models by learning an auxiliary HyperNet to approximate model weights, allowing for rapid comparison of a wide range of network architectures at the cost of a single training run.

Installation

To run this script, you will need PyTorch and a CUDA-capable GPU. If you wish to run it on CPU, just remove all the .cuda() calls.

Note that this code was written in PyTorch 0.12, and is not guaranteed to work on 0.2 until next week when I get a chance to update my own version. Please also be aware that, while thoroughly commented, this is research code for a heckishly complex project. I'll be doing more cleanup work to improve legibility soon.

Running

To run with default parameters, simply call

python train.py

This will by default train a SMASH net with nominally the same parametric budget as a WRN-40-4. Note that validation scores during training are calculated using a random architecture for each batch, and are therefore sort of an "average" measure.

After training, to sample and evaluate SMASH scores, call

python eval.py --SMASH=YOUR_MODEL_NAME_HERE_.pth

This will by default sample 500 random architectures, then perturb the best-found architecture 100 times, then employ a sort of Markov Chain to further perturb the best found architecture.

To select the best architecture and train a resulting net, then call

python train.py --SMASH=YOUR_MODEL_NAME_HERE_archs.npz

This will by default take the best architectuure There are lots of different options, including a number of experimental settings such as architectural gradient descent by proxy, in-op multiplicative gating, variable nonlinearities, setting specific op configuration types. Take a look at the train_parser in utils.py for details, though note that some of these weirder ones may be deprecated.

This code has boilerplate for loading Imagenet32x32 and ModelNet, but doesn't download or preprocess them on its own. It supports model parallelism on a single node, and half-precision training, though simple weightnorm is unstable in FP16 so you probably can't train a SMASH network with it.

Notes

This README doc is in very early stages, and will be updated soon.

Acknowledgments

Owner
Andy Brock
Dimensionality Diabolist
Andy Brock
Simple Python version management

Simple Python Version Management: pyenv pyenv lets you easily switch between multiple versions of Python. It's simple, unobtrusive, and follows the UN

pyenv 30.1k Jan 04, 2023
This tool is used to install `pyenv` and friends.

pyenv installer This tool installs pyenv and friends. It is inspired by rbenv-installer. Prerequisites In general, compiling your own Python interpret

pyenv 3.5k Jan 03, 2023
A simple but powerful Python packer to run any project with any virtualenv dependencies anywhwere.

PyEmpaq A simple but powerful Python packer to run any project with any virtualenv dependencies anywhwere. With PyEmpaq you can convert any Python pro

Facundo Batista 23 Sep 22, 2022
Manage python virtual environments on the working notebook server

notebook-environments Manage python virtual environments on the working notebook server. Installation It is recommended to use this package together w

Vladislav Punko 44 Nov 02, 2022
Run a command in the named virtualenv.

Vex Run a command in the named virtualenv. vex is an alternative to virtualenv's source wherever/bin/activate and deactivate, and virtualenvwrapper's

Sasha Hart 374 Dec 21, 2022
to-requirements.txt allows to automatically add and delete modules to requirements.txt installing them using pip.

to-requirements.txt | Automatically update requirements.txt to-requirements.txt allows to automatically add and delete modules to requirements.txt ins

Ilya 16 Dec 29, 2022
Ready-to-run Docker images containing Jupyter applications

Jupyter Docker Stacks are a set of ready-to-run Docker images containing Jupyter applications and interactive computing tools.

Project Jupyter 7k Jan 03, 2023
A fast and easy python virtual environment creator for linux with some pre-installed libraries.

python-venv-creator A fast and easy python virtual environment created for linux with some optional pre-installed libraries. Dependencies: The followi

2 Apr 19, 2022
Define requirements inside your python code and scriptenv makes them ready to import.

scriptenv Define requirements inside your python code and scriptenv makes them ready to import. Getting Started Install scriptenv $ pip install script

Stefan Hoelzl 6 Nov 04, 2022
Python Development Workflow for Humans.

Pipenv: Python Development Workflow for Humans [ ~ Dependency Scanning by PyUp.io ~ ] Pipenv is a tool that aims to bring the best of all packaging wo

Python Packaging Authority 23.5k Jan 01, 2023
Virtual Python Environment builder

virtualenv A tool for creating isolated virtual python environments. Installation Documentation Changelog Issues PyPI Github Code of Conduct Everyone

Python Packaging Authority 4.3k Dec 30, 2022
The GNS3 server manages emulators such as Dynamips, VirtualBox or Qemu/KVM

GNS3-server This is the GNS3 server repository. The GNS3 server manages emulators such as Dynamips, VirtualBox or Qemu/KVM. Clients like the GNS3 GUI

GNS3 644 Dec 30, 2022
Python virtualenvs in Debian packages

dh-virtualenv Contents Overview Presentations, Blogs & Other Resources Using dh-virtualenv How does it work? Running tests Building the package in a D

Spotify 1.5k Jan 02, 2023
A pythonic interface to high-throughput virtual screening software

pyscreener A pythonic interface to high-throughput virtual screening software Overview This repository contains the source of pyscreener, both a libra

56 Dec 15, 2022
An experimental technique for efficiently exploring neural architectures.

SMASH: One-Shot Model Architecture Search through HyperNetworks An experimental technique for efficiently exploring neural architectures. This reposit

Andy Brock 478 Aug 04, 2022
PyDynamica is a freely available agent-based economy simulation

PyDynamica PyDynamica is a pure python implementation of Sociodynamica, a virtual environment to simulate a simple economy with minimal dependencies.

4 Sep 10, 2022
macOS development environment setup: Setting up a new developer machine can be an ad-hoc, manual, and time-consuming process.

dev-setup Motivation Setting up a new developer machine can be an ad-hoc, manual, and time-consuming process. dev-setup aims to simplify the process w

Donne Martin 5.9k Jan 02, 2023
a pyenv plugin to manage virtualenv (a.k.a. python-virtualenv)

pyenv-virtualenv pyenv-virtualenv is a pyenv plugin that provides features to manage virtualenvs and conda environments for Python on UNIX-like system

pyenv 5.3k Jan 08, 2023
A PipEnv Environment Switcher

Pipes Pipenv Environment Switcher ⚡ Overview Pipes is a Pipenv companion CLI tool that provides a quick way to jump between your pipenv powered projec

Gui Talarico 131 Sep 04, 2022
Fish shell tool for managing Python virtual environments

VirtualFish VirtualFish is a Python virtual environment manager for the Fish shell. You can get started by reading the documentation. (It’s quite shor

Justin Mayer 968 Dec 24, 2022