NN Template
Generic template to bootstrap your PyTorch project. Click on Use this Template and avoid writing boilerplate code for:
- PyTorch Lightning, lightweight PyTorch wrapper for high-performance AI research.
- Hydra, a framework for elegantly configuring complex applications.
- DVC, track large files, directories, or ML models. Think "Git for data".
- Weights and Biases, organize and analyze machine learning experiments. (educational account available)
nn-template is opinionated so you don't have to be. If you use this template, please add to your
README.
Usage Examples
Checkout the mwe branch to view a minimum working example on MNIST.
Structure
.
├── conf # Hydra compositional config
│ ├── default.yaml # current experiment configuration
│ ├── data
│ ├── hydra
│ ├── logging
│ ├── model
│ ├── optim
│ └── train
├── data # datasets
├── experiments # local logs
├── README.md
├── requirements.txt # basic requirements
└── src
├── common # common Python modules
├── pl_data # PyTorch Lightning datamodules and datasets
├── pl_modules # PyTorch Lightning modules
└── run.py # entry point to run current conf
Data Version Control
DVC runs alongside git and uses the current commit hash to version control the data.
Initialize the dvc repository:
$ dvc init
To start tracking a file or directory, use dvc add:
$ dvc add data/ImageNet
DVC stores information about the added file (or a directory) in a special .dvc file named data/ImageNet.dvc, a small text file with a human-readable format. This file can be easily versioned like source code with Git, as a placeholder for the original data (which gets listed in .gitignore):
git add data/ImageNet.dvc data/.gitignore
git commit -m "Add raw data"
Making changes
When you make a change to a file or directory, run dvc add again to track the latest version:
$ dvc add data/ImageNet
Switching between versions
The regular workflow is to use git checkout first to switch a branch, checkout a commit, or a revision of a .dvc file, and then run dvc checkout to sync data:
$ git checkout <...>
$ dvc checkout
Read more in the docs!
Weights and Biases
Weights & Biases helps you keep track of your machine learning projects. Use tools to log hyperparameters and output metrics from your runs, then visualize and compare results and quickly share findings with your colleagues.
This is an example of a simple dashboard.
Quickstart
Login to your wandb account, running once wandb login. Configure the logging in conf/logging/*.
Read more in the docs. Particularly useful the log method, accessible from inside a PyTorch Lightning module with self.logger.experiment.log.
W&B is our logger of choice, but that is a purely subjective decision. Since we are using Lightning, you can replace
wandbwith the logger you prefer (you can even build your own). More about Lightning loggers here.
Hydra
Hydra is an open-source Python framework that simplifies the development of research and other complex applications. The key feature is the ability to dynamically create a hierarchical configuration by composition and override it through config files and the command line. The name Hydra comes from its ability to run multiple similar jobs - much like a Hydra with multiple heads.
The basic functionalities are intuitive: it is enough to change the configuration files in conf/* accordingly to your preferences. Everything will be logged in wandb automatically.
Consider creating new root configurations conf/myawesomeexp.yaml instead of always using the default conf/default.yaml.
Sweeps
You can easily perform hyperparameters sweeps, which override the configuration defined in /conf/*.
The easiest one is the grid-search. It executes the code with every possible combinations of the specified hyperparameters:
PYTHONPATH=. python src/run.py -m optim.optimizer.lr=0.02,0.002,0.0002 optim.lr_scheduler.T_mult=1,2 optim.optimizer.weight_decay=0,1e-5
You can explore aggregate statistics or compare and analyze each run in the W&B dashboard.
We recommend to go through at least the Basic Tutorial, and the docs about Instantiating objects with Hydra.
PyTorch Lightning
Lightning makes coding complex networks simple. It is not a high level framework like keras, but forces a neat code organization and encapsulation.
You should be somewhat familiar with PyTorch and PyTorch Lightning before using this template.
Environment Variables
System specific variables (e.g. absolute paths to datasets) should not be under version control, otherwise there will be conflicts between different users.
The best way to handle system specific variables is through environment variables.
You can define new environment variables in a .env file in the project root. A copy of this file (e.g. .env.template) can be under version control to ease new project configurations.
To define a new variable write inside .env:
export MY_VAR=/home/user/my_system_path
You can dynamically resolve the variable name from Python code with:
get_env('MY_VAR')
and in the Hydra .yaml configuration files with:
${env:MY_VAR}
1.4k Jan 1, 2023
1 Dec 20, 2021
789 Jan 4, 2023
21 Apr 9, 2022
48 Sep 27, 2022
6 Mar 17, 2022
378 Dec 30, 2022
47 Nov 22, 2022
![[ICCV2021] IICNet: A Generic Framework for Reversible Image Conversion](https://github.com/felixcheng97/IICNet/raw/main/figures/front.png)
55 Dec 6, 2022
15 Jun 11, 2022
14 Dec 15, 2022
87 Dec 20, 2022
8 Oct 08, 2022
339 Oct 27, 2022
25 Dec 12, 2022
0 Feb 26, 2022
28 Dec 15, 2022
11 Mar 14, 2022
1.4k Jan 02, 2023
4 Jun 01, 2022
3.2k Jan 02, 2023
107 Jan 08, 2023
66 Dec 26, 2022
1 Apr 10, 2022
12 Feb 14, 2022
0 Dec 13, 2021
266 Dec 27, 2022
10 Dec 07, 2022
20 Nov 21, 2022