Nest - A flexible tool for building and sharing deep learning modules

Overview

Nest - A flexible tool for building and sharing deep learning modules

Nest is a flexible deep learning module manager, which aims at encouraging code reuse and sharing. It ships with a bunch of useful features, such as CLI based module management, runtime checking, and experimental task runner, etc. You can integrate Nest with PyTorch, Tensorflow, MXNet, or any deep learning framework you like that provides a python interface.

Moreover, a set of Pytorch-backend Nest modules, e.g., network trainer, data loader, optimizer, dataset, visdom logging, are already provided. More modules and framework support will be added later.


Prerequisites

  • System (tested on Ubuntu 14.04LTS, Win10, and MacOS High Sierra)
  • Python >= 3.5.4
  • Git

Installation

# directly install via pip
pip install git+https://github.com/ZhouYanzhao/Nest.git

# manually download and install
git clone https://github.com/ZhouYanzhao/Nest.git
pip install ./Nest

Basic Usage

The official website and documentation are under construction.

Create your first Nest module

  1. Create "hello.py" under your current path with the following content:

    from nest import register
    
    @register(author='Yanzhao', version='1.0.0')
    def hello_nest(name: str) -> str:
        """My first Nest module!"""
    
        return 'Hello ' + name

    Note that the type of module parameters and return values must be clearly defined. This helps the user to better understand the module, and at runtime Nest automatically checks whether each module receives and outputs as expected, thus helping you to identify potential bugs earlier.

  2. Execute the following command in your shell to verify the module:

    str # Documentation: # My first Nest module! # author: Yanzhao # module_path: /Users/yanzhao/Workspace/Nest.doc # version: 1.0.0 ">
    $ nest module list -v
    # Output:
    # 
    # 1 Nest module found.
    # [0] main.hello_nest (1.0.0) by "Yanzhao":
    #     hello_nest(
    #         name:str) -> str
    
    # Documentation:
    #     My first Nest module!
    #     author: Yanzhao
    #     module_path: /Users/yanzhao/Workspace/Nest.doc
    #     version: 1.0.0 

    Note that all modules under current path are registered under the "main" namespace.

    With the CLI tool, you can easily manage Nest modules. Execute nest -h for more details.

  3. That's it. You just created a simple Nest module!

Use your Nest module in Python

  1. Open an interactive python interpreter under the same path of "hello.py" and run following commands:

    >> modules.hello_nest('Yanzhao', wrong=True) # Output: # # Unexpected param(s) "wrong" for Nest module: # hello_nest( # name:str) -> str ">
    >>> from nest import modules
    >>> print(modules.hello_nest) # access the module
    # Output:
    # 
    # hello_nest(
    # name:str) -> str
    >>> print(modules['*_nes?']) # wildcard search
    # Output:
    # 
    # hello_nest(
    # name:str) -> str
    >>> print(modules['r/main.\w+_nest']) # regex search
    # Output:
    # 
    # hello_nest(
    # name:str) -> str
    >>> modules.hello_nest('Yanzhao') # use the module
    # Output:
    #
    # 'Hello Yanzhao'
    >>> modules.hello_nest(123) # runtime type checking
    # Output:
    #
    # TypeError: The param "name" of Nest module "hello_nest" should be type of "str". Got "123".
    >>> modules.hello_nest('Yanzhao', wrong=True)
    # Output:
    #
    # Unexpected param(s) "wrong" for Nest module:
    # hello_nest(
    # name:str) -> str

    Note that Nest automatically imports modules and checks them as they are used to make sure everything is as expected.

  2. You can also directly import modules like this:

    >>> from nest.main.hello import hello_nest
    >>> hello_nest('World')
    # Output:
    #
    # 'Hello World'

    The import syntax is from nest. . import

  3. Access to Nest modules through code is flexible and easy.

Debug your Nest modules

  1. Open an interactive python interpreter under the same path of "hello.py" and run following commands:

    >>> from nest import modules
    >>> modules.hello_nest('Yanzhao')
    # Output:
    #
    # 'Hello Yanzhao'
  2. Keep the interpreter OPEN and use an externel editor to modify the "hello.py":

    # change Line7 from "return 'Hello ' + name" to
    return 'Nice to meet you, ' + name
  3. Back to the interpreter and rerun the same command:

    >>> modules.hello_nest('Yanzhao')
    # Output:
    #
    # 'Nice to meet you, Yanzhao'

    Note that Nest detects source file modifications and automatically reloads the module.

  4. You can use this feature to develop and debug your Nest modules efficiently.

Install Nest modules from local path

  1. Create a folder my_namespace and move the hello.py into it:

    $ mkdir my_namespace
    $ mv hello.py ./my_namespace/
  2. Create a new file more.py under the folder my_namespace with the following content:

    float: """Multiply two numbers.""" return a * b ">
    from nest import register
    
    @register(author='Yanzhao', version='1.0.0')
    def sum(a: int, b: int) -> int:
        """Sum two numbers."""
    
        return a + b
    
    # There is no need to repeatedly declare meta information
    # as modules within the same file automatically reuse the 
    # previous information. But overriding is also supported.
    @register(version='2.0.0')
    def mul(a: float, b: float) -> float:
        """Multiply two numbers."""
        
        return a * b

    Now we have:

    current path/
    ├── my_namespace/
    │   ├── hello.py
    │   ├── more.py
    
  3. Run the following command in the shell:

    Search paths. Continue? (Y/n) [Press ] ">
    $ nest module install ./my_namespace hello_word
    # Output:
    #
    # Install "./my_namespace/" -> Search paths. Continue? (Y/n) [Press 
          
           ]
          

    This command will add "my_namespace" folder to Nest's search path, and register all Nest modules in it under the namespace "hello_word". If the last argument is omitted, the directory name, "my_namespace" in this case, is used as the namespace.

  4. Verify the installation via CLI:

    $ nest module list
    # Output:
    #
    # 3 Nest modules found.
    # [0] hello_world.hello_nest (1.0.0)
    # [1] hello_world.mul (2.0.0)
    # [2] hello_world.sum (1.0.0)

    Note that those Nest modules can now be accessed regardless of your working path.

  5. Verify the installation via Python interpreter:

    $ ipython # open IPython interpreter
    >>> from nest import modules
    >>> print(len(modules))
    # Output:
    #
    # 3
    >>> modules.[Press <Tab>] # IPython Auto-completion
    # Output:
    #
    # hello_nest
    # mul
    # sum
    >>> modules.sum(3, 2)
    # Output:
    #
    # 5
    >>> modules.mul(2.5, 4.0)
    # Output:
    #
    # 10.0
  6. Thanks to the auto-import feature of Nest, you can easily share modules between different local projects.

Install Nest modules from URL

  1. You can use the CLI tool to install modules from URL:

    # select one of the following commands to execute
    # 0. install from Github repo via short URL (GitLab, Bitbucket are also supported)
    $ nest module install [email protected]/Nest:pytorch pytorch
    # 1. install from Git repo
    $ nest module install "-b pytorch https://github.com/ZhouYanzhao/Nest.git" pytorch
    # 2. install from zip file URL
    $ nest module install "https://github.com/ZhouYanzhao/Nest/archive/pytorch.zip" pytorch

    The last optional argument is used to specify the namespace, "pytorch" in this case.

  2. Verify the installation:

    $ nest module list
    # Output:
    #
    # 26 Nest modules found.
    # [0] hello_world.hello_nest (1.0.0)
    # [1] hello_world.mul (2.0.0)
    # [2] hello_world.sum (1.0.0)
    # [3] pytorch.adadelta_optimizer (0.1.0)
    # [4] pytorch.checkpoint (0.1.0)
    # [5] pytorch.cross_entropy_loss (0.1.0)
    # [6] pytorch.fetch_data (0.1.0)
    # [7] pytorch.finetune (0.1.0)
    # [8] pytorch.image_transform (0.1.0)
    # ...

Uninstall Nest modules

  1. You can remove modules from Nest's search path by executing:

    # given namespace
    $ nest module remove hello_world
    # given path to the namespace
    $ nest module remove ./my_namespace/
  2. You can also delete the corresponding files by appending a --delete or -d flag:

    $ nest module remove hello_world --delete

Version control Nest modules

  1. When installing modules, Nest adds the namespace to its search path without modifying or moving the original files. So you can use any version control system you like, e.g., Git, to manage modules. For example:

    $ cd <path of the namespace>
    # update modules
    $ git pull
    # specify version
    $ git checkout v1.0
  2. When developing a Nest module, it is recommended to define meta information for the module, such as the author, version, requirements, etc. Those information will be used by Nest's CLI tool. There are two ways to set meta information:

    • define meta information in code
    from nest import register
    
    @register(author='Yanzhao', version='1.0')
    def my_module() -> None:
        """My Module"""
        pass
    • define meta information in a nest.yml under the path of namespace
    author: Yanzhao
    version: 1.0
    requirements:
        - {url: opencv, tool: conda}
        # default tool is pip
        - torch>=0.4

    Note that you can use both ways at the same time.

Use Nest to manage your experiments

  1. Make sure you have Pytorch-backend modules installed, and if not, execute the following command:

    $ nest module install [email protected]/Nest:pytorch pytorch
  2. Create "train_mnist.yml" with the following content:

    _name: network_trainer
    data_loaders:
      _name: fetch_data
      dataset: 
        _name: mnist
        data_dir: ./data
      batch_size: 128
      num_workers: 4
      transform:
        _name: image_transform
        image_size: 28
        mean: [0.1307]
        std: [0.3081]
      train_splits: [train]
      test_splits: [test]
    model:
      _name: lenet5
    criterion:
      _name: cross_entropy_loss
    optimizer:
      _name: adadelta_optimizer
    meters:
      top1:
        _name: topk_meter
        k: 1
    max_epoch: 10
    device: cpu
    hooks:
      on_end_epoch: 
        - 
          _name: print_state
          formats:
            - 'epoch: {epoch_idx}'
            - 'train_acc: {metrics[train_top1]:.1f}%'
            - 'test_acc: {metrics[test_top1]:.1f}%'   

    Check HERE for more comprehensive demos.

  3. Run your experiments through CLI:

    $ nest task run ./train_mnist.yml
  4. You can also use Nest's task runner in your code:

    >>> from nest import run_tasks
    >>> run_tasks('./train_mnist.yml')
  5. Based on the task runner feature, Nest modules can be flexibly replaced and assembled to create your desired experiment settings.

Contact

Yanzhao Zhou

Issues

Feel free to submit bug reports and feature requests.

Contribution

Pull requests are welcome.

License

MIT

Copyright © 2018-present, Yanzhao Zhou

Owner
ZhouYanzhao
ZhouYanzhao
Tensorboard for pytorch (and chainer, mxnet, numpy, ...)

tensorboardX Write TensorBoard events with simple function call. The current release (v2.3) is tested on anaconda3, with PyTorch 1.8.1 / torchvision 0

Tzu-Wei Huang 7.5k Dec 28, 2022
A python library for time-series smoothing and outlier detection in a vectorized way.

tsmoothie A python library for time-series smoothing and outlier detection in a vectorized way. Overview tsmoothie computes, in a fast and efficient w

Marco Cerliani 517 Dec 28, 2022
DFM: A Performance Baseline for Deep Feature Matching

DFM: A Performance Baseline for Deep Feature Matching Python (Pytorch) and Matlab (MatConvNet) implementations of our paper DFM: A Performance Baselin

143 Jan 02, 2023
Photographic Image Synthesis with Cascaded Refinement Networks - Pytorch Implementation

Photographic Image Synthesis with Cascaded Refinement Networks-Pytorch (https://arxiv.org/abs/1707.09405) This is a Pytorch implementation of cascaded

Soumya Tripathy 63 Mar 27, 2022
Denoising Diffusion Implicit Models

Denoising Diffusion Implicit Models (DDIM) Jiaming Song, Chenlin Meng and Stefano Ermon, Stanford Implements sampling from an implicit model that is t

465 Jan 05, 2023
Iran Open Source Hackathon

Iran Open Source Hackathon is an open-source hackathon (duh) with the aim of encouraging participation in open-source contribution amongst Iranian dev

OSS Hackathon 121 Dec 25, 2022
Repository for GNSS-based position estimation using a Deep Neural Network

Code repository accompanying our work on 'Improving GNSS Positioning using Neural Network-based Corrections'. In this paper, we present a Deep Neural

32 Dec 13, 2022
Parametric Contrastive Learning (ICCV2021)

Parametric-Contrastive-Learning This repository contains the implementation code for ICCV2021 paper: Parametric Contrastive Learning (https://arxiv.or

DV Lab 156 Dec 21, 2022
A curated list of awesome deep long-tailed learning resources.

A curated list of awesome deep long-tailed learning resources.

vanint 210 Dec 25, 2022
Detectorch - detectron for PyTorch

Detectorch - detectron for PyTorch (Disclaimer: this is work in progress and does not feature all the functionalities of detectron. Currently only inf

Ignacio Rocco 558 Dec 23, 2022
Offline Multi-Agent Reinforcement Learning Implementations: Solving Overcooked Game with Data-Driven Method

Overcooked-AI We suppose to apply traditional offline reinforcement learning technique to multi-agent algorithm. In this repository, we implemented be

Baek In-Chang 14 Sep 16, 2022
Imaging, analysis, and simulation software for radio interferometry

ehtim (eht-imaging) Python modules for simulating and manipulating VLBI data and producing images with regularized maximum likelihood methods. This ve

Andrew Chael 5.2k Dec 28, 2022
Vanilla and Prototypical Networks with Random Weights for image classification on Omniglot and mini-ImageNet. Made with Python3.

vanilla-rw-protonets-project Vanilla Prototypical Networks and PNs with Random Weights for image classification on Omniglot and mini-ImageNet. Made wi

Giovani Candido 8 Aug 31, 2022
Implementation of Research Paper "Learning to Enhance Low-Light Image via Zero-Reference Deep Curve Estimation"

Zero-DCE and Zero-DCE++(Lite architechture for Mobile and edge Devices) Papers Abstract The paper presents a novel method, Zero-Reference Deep Curve E

Tauhid Khan 15 Dec 10, 2022
Official pytorch code for "APP: Anytime Progressive Pruning"

APP: Anytime Progressive Pruning Diganta Misra1,2,3, Bharat Runwal2,4, Tianlong Chen5, Zhangyang Wang5, Irina Rish1,3 1 Mila - Quebec AI Institute,2 L

Landskape AI 12 Nov 22, 2022
Improved Fitness Optimization Landscapes for Sequence Design

ReLSO Improved Fitness Optimization Landscapes for Sequence Design Description Citation How to run Training models Original data source Description In

Krishnaswamy Lab 44 Dec 20, 2022
PyTorch implementation of Algorithm 1 of "On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models"

Code for On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models This repository will reproduce the main results from our pape

Mitch Hill 32 Nov 25, 2022
An official reimplementation of the method described in the INTERSPEECH 2021 paper - Speech Resynthesis from Discrete Disentangled Self-Supervised Representations.

Speech Resynthesis from Discrete Disentangled Self-Supervised Representations Implementation of the method described in the Speech Resynthesis from Di

Facebook Research 253 Jan 06, 2023
Quantify the difference between two arbitrary curves in space

similaritymeasures Quantify the difference between two arbitrary curves Curves in this case are: discretized by inidviudal data points ordered from a

Charles Jekel 175 Jan 08, 2023
Adaptive Pyramid Context Network for Semantic Segmentation (APCNet CVPR'2019)

Adaptive Pyramid Context Network for Semantic Segmentation (APCNet CVPR'2019) Introduction Official implementation of Adaptive Pyramid Context Network

21 Nov 09, 2022