Hivemind: decentralized deep learning in PyTorch
Hivemind is a PyTorch library to train large neural networks across the Internet. Its intended usage is training a single Transformer model on hundreds of computers from different universities, companies, and volunteers.
Key Features
- Train neural networks of arbitrary size: parts of their layers are distributed across the participants.
- Distributed training without a master node: Distributed Hash Table allows connecting computers in a decentralized network.
- Fault-tolerant backpropagation: forward and backward passes succeed even if some nodes are unresponsive or take too long to respond.
- Decentralized parameter averaging: iteratively aggregate updates from multiple workers without the need to synchronize across the entire network.
To learn more about the ideas behind this library, see https://learning-at-home.github.io or read the NeurIPS 2020 paper.
Installation
Before installing hivemind, make sure that your environment has Python 3.7+ and PyTorch with a version at least as new as 1.6.0.
To start using this library, you can either use the pip package manager or build it from source. Since currently the release cycle is not established yet, we recommend installing hivemind from source to keep up with the latest bugfixes and improvements.
With pip
If your versions of Python and PyTorch match the requirements, you can install hivemind from pip:
pip install hivemind
From source
To install hivemind from source, simply clone the repository and install
git clone https://github.com/learning-at-home/hivemind.git
cd hivemind
pip install .
If you would like to verify that your installation is working properly, you can install with pip install -e .[dev] instead. Then, you can run the tests with pytest tests/.
Documentation
- Quickstart: install hivemind, set up a server and train experts
- Documentation & guides are available at learning-at-home.readthedocs.io
Contributing
Hivemind is currently at the active development stage, and we welcome all contributions. Everything, from bug fixes and documentation improvements to entirely new features, is equally appreciated.
If you want to contribute to hivemind but don't know where to start, take a look at the unresolved issues. Open a new issue or join our chat room in case you want to discuss new functionality or report a possible bug. Bug fixes are always welcome, but new features should be preferably discussed with maintainers beforehand.
If you want to start contributing to the source code of hivemind, please see the contributing guidelines first. To learn more about other ways to contribute, read our guide.
Citation
If you found hivemind useful for your experiments, you can cite the paper that inspired it:
@inproceedings{ryabinin2020crowdsourced,
author = {Ryabinin, Max and Gusev, Anton},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},
pages = {3659--3672},
publisher = {Curran Associates, Inc.},
title = {Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts},
url = {https://proceedings.neurips.cc/paper/2020/file/25ddc0f8c9d3e22e03d3076f98d83cb2-Paper.pdf},
volume = {33},
year = {2020}
}
The initial implementation of hivemind used for the paper is available at mryab/learning-at-home.
In the documentation, we list several related projects and acknowledgements.
![[BUG] Dead lock when 'Downloading parameters' cost Took too much time](https://avatars.githubusercontent.com/u/14099092?v=4)
![[BUG] Enable to train a bloat16-compressed model](https://avatars.githubusercontent.com/u/79724149?v=4)
48 Dec 12, 2022
4 Feb 16, 2022
632 Dec 29, 2022
34 Dec 21, 2022
465 Jan 02, 2023
156 Dec 14, 2022
6 Mar 12, 2022
519 Jan 03, 2023
164 Jan 04, 2023
3.8k Jan 04, 2023
1k Dec 24, 2022
1 Feb 01, 2022
1 Jan 06, 2022
0 Jan 20, 2022
8 Aug 10, 2022
859 Dec 10, 2022
1 Feb 06, 2022
119 Nov 24, 2022
250 Jan 07, 2023
0 Jan 18, 2022