Adaptable tools to make reinforcement learning and evolutionary computation algorithms.

Last update: Jan 01, 2023

Overview

Pearl

The Parallel Evolutionary and Reinforcement Learning Library (Pearl) is a pytorch based package with the goal of being excellent for rapid prototyping of new adaptive decision making algorithms in the intersection between reinforcement learning (RL) and evolutionary computation (EC). As such, this is not intended to provide template pre-built algorithms as a baseline, but rather flexible tools to allow the user to quickly build and test their own implementations and ideas. A technical report can be found here.

Main Features

Features	Pearl
RL algorithms (e.g. Actor Critic)	✔️
EC algorithms (e.g. Genetic Algorithm)	✔️
Hybrid algorithms (e.g. CEM-DDPG)	✔️
Multi-agent suppport	✔️
Tensorboard integration	✔️
Modular and extensible components	✔️
Opinionated module settings	✔️
Custom callbacks	✔️

User Guide

Installation

There are two options to install this package:

pip install pearll
git clone [email protected]:LondonNode/Pearl.git

Module Guide

agents: implementations of RL and EC agents where the other modular components are put together
buffers: these handle storing and sampling of trajectories
callbacks: inject logic for every step made in an environment (e.g. save model, early stopping)
common: common methods applicable to all other modules (e.g. enumerations) and a main utils.py file with some useful general logic
explorers: action explorers for enhanced exploration by adding noise to actions and random exploration for first n steps
models: neural network structures which are structured as encoder -> torso -> head
signal_processing: signal processing logic for extra modularity (e.g. TD returns, GAE)
updaters: update neural networks and adaptive/iterative algorithms
settings.py: settings objects for the above components, can be extended for custom components

Agent Templates

See pearll/agents/templates.py for the templates to create your own agents! For more examples, see specific agent implementations under pearll/agents.

Agent Performance

To see training performance, use the command tensorboard --logdir runs or tensorboard --logdir <tensorboard_log_path> defined in your algorithm class initialization.

Python Scripts

To run these you'll need to go to wherever the library is installed, cd pearll.

demo.py: script to run very basic demos of agents with pre-defined hyperparameters, run python3 -m pearll.demo -h for more info
plot.py: script to plot more complex plots that can't be obtained via Tensorboard (e.g. multiple subplots), run python3 -m pearll.plot -h for more info

Developer Guide

Scripts

Linux

scripts/setup_dev.sh: setup your virtual environment
scripts/run_tests.sh: run tests

Windows

scripts/windows_setup_dev.bat: setup your virtual environment
scripts/windows_run_tests.bat: run tests

Dependency Management

Pearl uses poetry for dependency management and build release instead of pip. As a quick guide:

Run poetry add [package] to add more package dependencies.
Poetry automatically handles the virtual environment used, check pyproject.toml for specifics on the virtual environment setup.
If you want to run something in the poetry virtual environment, add poetry run as a prefix to the command you want to execute. For example, to run a python file: poetry run python3 script.py.

Credit

Citing Pearl

@misc{tangri2022pearl,
      title={Pearl: Parallel Evolutionary and Reinforcement Learning Library}, 
      author={Rohan Tangri and Danilo P. Mandic and Anthony G. Constantinides},
      year={2022},
      eprint={2201.09568},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

Acknowledgements

Pearl was inspired by Stable Baselines 3 and Tonic

BESS: Balanced Evolutionary Semi-Stacking for Disease Detection via Partially Labeled Imbalanced Tongue Data

Balanced-Evolutionary-Semi-Stacking Code for the paper ''BESS: Balanced Evolutionary Semi-Stacking for Disease Detection via Partially Labeled Imbalan

0 Jan 16, 2022

Systemic Evolutionary Chemical Space Exploration for Drug Discovery

SECSE SECSE: Systemic Evolutionary Chemical Space Explorer Chemical space exploration is a major task of the hit-finding process during the pursuit of

64 Dec 16, 2022

Deep learning with dynamic computation graphs in TensorFlow

TensorFlow Fold TensorFlow Fold is a library for creating TensorFlow models that consume structured data, where the structure of the computation graph

1.8k Dec 28, 2022

A toolkit for developing and comparing reinforcement learning algorithms.

Status: Maintenance (expect bug fixes and minor updates) OpenAI Gym OpenAI Gym is a toolkit for developing and comparing reinforcement learning algori

29.6k Jan 8, 2023

PyTorch implementations of deep reinforcement learning algorithms and environments

Deep Reinforcement Learning Algorithms with PyTorch This repository contains PyTorch implementations of deep reinforcement learning algorithms and env

4.7k Jan 4, 2023

Pytorch implementations of popular off-policy multi-agent reinforcement learning algorithms, including QMix, VDN, MADDPG, and MATD3.

Off-Policy Multi-Agent Reinforcement Learning (MARL) Algorithms This repository contains implementations of various off-policy multi-agent reinforceme

183 Dec 28, 2022

Reinforcement learning framework and algorithms implemented in PyTorch.

2.1k Jan 4, 2023

Independent and minimal implementations of some reinforcement learning algorithms using PyTorch (including PPO, A3C, A2C, ...).

PyTorch RL Minimal Implementations There are implementations of some reinforcement learning algorithms, whose characteristics are as follow: Less pack

4 Dec 31, 2022

PyTorch version of Stable Baselines, reliable implementations of reinforcement learning algorithms.

4.7k Jan 1, 2023

Comments

Bump pillow from 9.0.0 to 9.0.1
Bumps pillow from 9.0.0 to 9.0.1.

Release notes

Sourced from pillow's releases.

9.0.1

https://pillow.readthedocs.io/en/stable/releasenotes/9.0.1.html

Changes

In show_file, use os.remove to remove temporary images. CVE-2022-24303 #6010 [@radarhere, @hugovk]

Restrict builtins within lambdas for ImageMath.eval. CVE-2022-22817 #6009 [radarhere]

Changelog

Sourced from pillow's changelog.

9.0.1 (2022-02-03)

In show_file, use os.remove to remove temporary images. CVE-2022-24303 #6010 [radarhere, hugovk]

Restrict builtins within lambdas for ImageMath.eval. CVE-2022-22817 #6009 [radarhere]

Commits

6deac9e 9.0.1 version bump

c04d812 Update CHANGES.rst [ci skip]

4fabec3 Added release notes for 9.0.1

02affaa Added delay after opening image with xdg-open

ca0b585 Updated formatting

427221e In show_file, use os.remove to remove temporary images

c930be0 Restrict builtins within lambdas for ImageMath.eval

75b69dd Dont need to pin for GHA

cd938a7 Autolink CWE numbers with sphinx-issues

2e9c461 Add CVE IDs

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 1
Feature/hybrid

Overhaul models and base agent structure to accommodate RL, MARL, EC in optimizing static functions and RL environments and hybrid algorithms combining RL and EC.

opened by 09tangriro 1
MORE AGENTS

The more agents created the better proof that the tools underlying work as intended.

Agents should be tested on particular environments to ensure performance.
feature good first issue

opened by 09tangriro 0

Releases(v0.4.1)

v0.4.1(May 9, 2022)

Bug fixes and optimizations.

See PR #11
Source code(tar.gz)
Source code(zip)
v0.4.0(May 8, 2022)

Optimizations interfacing with GPU devices. See PR #10
Source code(tar.gz)
Source code(zip)
v0.3.1(Apr 5, 2022)
Bug fixes:

allow different size discrete space output for DiscreteHead.

Update docstrings for pearll/updaters/environment module.

Source code(tar.gz)
Source code(zip)
v0.3.0(Mar 28, 2022)
Introduce model-based RL tools.

Validate model-based RL tools with implementation of DynaQ algorithm.

Cleaner signal_processing module interface using functools.

Source code(tar.gz)
Source code(zip)
v0.2.2(Mar 4, 2022)

Fixed issue running multi-agent algorithms on cuda devices. Now full support for cuda.
Source code(tar.gz)
Source code(zip)
v0.2.1(Mar 2, 2022)
Various bug fixes:

to_numpy cuda support.

FlattenEncoder flattens inputs appropriately.

Callbacks more robust.

Also added a tutorial library.
Source code(tar.gz)
Source code(zip)
v0.2.0(Jan 25, 2022)

Various bug fixes and tweaks to the interface.
Source code(tar.gz)
Source code(zip)
v0.1.0(Jan 11, 2022)

Pre-release before paper submission.
Source code(tar.gz)
Source code(zip)

Owner

GitHub Repository

Semantic code search implementation using Tensorflow framework and the source code data from the CodeSearchNet project

Semantic Code Search Semantic code search implementation using Tensorflow framework and the source code data from the CodeSearchNet project. The model

24 Nov 29, 2022

A generalist algorithm for cell and nucleus segmentation.

Cellpose | A generalist algorithm for cell and nucleus segmentation. Cellpose was written by Carsen Stringer and Marius Pachitariu. To learn about Cel

733 Dec 29, 2022

Code for the paper One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation, CVPR 2021.

One Thing One Click One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation (CVPR2021) Code for the paper One Thi

44 Dec 12, 2022

Monocular Depth Estimation - Weighted-average prediction from multiple pre-trained depth estimation models

merged_depth runs (1) AdaBins, (2) DiverseDepth, (3) MiDaS, (4) SGDepth, and (5) Monodepth2, and calculates a weighted-average per-pixel absolute dept

39 Nov 21, 2022

[ACL-IJCNLP 2021] Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning

CLNER The code is for our ACL-IJCNLP 2021 paper: Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning CLNER is a

71 Dec 08, 2022

Code Repository for Liquid Time-Constant Networks (LTCs)

Liquid time-constant Networks (LTCs) [Update] A Pytorch version is added in our sister repository: https://github.com/mlech26l/keras-ncp This is the o

553 Dec 27, 2022

Code for the paper "Implicit Representations of Meaning in Neural Language Models"

Implicit Representations of Meaning in Neural Language Models Preliminaries Create and set up a conda environment as follows: conda create -n state-pr

39 Nov 03, 2022

Editing a classifier by rewriting its prediction rules

This repository contains the code and data for our paper: Editing a classifier by rewriting its prediction rules Shibani Santurkar*, Dimitris Tsipras*

86 Dec 27, 2022

Recurrent Neural Network Tutorial, Part 2 - Implementing a RNN in Python and Theano

Please read the blog post that goes with this code! Jupyter Notebook Setup System Requirements: Python, pip (Optional) virtualenv To start the Jupyter

863 Dec 15, 2022

A complete, self-contained example for training ImageNet at state-of-the-art speed with FFCV

ffcv ImageNet Training A minimal, single-file PyTorch ImageNet training script designed for hackability. Run train_imagenet.py to get... ...high accur

92 Dec 31, 2022

People log into different sites every day to get information and browse through these sites one by one

HyperLink People log into different sites every day to get information and browse through these sites one by one. And they are exposed to advertisemen

0 Feb 17, 2022

Toolchain to build Yoshi's Island from source code

Project-Y Toolchain to build Yoshi's Island (J) V1.0 from source code, by MrL314 Last updated: September 17, 2021 Setup To begin, download this toolch

19 Apr 18, 2022

PyTorch implementation of Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation.

ALiBi PyTorch implementation of Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. Quickstart Clone this reposit

4 Jul 27, 2022

PyMatting: A Python Library for Alpha Matting

Given an input image and a hand-drawn trimap (top row), alpha matting estimates the alpha channel of a foreground object which can then be composed onto a different background (bottom row).

1.4k Dec 30, 2022

An abstraction layer for mathematical optimization solvers.

MathOptInterface Documentation Build Status Social An abstraction layer for mathematical optimization solvers. Replaces MathProgBase. Citing MathOptIn

284 Jan 04, 2023

A research toolkit for particle swarm optimization in Python

PySwarms is an extensible research toolkit for particle swarm optimization (PSO) in Python. It is intended for swarm intelligence researchers, practit

1k Dec 30, 2022

Simple helper library to convert a collection of numpy data to tfrecord, and build a tensorflow dataset from the tfrecord.

numpy2tfrecord Simple helper library to convert a collection of numpy data to tfrecord, and build a tensorflow dataset from the tfrecord. Installation

2 Jan 16, 2022