The starter repository for submissions to the GeneDisco challenge for optimized experimental design in genetic perturbation experiments

Overview

GeneDisco ICLR-22 Challenge Starter Repository

Python version Library version

The starter repository for submissions to the GeneDisco challenge for optimized experimental design in genetic perturbation experiments.

GeneDisco (to be published at ICLR-22) is a benchmark suite for evaluating active learning algorithms for experimental design in drug discovery. GeneDisco contains a curated set of multiple publicly available experimental data sets as well as open-source implementations of state-of-the-art active learning policies for experimental design and exploration.

Install

pip install -r requirements.txt

Use

Setup

  • Create a cache directory. This will hold any preprocessed and downloaded datasets for faster future invocation.
    • $ mkdir /path/to/genedisco_cache
    • Replace the above with your desired cache directory location.
  • Create an output directory. This will hold all program outputs and results.
    • $ mkdir /path/to/genedisco_output
    • Replace the above with your desired output directory location.

How to Run the Full Benchmark Suite?

Experiments (all baselines, acquisition functions, input and target datasets, multiple seeds) included in GeneDisco can be executed sequentially for e.g. acquired batch size 64, 8 cycles and a bayesian_mlp model using:

run_experiments \
  --cache_directory=/path/to/genedisco_cache  \
  --output_directory=/path/to/genedisco_output  \
  --acquisition_batch_size=64  \
  --num_active_learning_cycles=8  \
  --max_num_jobs=1

Results are written to the folder at /path/to/genedisco_cache, and processed datasets will be cached at /path/to/genedisco_cache (please replace both with your desired paths) for faster startup in future invocations.

Note that due to the number of experiments being run by the above command, we recommend execution on a compute cluster.
The GeneDisco codebase also supports execution on slurm compute clusters (the slurm command must be available on the executing node) using the following command and using dependencies in a Python virtualenv available at /path/to/your/virtualenv (please replace with your own virtualenv path):

run_experiments \
  --cache_directory=/path/to/genedisco_cache  \
  --output_directory=/path/to/genedisco_output  \
  --acquisition_batch_size=64  \
  --num_active_learning_cycles=8  \
  --schedule_on_slurm \
  --schedule_children_on_slurm \
  --remote_execution_virtualenv_path=/path/to/your/virtualenv

Other scheduling systems are currently not supported by default.

How to Run A Single Isolated Experiment (One Learning Cycle)?

To run one active learning loop cycle, for example, with the "topuncertain" acquisition function, the "achilles" feature set and the "schmidt_2021_ifng" task, execute the following command:

active_learning_loop  \
    --cache_directory=/path/to/genedisco/genedisco_cache \
    --output_directory=/path/to/genedisco/genedisco_output \
    --model_name="bayesian_mlp" \
    --acquisition_function_name="topuncertain" \
    --acquisition_batch_size=64 \
    --num_active_learning_cycles=8 \
    --feature_set_name="achilles" \
    --dataset_name="schmidt_2021_ifng" 

How to Evaluate a Custom Acquisition Function?

To run a custom acquisition function, set --acquisition_function_name="custom" and --acquisition_function_path to the file path that contains your custom acquisition function (e.g. main.py in this repo).

active_learning_loop  \
    --cache_directory=/path/to/genedisco/genedisco_cache \
    --output_directory=/path/to/genedisco/genedisco_output \
    --model_name="bayesian_mlp" \
    --acquisition_function_name="custom" \
    --acquisition_function_path=/path/to/src/main.py \
    --acquisition_batch_size=64 \
    --num_active_learning_cycles=8 \
    --feature_set_name="achilles" \
    --dataset_name="schmidt_2021_ifng" 

...where "/path/to/custom_acquisition_function.py" contains code for your custom acquisition function corresponding to the BaseBatchAcquisitionFunction interface, e.g.:

import numpy as np
from typing import AnyStr, List
from slingpy import AbstractDataSource
from slingpy.models.abstract_base_model import AbstractBaseModel
from genedisco.active_learning_methods.acquisition_functions.base_acquisition_function import \
    BaseBatchAcquisitionFunction

class RandomBatchAcquisitionFunction(BaseBatchAcquisitionFunction):
    def __call__(self,
                 dataset_x: AbstractDataSource,
                 batch_size: int,
                 available_indices: List[AnyStr], 
                 last_selected_indices: List[AnyStr] = None, 
                 model: AbstractBaseModel = None,
                 temperature: float = 0.9,
                 ) -> List:
        selected = np.random.choice(available_indices, size=batch_size, replace=False)
        return selected

Note that the last class implementing BaseBatchAcquisitionFunction is loaded by GeneDisco if there are multiple valid acquisition functions present in the loaded file.

Submission instructions

For submission, you will need two things:

Please note that all your submitted code must either be loaded via a dependency in requirements.txt or be present in the src/ directory in this starter repository for the submission to succeed.

Once you have set up your submission environment, you will need to create a lightweight container image that contains your acquisition function.

Submission steps

  • Navigate to the directory to which you have cloned this repo to.
    • $ cd /path/to/genedisco-starter
  • Ensure you have ONE acquisition function (inheriting from BaseBatchAcquisitionFunction) in main.py
    • This is your pre-defined program entry point.
  • Build your container image
    • $ docker build -t submission:latest .
  • Save your image name to a shell variable
    • $ IMAGE="submission:latest"
  • Use the EvalAI-CLI command to submit your image
    • Run the following command to submit your container image:
      • $ evalai push $IMAGE --phase gsk-genedisco-challenge-1528
      • Please note that you have a maximum number of submissions that any submission will be counted against.

That’s it! Our pipeline will take your image and test your function.

If you have any questions or concerns, please reach out to us at [email protected]

Citation

Please consider citing, if you reference or use our methodology, code or results in your work:

@inproceedings{mehrjou2022genedisco,
    title={{GeneDisco: A Benchmark for Experimental Design in Drug Discovery}},
    author={Mehrjou, Arash and Soleymani, Ashkan and Jesson, Andrew and Notin, Pascal and Gal, Yarin and Bauer, Stefan and Schwab, Patrick},
    booktitle={{International Conference on Learning Representations (ICLR)}},
    year={2022}
}

License

License

Authors

Arash Mehrjou, GlaxoSmithKline plc
Jacob A. Sackett-Sanders, GlaxoSmithKline plc
Patrick Schwab, GlaxoSmithKline plc

Acknowledgements

PS, JSS and AM are employees and shareholders of GlaxoSmithKline plc.

King is a simple boilerplate from a bigger Discord Bot project created for my Discord Server.

King A simple Discord bot boilerplate. King is a simple boilerplate from a bigger Discord Bot project created for my Discord Server. I intend to showc

Xminent 0 Aug 21, 2021
A simple cookiecutter to create Python Telegram bots, wrapped with Django.

PTB Django cookiecutter A simple cookiecutter to create Python Telegram bots, wrapped with Django. Based on this cool projects python-telegram-bot (PT

Carlos Lugones 20 Nov 12, 2022
Django Boilerplate - Material Kit Design | AppSeed

Django Boilerplate - Material Kit Design | AppSeed

App Generator 45 Dec 23, 2022
Forkable, Minimal Template for Starknet Projects.

Forkable, Minimal Template for Starknet Projects.

andreas 44 Oct 09, 2022
Mad-cookiecutter - Cookiecutter templates for MaD projects

MaD Cookiecutter Templates A set of templates that can be used to quickly get st

Machine Learning and Data Analytics Lab FAU 1 Jan 10, 2022
Brif is a boilerplate tool based on Docker and FastAPI, designed to streamline the development and deployment of IIIF compliant platforms.

brif A boilerplate tool based on Docker, designed to streamline the development and deployment of IIIF compliant platforms. Embedded with FastAPI, Cel

Pierre 8 Sep 17, 2022
Template to quickly start your playwright-python project

Playwright-python template 🍪 Template to quickly start your playwright-python project Getting started • Demo • Configuration Getting started Clone th

Constantin 1 Dec 13, 2021
Bleeding edge django template focused on code quality and security.

wemake-django-template Bleeding edge django2.2 template focused on code quality and security. Purpose This project is used to scaffold a django projec

wemake.services 1.6k Jan 08, 2023
Boilerplate code for a Python Flask API

MrMat :: Python :: API :: Flask Boilerplate code for a Python Flask API This variant of a Python Flask API is code-first and using native Flask Featur

0 Dec 26, 2021
The Django Base Site is a Django site that is built using the best Django practices and comes with all the common Django packages that you need to jumpstart your next project.

Django Base Site The Django Base Site is a Django site that is built using the best Django practices and comes with all the common Django packages tha

Brent O'Connor 167 Jan 03, 2023
A Django project skeleton that is modern and cutting edge.

{% comment "This comment section will be deleted in the generated project" %} Edge A Fantastic Django project starter. Features Ready Bootstrap-themed

Arun Ravindran 827 Dec 15, 2022
Cookiecutter Django is a framework for jumpstarting production-ready Django projects quickly.

Cookiecutter Django Powered by Cookiecutter, Cookiecutter Django is a framework for jumpstarting production-ready Django projects quickly. Documentati

Daniel Roy Greenfeld 10k Jan 01, 2023
Template for creating ds simple projects

ds-project-template Template for creating ds simple projects Requirements pyenv python==3.9.4 Setup For this purpose you use following commands: pytho

1 Dec 17, 2021
A template for some new Python tool or package with a reasonable basic setup.

python-app-template A template with a reasonable basic setup, including: black (formatting) flake8 (linting) mypy (type checking) isort (import sortin

Anton Pirogov 3 Jul 19, 2022
A cookiecutter template for python scripts

cookiecutter-py-script A cookiecutter template for python scripts Prerequisites Git Usage pip install cookiecutter

Zeheng Li 1 Dec 14, 2022
Starter project for python based lambda project.

Serverless Python Starter Starter project for python based lambda project. Features FastAPI - Frontend dev with Hot Reload API Gateway Integration (+r

4 Feb 22, 2022
Django project/application starter for lazybones :)

Django Project Starter Template My custom project starter for Django! I’ll try to support every upcoming Django releases as much as I can! Requirement

Uğur Özyılmazel 40 Jul 16, 2022
CRUD app to create and save code snippets, Flask/Python restful Api/backend and React/Typescript frontend

MS3 Cheat-Hub A cheatsheet hub. An app that organizes your code snippets into collections of cheat sheets and allows you to view, like and save others

Joem Elias Sanez 21 Dec 28, 2022
Simple boilerplate ready for development

StartApp Simple boilerplate ready for development Notes Currently supported frameworks are: FastApi, Flask 🔨 Installation $ sudo pip3 install starta

Sabuhi 16 Oct 16, 2022
A template repo for use in the Advent of Code

AoC-Template A template repo for use in the Advent of Code The README_template.md must contain "STATS_TABLE" to be replaced by the generated table, an

0 Jan 14, 2022