Python Jupyter kernel using Poetry for reproducible notebooks

Overview

Poetry Kernel

Use per-directory Poetry environments to run Jupyter kernels. No need to install a Jupyter kernel per Python virtual environment!

The idea behind this project is to allow you to capture the exact state of your environment. This means you can email your work to your peers, and they'll have exactly the same set of packages that you do! Reproducibility!

Why not virtual environments (venvs)?

Virtual environments were (and are) an important advancement to Python's package management story, but they have a few shortcomings:

  • They are not great for reproducibility. Usually, you'll create a new virtual environment using a requirements.txt which includes all the direct dependencies (numpy, pandas, etc.), but not transient dependencies (pandas depends on pytz for timezone support, for example). And usually, even the direct dependencies are specified only as minimum (or semver) ranges (e.g., numpy>=1.21) which can make it hard or impossible to accurately recreate the venv later.
  • With Jupyter, they usually require that the kernels be installed globally. This means you'll need need to have a separate kernelspec for every venv you want to use with Jupyter.

Poetry uses venvs transparently under the hood by constructing them from the pyproject.toml and poetry.lock files. The poetry.lock file records the exact state of dependencies (and transient dependencies) and can be used to more accurately reproduce the environment.

Additionally, Poetry Kernel means you only have to install one kernelspec. It then uses the pyproject.toml file from the directory of the notebook (or any parent directory) to choose which environment to run the notebook in.

Shameless plug

The reason we created this package was to make sure that the code environments created for running student code on Pathbird exactly match your development environment. Interested in developing interactive, engaging, inquiry-based lessons for your students? Check out Pathbird for more information!

Usage

  1. Install Poetry if not yet installed.
  2. Install this package:
    # NOTE: Do **NOT** install this package in your Poetry project, it should be
    # installed at the system or user level.
    pip3 install --user poetry-kernel
  3. Initialize a Poetry project (only required if you do not have an existing Poetry project ready to use):
    poetry init -n
  4. IMPORTANT: Add ipykernel to your project's dependencies:
    # In the directory of your Poetry project
    poetry add ipykernel
  5. Start a "Poetry" Jupyter kernel and see it in action! Jupyter launcher screenshot

Troubleshooting

Kernel isn't starting ("No Kernel" message)

Pro-tip: Check the output of the terminal window where you launched Jupyter. It will usually explain why the kernel is failing to start.

  1. Make sure that you are launching a notebook in a directory/folder that contains a Poetry project (pyproject.toml and poetry.lock files). You can turn a directory into a Poetry project by running:
poetry init -n
  1. Make sure that you've installed ipykernel into your project:
poetry add ipykernel
  1. Make sure the Poetry project is installed! This is especially important for projects that you have downloaded from others (warning: installing a Poetry project could run arbitrary code on your computer, make sure you trust your download first!):

    poetry install
  2. Still can't figure it out? Open an issue!

A package I added won't import properly

If you added the package after starting the kernel, you might need to restart the kernel for it to see the new package.

FAQ

See FAQ.md.

Comments
  • Windows Does not have SIGKILL

    Windows Does not have SIGKILL

    I believe the following line needs more attention to be compatible with Windows considering windows does not have SIGKILL: https://github.com/pathbird/poetry-kernel/blob/main/poetry_kernel/main.py#L39

    bug good first issue prs accepted 
    opened by amirhessam88 4
  • FileNotFoundError: [Errno 2] No such file or directory: 'poetry'.

    FileNotFoundError: [Errno 2] No such file or directory: 'poetry'.

    Hey, cool project!

    I got the following error, not sure how informative it is for you. Happy to dig into logs etc. I'm using a Gitpod workspace with a public github repo too, so could potentially offer a completely reproducible clean environment.

    Failed to start the Kernel. 
    FileNotFoundError: [Errno 2] No such file or directory: 'poetry'. 
    
    opened by andrewcstewart 3
  • Check in parent_dirs for pyproject.toml

    Check in parent_dirs for pyproject.toml

    This now loops through parent directories to check for pyproject.toml to allow for kernel execution when running notebooks in sub directories. I used the same code from poetry itself to ensure compatibility

    opened by nick-gorse 3
  • modified forward_signals to be compatible with windows

    modified forward_signals to be compatible with windows

    in regard to issue #3

    removed signal.SIGTERM from forward_signals, as well as two others as they could not be used by signal.signal (ValueError: invalid signal value)

    from https://docs.python.org/3/library/signal.html signal.CTRL_C_EVENT The signal corresponding to the Ctrl+C keystroke event. This signal can only be used with os.kill()

    same for signal.CTRL_BREAK_EVENT

    opened by gpfv 0
  • Check in parent_dirs for pyproject.toml

    Check in parent_dirs for pyproject.toml

    This now loops through parent directories to check for pyproject.toml to allow for kernel execution when running notebooks in sub directories. I used the same code as used poetry itself to ensure compatibility

    opened by nick-gorse 0
  • poetry-kernel uses Jupyter env instead of notebook env

    poetry-kernel uses Jupyter env instead of notebook env

    If a poetry pyproject.toml is used to create the environment for the Jupyter lab (adding poetry-kernel as a dependency), then no matter in which working directory the jupyter server is started, any notebook that is opened with the "poetry" kernel will have the Jupyter environment, not the environment of the notebook's project.

    Minimal example: https://github.com/drakesiardxy/poetry-kernel-bug To replicate: Create the jupyer-base env and the kernel_a env separately, then start the jupyter server with the first environment and attempt to run kernel_a.ipynb using the "Poetry" kernel. pandas will be missing, because the notebook will have been launched with the environment of the server, not the environment belonging to the notebook's project.

    opened by drakesiardxy 0
  • Is it possible to make poetry-kernel work with JupyterHub?

    Is it possible to make poetry-kernel work with JupyterHub?

    Hi!

    We have a multi-user JupyterHub instance and we would like to use your library. So far we have only been able to see the poetry-kernel button in the kernel selection screen by installing poetry kernel at the user level (if we install poetry kernel from the user that launches JupyterHub, the root user, the button does not show). But the problem that we have is that when we press the poetry button and create a new notebook in a folder with a poetry project (and ipykernel installed), the kernel is never connected so no code can be executed. The generated logs are the following:

    Apr 26 15:30:26 labs-ubuntu-20-04 python3[446728]: [I 2022-04-26 15:30:26.298 SingleUserLabApp restarter:66] AsyncIOLoopKernelRestarter: restarting kernel (2/5), new random ports
    Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]: Traceback (most recent call last):
    Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:   File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:     return _run_code(code, main_globals, None,
    Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:   File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:     exec(code, run_globals)
    Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:   File "/home/marta/.local/lib/python3.8/site-packages/poetry_kernel/__main__.py", line 68, in <module>
    Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:     main()
    Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:   File "/home/marta/.local/lib/python3.8/site-packages/poetry_kernel/__main__.py", line 37, in main
    Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:     proc = subprocess.Popen(cmd)
    Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
    Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:     self._execute_child(args, executable, preexec_fn, close_fds,
    Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
    Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]:     raise child_exception_type(errno_num, err_msg, err_filename)
    Apr 26 15:30:26 labs-ubuntu-20-04 python3[1028846]: FileNotFoundError: [Errno 2] No such file or directory: 'poetry'
    

    This looks like poetry is not being found by the library although it is indeed installed both at the root and user level. Is there a way to solve this? Or is this case out of the scope of the library for now?

    opened by MsLimon 2
Releases(v0.1.2)
  • v0.1.2(Mar 30, 2022)

    What's Changed

    • modified forward_signals to be compatible with windows by @gpfv in https://github.com/pathbird/poetry-kernel/pull/4

    New Contributors

    • @gpfv made their first contribution in https://github.com/pathbird/poetry-kernel/pull/4

    Full Changelog: https://github.com/pathbird/poetry-kernel/compare/v0.1.1...v0.1.2

    Source code(tar.gz)
    Source code(zip)
Owner
Pathbird
Pathbird is a platform for instructors to build interactive, engaging, inquiry-based lessons for computational courses.
Pathbird
Code for the CVPR 2021 paper "Triple-cooperative Video Shadow Detection"

Triple-cooperative Video Shadow Detection Code and dataset for the CVPR 2021 paper "Triple-cooperative Video Shadow Detection"[arXiv link] [official l

Zhihao Chen 24 Oct 04, 2022
Riemann Noise Injection With PyTorch

Riemann Noise Injection - PyTorch A module for modeling GAN noise injection based on Riemann geometry, as described in Ruili Feng, Deli Zhao, and Zhen

2 May 27, 2022
Part-Aware Data Augmentation for 3D Object Detection in Point Cloud

Part-Aware Data Augmentation for 3D Object Detection in Point Cloud This repository contains a reference implementation of our Part-Aware Data Augment

Jaeseok Choi 62 Jan 03, 2023
A large-image collection explorer and fast classification tool

IMAX: Interactive Multi-image Analysis eXplorer This is an interactive tool for visualize and classify multiple images at a time. It written in Python

Matias Carrasco Kind 23 Dec 16, 2022
A production-ready, scalable Indexer for the Jina neural search framework, based on HNSW and PSQL

🌟 HNSW + PostgreSQL Indexer HNSWPostgreSQLIndexer Jina is a production-ready, scalable Indexer for the Jina neural search framework. It combines the

Jina AI 25 Oct 14, 2022
Pyserini is a Python toolkit for reproducible information retrieval research with sparse and dense representations.

Pyserini Pyserini is a Python toolkit for reproducible information retrieval research with sparse and dense representations. Retrieval using sparse re

Castorini 706 Dec 29, 2022
The Python code for the paper A Hybrid Quantum-Classical Algorithm for Robust Fitting

About The Python code for the paper A Hybrid Quantum-Classical Algorithm for Robust Fitting The demo program was only tested under Conda in a standard

Anh-Dzung Doan 5 Nov 28, 2022
IEEE-CIS Technical Challenge on Predict+Optimize for Renewable Energy Scheduling

IEEE-CIS Technical Challenge on Predict+Optimize for Renewable Energy Scheduling This is my code, data and approach for the IEEE-CIS Technical Challen

3 Sep 18, 2022
Automatic learning-rate scheduler

AutoLRS This is the PyTorch code implementation for the paper AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly published

Yuchen Jin 33 Nov 18, 2022
RCT-ART is an NLP pipeline built with spaCy for converting clinical trial result sentences into tables through jointly extracting intervention, outcome and outcome measure entities and their relations.

Randomised controlled trial abstract result tabulator RCT-ART is an NLP pipeline built with spaCy for converting clinical trial result sentences into

2 Sep 16, 2022
Softlearning is a reinforcement learning framework for training maximum entropy policies in continuous domains. Includes the official implementation of the Soft Actor-Critic algorithm.

Softlearning Softlearning is a deep reinforcement learning toolbox for training maximum entropy policies in continuous domains. The implementation is

Robotic AI & Learning Lab Berkeley 997 Dec 30, 2022
Automatically download the cwru data set, and then divide it into training data set and test data set

Automatically download the cwru data set, and then divide it into training data set and test data set.自动下载cwru数据集,然后分训练数据集和测试数据集

6 Jun 27, 2022
Code accompanying "Evolving spiking neuron cellular automata and networks to emulate in vitro neuronal activity," accepted to IEEE SSCI ICES 2021

Evolving-spiking-neuron-cellular-automata-and-networks-to-emulate-in-vitro-neuronal-activity Code accompanying "Evolving spiking neuron cellular autom

SOCRATES: Self-Organizing Computational substRATES 2 Dec 02, 2022
Official PyTorch implementation of Joint Object Detection and Multi-Object Tracking with Graph Neural Networks

This is the official PyTorch implementation of our paper: "Joint Object Detection and Multi-Object Tracking with Graph Neural Networks". Our project website and video demos are here.

Richard Wang 443 Dec 06, 2022
A High-Quality Real Time Upscaler for Anime Video

Anime4K Anime4K is a set of open-source, high-quality real-time anime upscaling/denoising algorithms that can be implemented in any programming langua

15.7k Jan 06, 2023
DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers

DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers Authors: Jaemin Cho, Abhay Zala, and Mohit Bansal (

Jaemin Cho 98 Dec 15, 2022
【steal piano】GitHub偷情分析工具!

【steal piano】GitHub偷情分析工具! 你是否有这样的困扰,有一天你的仓库被很多人加了star,但是你却不知道这些人都是从哪来的? 别担心,GitHub偷情分析工具帮你轻松解决问题! 原理 GitHub偷情分析工具透过分析star的时间以及他们之间的follow关系,可以推测出每个st

黄巍 442 Dec 21, 2022
Robotics environments

Robotics environments Details and documentation on these robotics environments are available in OpenAI's blog post and the accompanying technical repo

Farama Foundation 121 Dec 28, 2022
Unofficial Implementation of MLP-Mixer, Image Classification Model

MLP-Mixer Unoffical Implementation of MLP-Mixer, easy to use with terminal. Train and test easly. https://arxiv.org/abs/2105.01601 MLP-Mixer is an arc

Oğuzhan Ercan 6 Dec 05, 2022
Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes (CVPR2021)

RSCD (BS-RSCD & JCD) Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes (CVPR2021) by Zhihang Zhong, Yinqiang Zheng, Imari Sato We co

81 Dec 15, 2022