This speeds up PyCharm's package index processes and avoids CPU & memory overloading

Overview
Avoid PyCharm Overloading
=========================

This speeds up PyCharm's package index processes and avoids CPU & memory
overloading.

On my machine (4c/32GB) with 5 conda environments and 10 projects, this reduces
PyCharm's package scanning from 15 minutes to 45 seconds.

Why?
----

When PyCharm has multiple conda environments, it will sometimes launch concurrent
package scanning across those environments. To do so, it uses the `conda` cli,
and a tool called conda_packaging_tool.py, available from the JetBrains
intellij-community github repository (see below).

Unfortunately, these tools do not behave well when launched in parallel, as
they tend to overload CPU & memory in this case.

How?
----

To avoid stalling these processes, we modify their sources as follows:

1. modify the conda cli

    $ conda activate base
    $ pip install runfast
    $ nano `which conda`

    Modify the main section:

     if __name__ == '__main__':
        from conda.cli import main
        from runfast import cached
        cached(main)

2. modify the conda_packaging_tool.py

    if __name__ == '__main__':
        from runfast import cached
        cached(main)

How does this work?
-------------------

Two simple steps:

1. When called in parallel, only one process is allowed to proceed.
2. `runfast.cached` caches the output of these tools (stdout, stderr) for 1 minute,
   given the same command line parameters

That is, when PyCharm launches 5 package scanning (`conda list`) commands, only
one of them will run immediately. If some of the scans are for the same environment,
only one of them will actually run, while the others simply return the same output.

How to clear the cache or avoid caching at all?

    $ export RUNFAST_NOCACHE=1

Testing
-------

a) conda packaging tool

    # first time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m13.271s
    user    0m9.631s
    sys     0m1.936s

    # second time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m3.159s
    user    0m0.139s
    sys     0m1.298s

b) conda cli

   # first time
   $ time conda list -p /path/to/env
   real    0m7.985s
   user    0m7.883s
   sys     0m0.087s

   # second time
   real    0m0.152s
   user    0m0.116s
   sys     0m0.032s

References
----------
* https://github.com/JetBrains/intellij-community/blob/master/python/helpers/conda_packaging_tool.py
* https://github.com/conda/conda/blob/33a142c16530fcdada6c377486f1c1a385738a96/conda/core/index.py#L53
Owner
tinkerer | software engineer | data scientist
Resizing using nnedi3/znedi3/nnedi3cl with center alignment and correct chroma placement

nnedi3_resample A VapourSynth script for easy resizing using nnedi3/znedi3/nnedi3cl with center alignment and correct chroma placement. Requirements n

Home Of VapourSynth Evolution 12 Sep 08, 2022
A subleq VM/interpreter created by me for no reason

What is Dumbleq? Dumbleq is a dumb Subleq VM/interpreter implementation created by me for absolutely no reason at all. What is Subleq? If you haven't

Phu Minh 2 Nov 13, 2022
Bible-App : Simple Tool To Show Bible Books

Bible App Simple Tool To Show Bible Books Socials: Language:

ميخائيل 5 Jan 18, 2022
Create VSCode Extensions with python

About Create vscode extensions with python. Installation Stable version: pip install vscode-ext Why use this? Why should you use this for building VSc

Swas.py 134 Jan 07, 2023
Self sustained producer-consumer(prosumer) policy study using Python and Gurobi

Prosumer Policy This project aims to model the optimum dispatch behaviour of households with PV and battery systems under different policy instrument

Tom Xu 3 Aug 31, 2022
Clackety Keyboards Powered by Python

KMK: Clackety Keyboards Powered by Python KMK is a feature-rich and beginner-friendly firmware for computer keyboards written and configured in Circui

KMK Firmware 780 Jan 03, 2023
Python project setup, updater, and launcher

Launcher Python project setup, updater, and launcher Purpose: Increase project productivity and provide features easily. Once installed as a git submo

DAAV, LLC 1 Jan 07, 2022
Uproot - A script to bring deeply nested files or directories to the surface

UPROOT Bring deeply nested files or folders to the surface Uproot helps convert

Ted 2 Jan 15, 2022
This repository contains Python games that I've worked on. You'll learn how to create python games with AI. I try to focus on creating board games without GUI in Jupyter-notebook.

92_Python_Games 🎮 Introduction 👋 This repository contains Python games that I've worked on. You'll learn how to create python games with AI. I try t

Milaan Parmar / Милан пармар / _米兰 帕尔马 166 Jan 01, 2023
Why write code when you can import it directly from GitHub Copilot?

Copilot Importer Why write code when you can import it directly from GitHub Copilot? What is Copilot Importer? The copilot python module will dynamica

Mythic 41 Jan 04, 2023
🌍💉 Global COVID-19 vaccination data at the regional level.

COVID-19 vaccination data at subnational level. To ensure its officiality, the source data is carefully verified.

sociepy 61 Sep 21, 2022
The Great Autoencoder Bake Off

The Great Autoencoder Bake Off The companion repository to a post on my blog. It contains all you need to reproduce the results. Features Currently fe

Tilman Krokotsch 61 Jan 06, 2023
Multifunctional Analysis of Regions through Input-Output

MARIO Multifunctional Analysis of Regions through Input-Output. (Documents) What is it MARIO is a python package for handling input-output tables and

14 Dec 25, 2022
Simple Kahoot Botter.

Kahoot A simple Botter made in Python 3 for Kahoot.com. Also sorry for the shitty code lol. How to Run You need Python 3 installed on your device. Aft

7 Jun 29, 2022
An open-source hyper-heuristic framework for multi-objective optimization

MOEA-HH An open-source hyper-heuristic framework for multi-objective optimization. Introduction The multi-objective optimization technique is widely u

Hengzhe Zhang 1 Feb 10, 2022
Roman numeral conversion with python

Roman numeral conversion Discipline: Programming Languages Student: Paulo Henrique Diniz de Lima Alencar. Language: Python Description Responsible for

Paulo Alencar 1 Jul 11, 2022
API development made easy: a smart Python 3 API framework

appkernel - API development made easy What is Appkernel? A super-easy to use API framework, enabling API creation from zero to production within minut

156 Sep 28, 2022
A collection of some leetcode challenges in python and JavaScript

Python and Javascript Coding Challenges Some leetcode questions I'm currently working on to open up my mind to better ways of problem solving. Impleme

Ted Ngeene 1 Dec 20, 2021
Personal Finance Forecaster - An AI tool for forecasting personal expenses

Personal Finance Forecaster - An AI tool for forecasting personal expenses

2 Mar 09, 2022
LPCV Winner Solution of Spring Team

LPCV Winner Solution of Spring Team

22 Jul 20, 2022