This speeds up PyCharm's package index processes and avoids CPU & memory overloading

Overview
Avoid PyCharm Overloading
=========================

This speeds up PyCharm's package index processes and avoids CPU & memory
overloading.

On my machine (4c/32GB) with 5 conda environments and 10 projects, this reduces
PyCharm's package scanning from 15 minutes to 45 seconds.

Why?
----

When PyCharm has multiple conda environments, it will sometimes launch concurrent
package scanning across those environments. To do so, it uses the `conda` cli,
and a tool called conda_packaging_tool.py, available from the JetBrains
intellij-community github repository (see below).

Unfortunately, these tools do not behave well when launched in parallel, as
they tend to overload CPU & memory in this case.

How?
----

To avoid stalling these processes, we modify their sources as follows:

1. modify the conda cli

    $ conda activate base
    $ pip install runfast
    $ nano `which conda`

    Modify the main section:

     if __name__ == '__main__':
        from conda.cli import main
        from runfast import cached
        cached(main)

2. modify the conda_packaging_tool.py

    if __name__ == '__main__':
        from runfast import cached
        cached(main)

How does this work?
-------------------

Two simple steps:

1. When called in parallel, only one process is allowed to proceed.
2. `runfast.cached` caches the output of these tools (stdout, stderr) for 1 minute,
   given the same command line parameters

That is, when PyCharm launches 5 package scanning (`conda list`) commands, only
one of them will run immediately. If some of the scans are for the same environment,
only one of them will actually run, while the others simply return the same output.

How to clear the cache or avoid caching at all?

    $ export RUNFAST_NOCACHE=1

Testing
-------

a) conda packaging tool

    # first time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m13.271s
    user    0m9.631s
    sys     0m1.936s

    # second time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m3.159s
    user    0m0.139s
    sys     0m1.298s

b) conda cli

   # first time
   $ time conda list -p /path/to/env
   real    0m7.985s
   user    0m7.883s
   sys     0m0.087s

   # second time
   real    0m0.152s
   user    0m0.116s
   sys     0m0.032s

References
----------
* https://github.com/JetBrains/intellij-community/blob/master/python/helpers/conda_packaging_tool.py
* https://github.com/conda/conda/blob/33a142c16530fcdada6c377486f1c1a385738a96/conda/core/index.py#L53
Owner
tinkerer | software engineer | data scientist
Predict if a fuse is usable on an appliance depending on the fuse rating

fuse-feasibility-analysis Predict if a fuse is usable on an appliance depending on the fuse rating , Power rating and resistance in the appliance

Sebastian Muchui 4 Jul 21, 2022
A basic DIY-project made using Python and MySQL

Banking-Using-Python-MySQL This is a basic DIY-project made using Python and MySQL. Pre-Requisite needed:-- MySQL command Line:- creating a database

ABHISHEK 0 Jul 03, 2022
A tool for removing PUPs using signatures

Unwanted program removal tool A tool for removing PUPs using signatures What is the unwanted program removal tool? The unwanted program removal tool i

4 Sep 20, 2022
Cairo hooks for pre-commit

pre-commit-cairo Cairo hooks for pre-commit. See pre-commit for more details Using pre-commit-cairo with pre-commit Add this to your .pre-commit-confi

Fran Algaba 16 Sep 21, 2022
Python module for creating the circuit simulation definitions for Elmer FEM

elmer_circuitbuilder Python module for creating the circuit simulation definitions for Elmer FEM. The circuit definitions enable easy setup of coils (

5 Oct 03, 2022
Inacap - Programa para pasar las notas de inacap a una hoja de cálculo rápidamente.

Inacap Programa en python para obtener varios datos académicos desde inacap y subirlos directamente a una hoja de cálculo. Cómo funciona Primero que n

Gabriel Barrientos 0 Jul 28, 2022
Izy - Python functions and classes that make python even easier than it is

izy Python functions and classes that make it even easier! You will wonder why t

5 Jul 04, 2022
This is a calculator of strike price distance for options.

Calculator-of-strike-price-distance-for-options This is a calculator of strike price distance for options. Options are a type of derivative. One strat

André Luís Lopes da Silva 4 Dec 30, 2022
Генератор отчетов на Python с использованием библиотеки docx для работы с word-файлами и запросов к сервису

Генератор отчетов на Python с использованием библиотеки docx для работы с word-файлами и запросов к сервису

Semyon Esaev 2 Jun 24, 2022
Virtual Assistant Using Python

-Virtual-Assistant-Using-Python Virtual desktop assistant is an awesome thing. If you want your machine to run on your command like Jarvis did for Ton

Bade om 1 Nov 13, 2021
Set up a sidechain for the XRPL quickly and easily

Sidechain Launch Kit Introduction This directory contains python scripts to tests and explore side chains. This document walks through the steps to se

Xpring Engineering 15 Dec 08, 2022
A professional version for LBS

呐 Yuki Pro~ 懒兵服御用版本,yuki小姐觉得没必要单独造一个仓库,但懒兵觉得有必要并强制执行 将na-yuki框架抽象为模块,功能拆分为独立脚本,使用脚本注释器使其作为py运行 文件结构: na_yuki_pro_example.py 是一个说明脚本,用来直观展示na,yuki! Pro

1 Dec 21, 2021
An OrpheusDL Tidal module

OrpheusDL - Tidal A Tidal module for the OrpheusDL modular archival music program Report Bug · Request Feature Table of content About OrpheusDL - Tida

Daniel 54 Dec 29, 2022
A10 cipher - A Hill 2x2 cipher that totally gone wrong

A10_cipher This is a Hill 2x2 cipher that totally gone wrong, it encrypts with H

Caner Çetin 15 Oct 19, 2022
Design-by-contract in Python3 with informative violation messages and inheritance

icontract icontract provides design-by-contract to Python3 with informative violation messages and inheritance. It also gives a base for a flourishing

275 Jan 02, 2023
rebalance is a simple Python 3.9+ library for rebalancing investment portfolios

rebalance rebalance is a simple Python 3.9+ library for rebalancing investment portfolios. It supports cash flow rebalancing with contributions and wi

Darik Harter 5 Feb 26, 2022
An extension module to make reaction based menus with disnake

disnake-ext-menus An experimental extension menu that makes working with reaction menus a bit easier. Installing python -m pip install -U disnake-ext-

1 Nov 25, 2021
Blender addon that enables exporting of xmodels from blender. Great for custom asset creation for cod games

Birdman's XModel Tools For Blender Greetings everyone in the custom cod community. This blender addon should finally enable exporting of custom assets

wast 2 Jul 02, 2022
OpenSea NFT API App using Python and Streamlit

opensea-nft-api-tutorial OpenSea NFT API App using Python and Streamlit Tutorial Video Walkthrough https://www.youtube.com/watch?v=49SupvcFC1M Instruc

64 Oct 28, 2022
py-js: python3 objects for max

Simple (and extensible) python3 externals for MaxMSP

Shakeeb Alireza 39 Nov 20, 2022