This speeds up PyCharm's package index processes and avoids CPU & memory overloading

Overview
Avoid PyCharm Overloading
=========================

This speeds up PyCharm's package index processes and avoids CPU & memory
overloading.

On my machine (4c/32GB) with 5 conda environments and 10 projects, this reduces
PyCharm's package scanning from 15 minutes to 45 seconds.

Why?
----

When PyCharm has multiple conda environments, it will sometimes launch concurrent
package scanning across those environments. To do so, it uses the `conda` cli,
and a tool called conda_packaging_tool.py, available from the JetBrains
intellij-community github repository (see below).

Unfortunately, these tools do not behave well when launched in parallel, as
they tend to overload CPU & memory in this case.

How?
----

To avoid stalling these processes, we modify their sources as follows:

1. modify the conda cli

    $ conda activate base
    $ pip install runfast
    $ nano `which conda`

    Modify the main section:

     if __name__ == '__main__':
        from conda.cli import main
        from runfast import cached
        cached(main)

2. modify the conda_packaging_tool.py

    if __name__ == '__main__':
        from runfast import cached
        cached(main)

How does this work?
-------------------

Two simple steps:

1. When called in parallel, only one process is allowed to proceed.
2. `runfast.cached` caches the output of these tools (stdout, stderr) for 1 minute,
   given the same command line parameters

That is, when PyCharm launches 5 package scanning (`conda list`) commands, only
one of them will run immediately. If some of the scans are for the same environment,
only one of them will actually run, while the others simply return the same output.

How to clear the cache or avoid caching at all?

    $ export RUNFAST_NOCACHE=1

Testing
-------

a) conda packaging tool

    # first time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m13.271s
    user    0m9.631s
    sys     0m1.936s

    # second time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m3.159s
    user    0m0.139s
    sys     0m1.298s

b) conda cli

   # first time
   $ time conda list -p /path/to/env
   real    0m7.985s
   user    0m7.883s
   sys     0m0.087s

   # second time
   real    0m0.152s
   user    0m0.116s
   sys     0m0.032s

References
----------
* https://github.com/JetBrains/intellij-community/blob/master/python/helpers/conda_packaging_tool.py
* https://github.com/conda/conda/blob/33a142c16530fcdada6c377486f1c1a385738a96/conda/core/index.py#L53
Owner
tinkerer | software engineer | data scientist
A visidata plugin for parsing f5 ltm/gtm/audit logs

F5 Log Visidata Plugin This plugin supports the default log format for: /var/log/ltm* /var/log/gtm* /var/log/apm* /var/log/audit* It extracts common l

James Deucker 1 Jan 06, 2022
This project intends to take the user's CEP (brazilian adress code) and return the local in which the CEP is placed.

This project aims to simply return the CEP's (the brazilian resident adress code) User of the application. The project uses a request and passes on to

Daniel Soares Saldanha 4 Nov 17, 2021
pybicyclewheel calulates the required spoke length for bicycle wheels

pybicyclewheel pybicyclewheel calulates the required spoke length for bicycle wheels. (under construcion) - homepage further readings wikipedia bicyc

karl 0 Aug 24, 2022
RFDesign - Protein hallucination and inpainting with RoseTTAFold

RFDesign: Protein hallucination and inpainting with RoseTTAFold Jue Wang (juewan

139 Jan 06, 2023
一个Graia-Saya的插件仓库

一个Graia-Saya的插件仓库 这是一个存储基于 Graia-Saya 的插件的仓库 如果您有这类项目

ZAPHAKIEL 111 Oct 24, 2022
A very simple boarding app with DRF

CRUD project with DRF A very simple boarding app with DRF. About The Project 유저 정보를 갖고 게시판을 다루는 프로젝트 입니다. Version Python: 3.9 DB: PostgreSQL 13 Django

1 Nov 13, 2021
A project to work with databases in 4 worksheets, insert, update, select, delete using Python and MySqI

A project to work with databases in 4 worksheets, insert, update, select, delete using Python and MySqI As a small project for school or college hope it is useful

Sina Org 1 Jan 11, 2022
Experimental proxy for dumping the unencrypted packet data from Brawl Stars (WIP)

Brawl Stars Proxy Experimental proxy for version 39.99 of Brawl Stars. It allows you to capture the packets being sent between the Brawl Stars client

4 Oct 29, 2021
Add your recently blog and douban states in your GitHub Profile

Add your recently blog and douban states in your GitHub Profile

Bingjie Yan 4 Dec 12, 2022
These are the scripts used for the project of ‘Assembly of a pan-genome for global cattle reveals missing sequence and novel structural variation, providing new insights into their diversity and evolution history’

script-SV-genotyping These are the scripts used for the project of ‘Assembly of a pan-genome for global cattle reveals missing sequence and novel stru

2 Aug 26, 2022
Random Programming Language Project

Crastle Random Programming Language Project Freedom of expression Are you a fan of curly brace languages? Then use curly braces! Not a fan of curly br

DevNugget 2 Dec 23, 2021
Plock : A stack based programming language

Plock : A stack based programming language

1 Oct 25, 2021
Python plugin/extra to load data files from an external source (such as AWS S3) to a local directory

Data Loader Plugin - Python Table of Content (ToC) Data Loader Plugin - Python Table of Content (ToC) Overview References Python module Python virtual

Cloud Helpers 2 Jan 10, 2022
TallerStereoVision Convencion Python Chile 2021

TallerStereoVision Convencion Python Chile 2021 Taller Stereo Vision & Python PyCon.cl 2021 Instalación Se recomienta utilizar Virtual Environment pyt

2 Oct 20, 2022
Advanced Variable Manager {AVM} [0.8.0]

Advanced Variable Manager {AVM} [0.8.0] By Grosse pastèque#6705 WARNING : This modules need some typing modifications ! If you try to run it without t

Big watermelon 1 Dec 11, 2021
Scrapper For Paste.pics

PrntScScrapper Scrapper for Paste.pics If you are bored you can find some random screenshots from prnt.sc Features Saving screenshots Open in Browser

Fareusz 1 Dec 29, 2021
Collection of system-wide scripts that I use on my Gentoo

linux-scripts Collection of scripts that I use on my Gentoo machine. I tend to put all scripts in /scripts directory. It is not likely that you would

Xoores 1 Jan 09, 2022
Free and open source qualitative research tool

Taguette A spin on the phrase "tag it!", Taguette is a free and open source qualitative research tool that allows users to: Import PDFs, Word Docs (.d

Remi Rampin 48 Jan 02, 2023
A simple project which is a ecm to found a good way to provide a path to img_dir in gooey

ECM to find a good way for img_dir Path in Gooey This code is just an ECM to find a good way to indicate a path of image in image_dir variable. We loo

Jean-Emmanuel Longueville 1 Oct 25, 2021
Gba-free-fonts - Free font resources for GBA game development

gba-free-fonts Free font resources for GBA game development This repo contains m

28 Dec 30, 2022