This speeds up PyCharm's package index processes and avoids CPU & memory overloading

Overview
Avoid PyCharm Overloading
=========================

This speeds up PyCharm's package index processes and avoids CPU & memory
overloading.

On my machine (4c/32GB) with 5 conda environments and 10 projects, this reduces
PyCharm's package scanning from 15 minutes to 45 seconds.

Why?
----

When PyCharm has multiple conda environments, it will sometimes launch concurrent
package scanning across those environments. To do so, it uses the `conda` cli,
and a tool called conda_packaging_tool.py, available from the JetBrains
intellij-community github repository (see below).

Unfortunately, these tools do not behave well when launched in parallel, as
they tend to overload CPU & memory in this case.

How?
----

To avoid stalling these processes, we modify their sources as follows:

1. modify the conda cli

    $ conda activate base
    $ pip install runfast
    $ nano `which conda`

    Modify the main section:

     if __name__ == '__main__':
        from conda.cli import main
        from runfast import cached
        cached(main)

2. modify the conda_packaging_tool.py

    if __name__ == '__main__':
        from runfast import cached
        cached(main)

How does this work?
-------------------

Two simple steps:

1. When called in parallel, only one process is allowed to proceed.
2. `runfast.cached` caches the output of these tools (stdout, stderr) for 1 minute,
   given the same command line parameters

That is, when PyCharm launches 5 package scanning (`conda list`) commands, only
one of them will run immediately. If some of the scans are for the same environment,
only one of them will actually run, while the others simply return the same output.

How to clear the cache or avoid caching at all?

    $ export RUNFAST_NOCACHE=1

Testing
-------

a) conda packaging tool

    # first time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m13.271s
    user    0m9.631s
    sys     0m1.936s

    # second time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m3.159s
    user    0m0.139s
    sys     0m1.298s

b) conda cli

   # first time
   $ time conda list -p /path/to/env
   real    0m7.985s
   user    0m7.883s
   sys     0m0.087s

   # second time
   real    0m0.152s
   user    0m0.116s
   sys     0m0.032s

References
----------
* https://github.com/JetBrains/intellij-community/blob/master/python/helpers/conda_packaging_tool.py
* https://github.com/conda/conda/blob/33a142c16530fcdada6c377486f1c1a385738a96/conda/core/index.py#L53
Owner
tinkerer | software engineer | data scientist
An interactive tool with which to explore the possible imaging performance of candidate ngEHT architectures.

ngEHTexplorer An interactive tool with which to explore the possible imaging performance of candidate ngEHT architectures. Welcome! ngEHTexplorer is a

Avery Broderick 7 Jan 28, 2022
A small Python library which gives you the IEEE-754 representation of a floating point number.

ieee754 ieee754 is small Python library which gives you the IEEE-754 representation of a floating point number. You can specify a precision given in t

Bora Canbula 5 Dec 20, 2022
Statically typed BNF with semantic actions; A frontend of frontend frameworks; Use your grammar everywhere.

Statically typed BNF with semantic actions; A frontend of frontend frameworks; Use your grammar everywhere.

Taine Zhao 56 Dec 14, 2022
You can change your mac address with this program.

1 - Warning! You can use this program with Kali Linux. Therefore if you don't install the Kali Linux. Firstly you need to install Kali Linux. 2 - Star

Mustafa Bahadır Doğrusöz 1 Jun 10, 2022
Python interface to ISLEX, an English IPA pronunciation dictionary with syllable and stress marking.

pysle Questions? Comments? Feedback? Pronounced like 'p' + 'isle'. An interface to a pronunciation dictionary with stress markings (ISLEX - the intern

Tim 38 Dec 14, 2022
Skip spotify ads by automatically restarting application when ad comes

SpotiByeAds No one likes interruptions! Don't you hate it when you're listening to your favorite jazz track or your EDM playlist and an ad for Old Spi

Partho 287 Dec 29, 2022
PyGo custom language, New but similar language programming

New but similar language programming. Now we are capable to program in a very similar language to Python but at the same time get the efficiency of Go.

Fernando Perez 4 Nov 19, 2022
Better Giveaways is a bot that will change the experience of using a giveaway bot forever.

Better-Giveaways Better Giveaways is a bot that will change the experience of using a giveaway bot forever. VoxelBotUtils/Novus, latest PyPi releases

Lightning 2 Jan 12, 2022
The goal of this program was to find the most common color in my living room.

The goal of this program was to find the most common color in my living room. I found a dataset online with colors names and their corr

1 Nov 09, 2021
Freeze your objects in python

gelidum Freeze your objects in python. Latin English Caelum est hieme frigidum et gelidum; myrtos oleas quaeque alia assiduo tepore laetantur, asperna

Diego J. 51 Dec 22, 2022
Convert long numbers into a human-readable format in Python

Convert long numbers into a human-readable format in Python

Alex Zaitsev 73 Dec 28, 2022
Trackthis - This library can be used to track USPS and UPS shipments.

Trackthis - This library can be used to track USPS and UPS shipments. It has the option of returning the raw API response, or optionally, it can be used to standardize the USPS and UPS responses so t

Aaron Guzman 0 Mar 29, 2022
A Python version of Canvacord

A copy of canvacord made in python! Table of contents Installation Examples Creating Images Links Downloads Installation Run any of these commands in

10 Mar 28, 2022
Collection of system-wide scripts that I use on my Gentoo

linux-scripts Collection of scripts that I use on my Gentoo machine. I tend to put all scripts in /scripts directory. It is not likely that you would

Xoores 1 Jan 09, 2022
A tool to determine optimal projects for Gridcoin crunchers. Maximize your magnitude!

FindTheMag FindTheMag helps optimize your BOINC client for Gridcoin mining. You can group BOINC projects into two groups: "preferred" projects and "mi

7 Oct 04, 2022
Excel cell checker with python

excel-cell-checker Description This tool checks a given .xlsx file has the struc

Paul Aumann 1 Jan 04, 2022
A simple armature retargeting tool for Blender

Simple-Retarget-Tool-Blender A simple armature retargeting tool for Blender Update V2: Set Rest Pose to easily apply rest pose. Preset Import/Export.

Fahad Hasan Pathik 74 Jan 04, 2023
Minterpy - Multidimensional interpolation in Python.

minterpy is an open-source Python package for a multivariate generalization of the classical Newton and Lagrange interpolation schemes as well as related tasks.

Center for Advanced Systems Understanding 18 Jan 06, 2023
A basic DIY-project made using Python and MySQL

Banking-Using-Python-MySQL This is a basic DIY-project made using Python and MySQL. Pre-Requisite needed:-- MySQL command Line:- creating a database

ABHISHEK 0 Jul 03, 2022
Python3 Interface to numa Linux library

py-libnuma is python3 interface to numa Linux library so that you can set task affinity and memory affinity in python level for your process which can help you to improve your code's performence.

Dalong 13 Nov 10, 2022