This speeds up PyCharm's package index processes and avoids CPU & memory overloading

Overview
Avoid PyCharm Overloading
=========================

This speeds up PyCharm's package index processes and avoids CPU & memory
overloading.

On my machine (4c/32GB) with 5 conda environments and 10 projects, this reduces
PyCharm's package scanning from 15 minutes to 45 seconds.

Why?
----

When PyCharm has multiple conda environments, it will sometimes launch concurrent
package scanning across those environments. To do so, it uses the `conda` cli,
and a tool called conda_packaging_tool.py, available from the JetBrains
intellij-community github repository (see below).

Unfortunately, these tools do not behave well when launched in parallel, as
they tend to overload CPU & memory in this case.

How?
----

To avoid stalling these processes, we modify their sources as follows:

1. modify the conda cli

    $ conda activate base
    $ pip install runfast
    $ nano `which conda`

    Modify the main section:

     if __name__ == '__main__':
        from conda.cli import main
        from runfast import cached
        cached(main)

2. modify the conda_packaging_tool.py

    if __name__ == '__main__':
        from runfast import cached
        cached(main)

How does this work?
-------------------

Two simple steps:

1. When called in parallel, only one process is allowed to proceed.
2. `runfast.cached` caches the output of these tools (stdout, stderr) for 1 minute,
   given the same command line parameters

That is, when PyCharm launches 5 package scanning (`conda list`) commands, only
one of them will run immediately. If some of the scans are for the same environment,
only one of them will actually run, while the others simply return the same output.

How to clear the cache or avoid caching at all?

    $ export RUNFAST_NOCACHE=1

Testing
-------

a) conda packaging tool

    # first time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m13.271s
    user    0m9.631s
    sys     0m1.936s

    # second time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m3.159s
    user    0m0.139s
    sys     0m1.298s

b) conda cli

   # first time
   $ time conda list -p /path/to/env
   real    0m7.985s
   user    0m7.883s
   sys     0m0.087s

   # second time
   real    0m0.152s
   user    0m0.116s
   sys     0m0.032s

References
----------
* https://github.com/JetBrains/intellij-community/blob/master/python/helpers/conda_packaging_tool.py
* https://github.com/conda/conda/blob/33a142c16530fcdada6c377486f1c1a385738a96/conda/core/index.py#L53
Owner
tinkerer | software engineer | data scientist
Basic Clojure REPL for Sublime Text

Basic Clojure REPL for Sublime Text Goals: Decomplected: just REPL, nothing more Zero dependencies: works directly with pREPL Compact: Display code ev

Nikita Prokopov 23 Dec 24, 2021
Suite of tools for retrieving USGS NWIS observations and evaluating National Water Model (NWM) data.

Documentation OWPHydroTools GitHub pages documentation Motivation We developed OWPHydroTools with data scientists in mind. We attempted to ensure the

36 Dec 11, 2022
Social reading and reviewing, decentralized with ActivityPub

BookWyrm Social reading and reviewing, decentralized with ActivityPub Contents Joining BookWyrm Contributing About BookWyrm What it is and isn't The r

BookWyrm 1.4k Jan 08, 2023
Add all JuliaLang unicode abbreviations to AutoKey.

Autokey Unicode characters Usage This script adds all the unicode character abbreviations supported by Julia to autokey. However, instead of [TAB], th

Randolf Scholz 49 Dec 02, 2022
馃寛Python cheatsheet for all standard libraries(Continuously Updated)

Python Standard Libraries Cheatsheet Depend on Python v3.9.8 All code snippets have been tested to ensure they work properly. Fork me on GitHub. 涓枃 En

nick 12 Dec 27, 2022
This alerts you when the avalanche score a goal

This alerts you when the avalanche score a goal

Davis Burrill 1 Jan 15, 2022
Python most simple|stupid programming language (MSPL)

Most Simple|Stupid Programming language. (MSPL) Stack - Based programming language "written in Python" Features: Interpretate code (Run). Generate gra

Kirill Zhosul 14 Nov 03, 2022
Program Input Nilai Mahasiswa Menggunakan Fungsi

PROGRAM INPUT NILAI MAHASISWA MENGGUNAKAN FUNGSI Nama : Maulana Reza Badrudin Nim : 312110510 Matkul : Bahas Pemograman DESKRIPSI Deklarasi dicti

Maulana Reza Badrudin 1 Jan 05, 2022
Security-related flags and options for C compilers

Getting the maximum of your C compiler, for security

135 Nov 11, 2022
Python library for ODE integration via Taylor's method and LLVM

heyoka.py Modern Taylor's method via just-in-time compilation Explore the docs 禄 Report bug 路 Request feature 路 Discuss The hey贸k葻a [...] is a kind of

Francesco Biscani 45 Dec 21, 2022
A telegram bot which programed to countdown.

countdown-vi this is a telegram bot which programed to countdown. usage well, first you should specify a exact interval. there is 5 column, very first

Arya Shabane 3 Feb 15, 2022
Reposit贸rio de c贸digo de curso de Djavue ministrado na Python Brasil 2021

djavue-python-brasil Reposit贸rio de c贸digo de curso de Djavue ministrado na Python Brasil 2021 Completamente baseado no curso Djavue. A diferen莽a est谩

Buser 15 Dec 26, 2022
Simple Wayland HotKey Daemon

swhkd Simple Wayland HotKey Daemon This project is still very new and I'm making new decisions everyday as to where I should drive this project. I'm u

Aakash Sen Sharma 407 Dec 30, 2022
BloodCheck enables Red and Blue Teams to manage multiple Neo4j databases and run Cypher queries against a BloodHound dataset.

BloodCheck BloodCheck enables Red and Blue Teams to manage multiple Neo4j databases and run Cypher queries against a BloodHound dataset. Installation

Mr B0b 16 Nov 05, 2021
pydock - Docker-based environment manager for Python

pydock - Docker-based environment manager for Python 鈿狅笍 pydock is still in beta mode, and very unstable. It is not recommended for anything serious. p

Alejandro Piad 16 Sep 18, 2021
A python trivium implemention

A python trivium implemention

tnt2402 1 Nov 12, 2021
Frappe app for authentication, can be used with FrappeVue-AdminLTE

Frappeauth App Frappe app for authentication, can be used with FrappeVue-AdminLTE

Anthony C. Emmanuel 9 Apr 13, 2022
Agora-token-helper - Some help tools for AgoraToken

Agora Token Helper Support AgoraToken version 001 - 006. But for security reason

Simple tooling for marking deprecated functions or classes and re-routing to the new successors' instance.

pyDeprecate Simple tooling for marking deprecated functions or classes and re-routing to the new successors' instance

Jirka Borovec 45 Nov 24, 2022
Irrigation Component V4 providing support for a custom card

Irrigation Component V4 This release sees the delivery of a custom card https://github.com/petergridge/irrigation_card to render the program options s

12 Oct 28, 2022