This speeds up PyCharm's package index processes and avoids CPU & memory overloading

Overview
Avoid PyCharm Overloading
=========================

This speeds up PyCharm's package index processes and avoids CPU & memory
overloading.

On my machine (4c/32GB) with 5 conda environments and 10 projects, this reduces
PyCharm's package scanning from 15 minutes to 45 seconds.

Why?
----

When PyCharm has multiple conda environments, it will sometimes launch concurrent
package scanning across those environments. To do so, it uses the `conda` cli,
and a tool called conda_packaging_tool.py, available from the JetBrains
intellij-community github repository (see below).

Unfortunately, these tools do not behave well when launched in parallel, as
they tend to overload CPU & memory in this case.

How?
----

To avoid stalling these processes, we modify their sources as follows:

1. modify the conda cli

    $ conda activate base
    $ pip install runfast
    $ nano `which conda`

    Modify the main section:

     if __name__ == '__main__':
        from conda.cli import main
        from runfast import cached
        cached(main)

2. modify the conda_packaging_tool.py

    if __name__ == '__main__':
        from runfast import cached
        cached(main)

How does this work?
-------------------

Two simple steps:

1. When called in parallel, only one process is allowed to proceed.
2. `runfast.cached` caches the output of these tools (stdout, stderr) for 1 minute,
   given the same command line parameters

That is, when PyCharm launches 5 package scanning (`conda list`) commands, only
one of them will run immediately. If some of the scans are for the same environment,
only one of them will actually run, while the others simply return the same output.

How to clear the cache or avoid caching at all?

    $ export RUNFAST_NOCACHE=1

Testing
-------

a) conda packaging tool

    # first time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m13.271s
    user    0m9.631s
    sys     0m1.936s

    # second time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m3.159s
    user    0m0.139s
    sys     0m1.298s

b) conda cli

   # first time
   $ time conda list -p /path/to/env
   real    0m7.985s
   user    0m7.883s
   sys     0m0.087s

   # second time
   real    0m0.152s
   user    0m0.116s
   sys     0m0.032s

References
----------
* https://github.com/JetBrains/intellij-community/blob/master/python/helpers/conda_packaging_tool.py
* https://github.com/conda/conda/blob/33a142c16530fcdada6c377486f1c1a385738a96/conda/core/index.py#L53
Owner
tinkerer | software engineer | data scientist
OTP-Bomber - An otp from MPL ID app, which can be spammed

OTP-Bomber An otp from MPL ID app, which can be spammed Note: Only available on

5 Oct 29, 2022
A modern Python build backend

trampolim A modern Python build backend. Features Task system, allowing to run arbitrary Python code during the build process (Planned) Easy to use CL

Filipe Laíns 39 Nov 08, 2022
Experimental proxy for dumping the unencrypted packet data from Brawl Stars (WIP)

Brawl Stars Proxy Experimental proxy for version 39.99 of Brawl Stars. It allows you to capture the packets being sent between the Brawl Stars client

4 Oct 29, 2021
A common, beautiful interface to tabular data, no matter the format

rows No matter in which format your tabular data is: rows will import it, automatically detect types and give you high-level Python objects so you can

Álvaro Justen 834 Jan 03, 2023
API development made easy: a smart Python 3 API framework

appkernel - API development made easy What is Appkernel? A super-easy to use API framework, enabling API creation from zero to production within minut

156 Sep 28, 2022
1000+ ready code templates to kickstart your next AI experiment

AI Seed Projects Start with ready code for your next AI experiment. Choose from 1000+ code templates, across a wide variety of use cases. All examples

BlobCity, Inc 98 Jan 03, 2023
Amitkumar Mishra 2 Jan 14, 2022
Code repository for the Pytheas submersible observation platform

Pytheas Main repository for the Pytheas submersible probe system. List of Acronyms/Terms USP - Underwater Sensor Platform - The primary platform in th

UltraChip 2 Nov 19, 2022
A test repository to build a python package and publish the package to Artifact Registry using GCB

A test repository to build a python package and publish the package to Artifact Registry using GCB. Then have the package be a dependency in a GCF function.

1 Feb 09, 2022
Google Fit Sensor Component

Google Fit Sensor Component

Ivan Vojtko 21 Dec 20, 2022
This is a batch script created to WEB-DL.

widevine-L3-WEB-DL-Script This is a batch script created to WEB-DL. Works well with .mpd files , for m3u8 please use n_m3u8 program (not included in t

Paranjay Singh 312 Dec 31, 2022
Writeup of NilbinSec's participation in the Winja CTF for c0c0n 2021

Winja-CTF-c0c0n-2021-Writeup NilbinSec's participation in the Winja CTF for c0c0n 2021 This repo covers NilbinSec's participation in the Winja CTF dur

1 Nov 15, 2021
Implemented Exploratory Data Analysis (EDA) using Python.Built a dashboard in Tableau and found that 45.87% of People suffer from heart disease.

Heart_Disease_Diagnostic_Analysis Objective 🎯 The aim of this project is to use the given data and perform ETL and data analysis to infer key metrics

Sultan Shaikh 4 Jan 28, 2022
Visualization of COVID-19 Omicron wave data in Seoul, Osaka, Tokyo, Hong Kong and Shanghai. 首尔、大阪、东京、香港、上海由新冠病毒 Omicron 变异株引起的本轮疫情数据可视化分析。

COVID-19 in East Asian Megacities This repository holds original Python code for processing and visualization COVID-19 data in East Asian megacities a

STONE 10 May 18, 2022
Winxp_python3.6.15 - Python 3.6.15 For Windows XP SP3

This is Python version 3.6.15 Copyright (c) 2001-2021 Python Software Foundation. All rights reserved. See the end of this file for further copyright

Alex Free 13 Sep 11, 2022
A set of simple functions to upload and fetch pastes on paste.uploadgram.me

pastegram-py A set of simple functions to upload and fetch pastes on paste.uploadgram.me. API Documentation Methods upload_paste(contents: bytes, file

Uploadgram 3 Sep 13, 2022
Utility/Raiding selfbot made by Shell and Roover.

Utility/Raiding selfbot made by Shell and Roover. We are open to suggestions and ideas.

Shell 2 Dec 08, 2021
Simple, configuration-driven backup software for servers and workstations

title permalink borgmatic index.html It's your data. Keep it that way. borgmatic is simple, configuration-driven backup software for servers and works

borgmatic collective 1.3k Dec 30, 2022
You can easily send campaigns, e-marketing have actually account using cash will thank you for using our tools, and you can support our Vodafone Cash +201090788026

*** Welcome User Sorry I Mean Hello Brother ✓ Devolper and Design : Mokhtar Abdelkreem ========================================== You Can Follow Us O

Mo Code 1 Nov 03, 2021
A timer for bird lovers, plays a random birdcall while displaying its image and info.

Birdcall Timer A timer for bird lovers. Siriema hatchling by Junior Peres Junior Background My partner needed a customizable timer for sitting and sta

Marcelo Sanches 1 Jul 08, 2022