This speeds up PyCharm's package index processes and avoids CPU & memory overloading

Overview
Avoid PyCharm Overloading
=========================

This speeds up PyCharm's package index processes and avoids CPU & memory
overloading.

On my machine (4c/32GB) with 5 conda environments and 10 projects, this reduces
PyCharm's package scanning from 15 minutes to 45 seconds.

Why?
----

When PyCharm has multiple conda environments, it will sometimes launch concurrent
package scanning across those environments. To do so, it uses the `conda` cli,
and a tool called conda_packaging_tool.py, available from the JetBrains
intellij-community github repository (see below).

Unfortunately, these tools do not behave well when launched in parallel, as
they tend to overload CPU & memory in this case.

How?
----

To avoid stalling these processes, we modify their sources as follows:

1. modify the conda cli

    $ conda activate base
    $ pip install runfast
    $ nano `which conda`

    Modify the main section:

     if __name__ == '__main__':
        from conda.cli import main
        from runfast import cached
        cached(main)

2. modify the conda_packaging_tool.py

    if __name__ == '__main__':
        from runfast import cached
        cached(main)

How does this work?
-------------------

Two simple steps:

1. When called in parallel, only one process is allowed to proceed.
2. `runfast.cached` caches the output of these tools (stdout, stderr) for 1 minute,
   given the same command line parameters

That is, when PyCharm launches 5 package scanning (`conda list`) commands, only
one of them will run immediately. If some of the scans are for the same environment,
only one of them will actually run, while the others simply return the same output.

How to clear the cache or avoid caching at all?

    $ export RUNFAST_NOCACHE=1

Testing
-------

a) conda packaging tool

    # first time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m13.271s
    user    0m9.631s
    sys     0m1.936s

    # second time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m3.159s
    user    0m0.139s
    sys     0m1.298s

b) conda cli

   # first time
   $ time conda list -p /path/to/env
   real    0m7.985s
   user    0m7.883s
   sys     0m0.087s

   # second time
   real    0m0.152s
   user    0m0.116s
   sys     0m0.032s

References
----------
* https://github.com/JetBrains/intellij-community/blob/master/python/helpers/conda_packaging_tool.py
* https://github.com/conda/conda/blob/33a142c16530fcdada6c377486f1c1a385738a96/conda/core/index.py#L53
Owner
tinkerer | software engineer | data scientist
SpaCy3Urdu: run command to setup assets(dataset from UD)

Project setup run command to setup assets(dataset from UD) spacy project assets It uses project.yml file and download the data from UD GitHub reposito

Muhammad Irfan 1 Dec 14, 2021
An alternative app for core Armoury Crate functions.

NoROG DISCLAIMER: Use at your own risk. This is alpha-quality software. It has not been extensively tested, though I personally run it daily on my lap

12 Nov 29, 2022
一个Graia-Saya的插件仓库

一个Graia-Saya的插件仓库 这是一个存储基于 Graia-Saya 的插件的仓库 如果您有这类项目

ZAPHAKIEL 111 Oct 24, 2022
Subcert is an subdomain enumeration tool, that finds all the subdomains from certificate transparency logs.

Subcert Subcert is a subdomain enumeration tool, that finds all the valid subdomains from certificate transparency logs. Table of contents Setup Demo

A3h1nt 59 Dec 16, 2022
✔️ Create to-do lists to easily manage your ideas and work.

Todo List + Add task + Remove task + List completed task + List not completed task + Set clock task time + View task statistics by date Changelog v 1.

Abbas Ataei 30 Nov 28, 2022
WhyNotWin11 - Detection Script to help identify why your PC isn't Windows 11 Release Ready

WhyNotWin11 - Detection Script to help identify why your PC isn't Windows 11 Release Ready

Robert C. Maehl 5.9k Dec 31, 2022
UFDR2DIR - A script to convert a Cellebrite UFDR to the original file structure

UFDR2DIR A script to convert a Cellebrite UFDR to it's original file and directo

DFIRScience 25 Oct 24, 2022
adbsync - An ADB syncing helper

adbsync - An ADB syncing helper What's this? Everytime I wanted to make a backup of my phone, or restore those files onto it, I had to use everytime t

Giovanni Gualtieri 3 Aug 05, 2022
MySQL Connectivity based project. Contains various functions of a Store-Management-System

An Intermediate Level Python - MySQL Connectivity based project. Contains various functions of a Store-Management-System.

Yash Wadhvani 2 Nov 21, 2022
Web interface for browsing, search and filtering recent arxiv submissions

Web interface for browsing, search and filtering recent arxiv submissions

Andrej 4.8k Jan 08, 2023
Daily knowledge pills to get better in Python.

Python daily pills Daily knowledge pills to get better Python code. Why Does your Python code suffers of any of this symptoms? Incorrect Indentation I

Jeferson Vaz dos Santos 35 Sep 19, 2022
Blender-3D-SH-Dma-plugin - Import and export Sonic Heroes Delta Morph animations (.anm) into Blender 3D

io_scene_sonic_heroes_dma This plugin for Blender 3D allows you to import and ex

Psycrow 3 Mar 22, 2022
python scripts and other files to generate induction encoder PCBs in Kicad

induction_encoder python scripts and other files to generate induction encoder PCBs in Kicad Targeting the Renesas IPS2200 encoder chips.

Taylor Alexander 8 Feb 16, 2022
A prototype COG-based tile server for sparse Mars datasets

Mars tiler Mars Tiler is a prototype web application that serves tiles from cloud-optimized GeoTIFFs, with an emphasis on supporting planetary dataset

Daven Quinn 3 Mar 23, 2022
a package that provides a marketstrategy for whitelisting on golem

filterms a package that provides a marketstrategy for whitelisting on golem watching requestor logs distribute 10 tasks asynchronously is fun. but you

KJM 3 Aug 03, 2022
Gunakan Dengan Bijak!!

YMBF Made with ❤️ by ikiwzXD_ menu Results notice me: if you get cp results, save 3/7 days then log in. Install script on Termux $ pkg update && pkg u

Ikiwz 0 Jul 11, 2022
Simple logger for Urbit pier size, with systemd timer template

urbit-piermon Simple logger for Urbit pier size, with systemd timer template. Syntax piermon.py -i [PATH TO PIER] -o [PATH TO OUTPUT CSV] systemd serv

1 Nov 07, 2021
디텍션 유틸 모음

Object detection utils 유틸모음 설명 링크 convert convert 관련코드 https://github.com/AI-infinyx/ob_utils/tree/main/convert crawl 구글, 네이버, 빙 등 크롤링 관련 https://gith

codetest 41 Jan 22, 2021
python scripts to perform coin die clustering (performed on Riedones3D).

python scripts to perform coin die clustering (performed on Riedones3D).

Sofiane 2 Apr 29, 2022
NasaApod - Astronomy Picture of the Day

Astronomy Picture of the Day Get interesting Astronomical pictures with a brief

Shripad Rao 1 Feb 15, 2022