This speeds up PyCharm's package index processes and avoids CPU & memory overloading

Overview
Avoid PyCharm Overloading
=========================

This speeds up PyCharm's package index processes and avoids CPU & memory
overloading.

On my machine (4c/32GB) with 5 conda environments and 10 projects, this reduces
PyCharm's package scanning from 15 minutes to 45 seconds.

Why?
----

When PyCharm has multiple conda environments, it will sometimes launch concurrent
package scanning across those environments. To do so, it uses the `conda` cli,
and a tool called conda_packaging_tool.py, available from the JetBrains
intellij-community github repository (see below).

Unfortunately, these tools do not behave well when launched in parallel, as
they tend to overload CPU & memory in this case.

How?
----

To avoid stalling these processes, we modify their sources as follows:

1. modify the conda cli

    $ conda activate base
    $ pip install runfast
    $ nano `which conda`

    Modify the main section:

     if __name__ == '__main__':
        from conda.cli import main
        from runfast import cached
        cached(main)

2. modify the conda_packaging_tool.py

    if __name__ == '__main__':
        from runfast import cached
        cached(main)

How does this work?
-------------------

Two simple steps:

1. When called in parallel, only one process is allowed to proceed.
2. `runfast.cached` caches the output of these tools (stdout, stderr) for 1 minute,
   given the same command line parameters

That is, when PyCharm launches 5 package scanning (`conda list`) commands, only
one of them will run immediately. If some of the scans are for the same environment,
only one of them will actually run, while the others simply return the same output.

How to clear the cache or avoid caching at all?

    $ export RUNFAST_NOCACHE=1

Testing
-------

a) conda packaging tool

    # first time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m13.271s
    user    0m9.631s
    sys     0m1.936s

    # second time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m3.159s
    user    0m0.139s
    sys     0m1.298s

b) conda cli

   # first time
   $ time conda list -p /path/to/env
   real    0m7.985s
   user    0m7.883s
   sys     0m0.087s

   # second time
   real    0m0.152s
   user    0m0.116s
   sys     0m0.032s

References
----------
* https://github.com/JetBrains/intellij-community/blob/master/python/helpers/conda_packaging_tool.py
* https://github.com/conda/conda/blob/33a142c16530fcdada6c377486f1c1a385738a96/conda/core/index.py#L53
Owner
tinkerer | software engineer | data scientist
Tie together `drf-spectacular` and `djangorestframework-dataclasses` for easy-to-use apis and openapi schemas.

Speccify Tie together drf-spectacular and djangorestframework-dataclasses for easy-to-use apis and openapi schemas. Usage @dataclass class MyQ

Lyst 4 Sep 26, 2022
Materials for the Introduction in Python , Linux , Git and Github

This repository contains all the materials of the presentation on the introduction of python, linux, git and Github.

AMMI 3 Aug 28, 2022
contextlib2 is a backport of the standard library's contextlib module to earlier Python versions.

contextlib2 is a backport of the standard library's contextlib module to earlier Python versions. It also sometimes serves as a real world proving gro

Jazzband 35 Dec 23, 2022
A data driven app for bicycle hiring in London(UK)

bicycle_hiring_app_deployed A data driven app for bicycle hiring in London(UK). It predicts expected number of bicycle hire in London. It asks users t

Rajarshi Roy Raju 1 Dec 10, 2021
Incident Response Process and Playbooks | Goal: Playbooks to be Mapped to MITRE Attack Techniques

PURPOSE OF PROJECT That this project will be created by the SOC/Incident Response Community Develop a Catalog of Incident Response Playbook for every

Austin Songer 987 Jan 02, 2023
Sodium is a general purpose programming language which is instruction-oriented

Sodium is a general purpose programming language which is instruction-oriented (a new programming concept that we are developing and devising)

Satin Wuker 22 Jan 11, 2022
Korg Volca Sample uploader for linux.

GnuVolca Korg Volca Sample uploader for linux. GnuVolca Usage Installation Via virtualenv Usage Store all the samples you want to upload on an empty d

Gonzalo Rafuls 12 Oct 11, 2022
A simple program to run through inputs for a 3n+1 problem

Author Tyler Windemuth Collatz_Conjecture A simple program to run through inputs for a 3n+1 problem Purpose: doesn't really have a purpose, did this t

0 Apr 22, 2022
Скрипт позволяет выгрузить участников чатов/каналов(по чату для комментариев) и сообщения в различные форматы файлов.

TG-Parser Парсер участников и сообщений из ТГ-Чатов и чатов для комментариев в ТГ-Каналах Возможности Выгрузка участников групп/каналов(по чату для ко

50 Jan 06, 2023
PIP Manager written in python Tkinter

PIP Manager About PIP Manager is designed to make Python Package handling easier by just a click of a button!! Available Features Installing packages

Will Payne 9 Dec 09, 2022
A simple script that shows important photography times. written in python.

A simple script that shows important photography times. written in python.

John Evans 13 Oct 16, 2022
Small Arrow Vortex clipboard processing library

Description Small Arrow Vortex clipboard processing library. Install You can install this library from PyPI with pip install av-clipboard-lib or compi

Delta Epsilon 1 Dec 18, 2021
Alternative StdLib for Nim for Python targets

Alternative StdLib for Nim for Python targets, hijacks Python StdLib for Nim

Juan Carlos 100 Jan 01, 2023
Get the stats of a (or more) Hypixel player(s)

Hypixel_Stats Get the statistics of a (or more) Hypixel player(s) Who needs this? Everyone who plays a lot of Minecraft and often plays on mc.hypixel.

Finnomator 1 Feb 12, 2022
Материалы для курса VK Углубленный Python, весна 2022

VK Углубленный Python, весна 2022 Материалы для курса VK Углубленный Python, весна 2022 Лекции и материалы (слайды, домашки, код с занятий) Введение,

10 Nov 02, 2022
Lightweight library for accessing data and configuration

accsr This lightweight library contains utilities for managing, loading, uploading, opening and generally wrangling data and configurations. It was ba

appliedAI Initiative 7 Mar 09, 2022
Reso is a low-level circuit design language and simulator, inspired by things like Redstone, Conway's Game of Life, and Wireworld.

Reso Reso is a low-level circuit design language and simulator, inspired by things like Redstone, Conway's Game of Life, and Wireworld. What is Reso?

Lynn 287 Nov 26, 2022
The code submitted for the Analytics Vidhya Jobathon - February 2022

Introduction On February 11th, 2022, Analytics Vidhya conducted a 3-day hackathon in data science. The top candidates had the chance to be selected by

11 Nov 21, 2022
a simple functional programming language compiler written in python

Functional Programming Language A compiler for my small functional language. Written in python with SLY lexer/parser generator library. Requirements p

Ashkan Laei 3 Nov 05, 2021
pgvector support for Python

pgvector-python pgvector support for Python Great for online recommendations 🎉 Supports Django, SQLAlchemy, Psycopg 2, Psycopg 3, and asyncpg Install

Andrew Kane 37 Dec 20, 2022