Avoid PyCharm Overloading
=========================
This speeds up PyCharm's package index processes and avoids CPU & memory
overloading.
On my machine (4c/32GB) with 5 conda environments and 10 projects, this reduces
PyCharm's package scanning from 15 minutes to 45 seconds.
Why?
----
When PyCharm has multiple conda environments, it will sometimes launch concurrent
package scanning across those environments. To do so, it uses the `conda` cli,
and a tool called conda_packaging_tool.py, available from the JetBrains
intellij-community github repository (see below).
Unfortunately, these tools do not behave well when launched in parallel, as
they tend to overload CPU & memory in this case.
How?
----
To avoid stalling these processes, we modify their sources as follows:
1. modify the conda cli
$ conda activate base
$ pip install runfast
$ nano `which conda`
Modify the main section:
if __name__ == '__main__':
from conda.cli import main
from runfast import cached
cached(main)
2. modify the conda_packaging_tool.py
if __name__ == '__main__':
from runfast import cached
cached(main)
How does this work?
-------------------
Two simple steps:
1. When called in parallel, only one process is allowed to proceed.
2. `runfast.cached` caches the output of these tools (stdout, stderr) for 1 minute,
given the same command line parameters
That is, when PyCharm launches 5 package scanning (`conda list`) commands, only
one of them will run immediately. If some of the scans are for the same environment,
only one of them will actually run, while the others simply return the same output.
How to clear the cache or avoid caching at all?
$ export RUNFAST_NOCACHE=1
Testing
-------
a) conda packaging tool
# first time
$ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
real 0m13.271s
user 0m9.631s
sys 0m1.936s
# second time
$ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
real 0m3.159s
user 0m0.139s
sys 0m1.298s
b) conda cli
# first time
$ time conda list -p /path/to/env
real 0m7.985s
user 0m7.883s
sys 0m0.087s
# second time
real 0m0.152s
user 0m0.116s
sys 0m0.032s
References
----------
* https://github.com/JetBrains/intellij-community/blob/master/python/helpers/conda_packaging_tool.py
* https://github.com/conda/conda/blob/33a142c16530fcdada6c377486f1c1a385738a96/conda/core/index.py#L53
This speeds up PyCharm's package index processes and avoids CPU & memory overloading
Overview
Resizing using nnedi3/znedi3/nnedi3cl with center alignment and correct chroma placement
nnedi3_resample A VapourSynth script for easy resizing using nnedi3/znedi3/nnedi3cl with center alignment and correct chroma placement. Requirements n
A subleq VM/interpreter created by me for no reason
What is Dumbleq? Dumbleq is a dumb Subleq VM/interpreter implementation created by me for absolutely no reason at all. What is Subleq? If you haven't
Bible-App : Simple Tool To Show Bible Books
Bible App Simple Tool To Show Bible Books Socials: Language:
Create VSCode Extensions with python
About Create vscode extensions with python. Installation Stable version: pip install vscode-ext Why use this? Why should you use this for building VSc
Self sustained producer-consumer(prosumer) policy study using Python and Gurobi
Prosumer Policy This project aims to model the optimum dispatch behaviour of households with PV and battery systems under different policy instrument
Clackety Keyboards Powered by Python
KMK: Clackety Keyboards Powered by Python KMK is a feature-rich and beginner-friendly firmware for computer keyboards written and configured in Circui
Python project setup, updater, and launcher
Launcher Python project setup, updater, and launcher Purpose: Increase project productivity and provide features easily. Once installed as a git submo
Uproot - A script to bring deeply nested files or directories to the surface
UPROOT Bring deeply nested files or folders to the surface Uproot helps convert
This repository contains Python games that I've worked on. You'll learn how to create python games with AI. I try to focus on creating board games without GUI in Jupyter-notebook.
92_Python_Games 🎮 Introduction 👋 This repository contains Python games that I've worked on. You'll learn how to create python games with AI. I try t
Why write code when you can import it directly from GitHub Copilot?
Copilot Importer Why write code when you can import it directly from GitHub Copilot? What is Copilot Importer? The copilot python module will dynamica
🌍💉 Global COVID-19 vaccination data at the regional level.
COVID-19 vaccination data at subnational level. To ensure its officiality, the source data is carefully verified.
The Great Autoencoder Bake Off
The Great Autoencoder Bake Off The companion repository to a post on my blog. It contains all you need to reproduce the results. Features Currently fe
Multifunctional Analysis of Regions through Input-Output
MARIO Multifunctional Analysis of Regions through Input-Output. (Documents) What is it MARIO is a python package for handling input-output tables and
Simple Kahoot Botter.
Kahoot A simple Botter made in Python 3 for Kahoot.com. Also sorry for the shitty code lol. How to Run You need Python 3 installed on your device. Aft
An open-source hyper-heuristic framework for multi-objective optimization
MOEA-HH An open-source hyper-heuristic framework for multi-objective optimization. Introduction The multi-objective optimization technique is widely u
Roman numeral conversion with python
Roman numeral conversion Discipline: Programming Languages Student: Paulo Henrique Diniz de Lima Alencar. Language: Python Description Responsible for
API development made easy: a smart Python 3 API framework
appkernel - API development made easy What is Appkernel? A super-easy to use API framework, enabling API creation from zero to production within minut
A collection of some leetcode challenges in python and JavaScript
Python and Javascript Coding Challenges Some leetcode questions I'm currently working on to open up my mind to better ways of problem solving. Impleme
Personal Finance Forecaster - An AI tool for forecasting personal expenses
Personal Finance Forecaster - An AI tool for forecasting personal expenses
LPCV Winner Solution of Spring Team
LPCV Winner Solution of Spring Team