Avoid PyCharm Overloading
=========================
This speeds up PyCharm's package index processes and avoids CPU & memory
overloading.
On my machine (4c/32GB) with 5 conda environments and 10 projects, this reduces
PyCharm's package scanning from 15 minutes to 45 seconds.
Why?
----
When PyCharm has multiple conda environments, it will sometimes launch concurrent
package scanning across those environments. To do so, it uses the `conda` cli,
and a tool called conda_packaging_tool.py, available from the JetBrains
intellij-community github repository (see below).
Unfortunately, these tools do not behave well when launched in parallel, as
they tend to overload CPU & memory in this case.
How?
----
To avoid stalling these processes, we modify their sources as follows:
1. modify the conda cli
$ conda activate base
$ pip install runfast
$ nano `which conda`
Modify the main section:
if __name__ == '__main__':
from conda.cli import main
from runfast import cached
cached(main)
2. modify the conda_packaging_tool.py
if __name__ == '__main__':
from runfast import cached
cached(main)
How does this work?
-------------------
Two simple steps:
1. When called in parallel, only one process is allowed to proceed.
2. `runfast.cached` caches the output of these tools (stdout, stderr) for 1 minute,
given the same command line parameters
That is, when PyCharm launches 5 package scanning (`conda list`) commands, only
one of them will run immediately. If some of the scans are for the same environment,
only one of them will actually run, while the others simply return the same output.
How to clear the cache or avoid caching at all?
$ export RUNFAST_NOCACHE=1
Testing
-------
a) conda packaging tool
# first time
$ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
real 0m13.271s
user 0m9.631s
sys 0m1.936s
# second time
$ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
real 0m3.159s
user 0m0.139s
sys 0m1.298s
b) conda cli
# first time
$ time conda list -p /path/to/env
real 0m7.985s
user 0m7.883s
sys 0m0.087s
# second time
real 0m0.152s
user 0m0.116s
sys 0m0.032s
References
----------
* https://github.com/JetBrains/intellij-community/blob/master/python/helpers/conda_packaging_tool.py
* https://github.com/conda/conda/blob/33a142c16530fcdada6c377486f1c1a385738a96/conda/core/index.py#L53
This speeds up PyCharm's package index processes and avoids CPU & memory overloading
Overview
Python screenshot library, replacement for the Pillow ImageGrab module on Linux.
tldr: Use Pillow The pyscreenshot module is obsolete in most cases. It was created because PIL ImageGrab module worked on Windows only, but now Linux
This is a small compiler to demonstrate how compilers work.
This is a small compiler to demonstrate how compilers work. It compiles our own dialect to C, while being written in Python.
With the initiation of the COVID vaccination drive across India for all individuals above the age of 18, I wrote a python script which alerts the user regarding open slots in the vicinity!
cowin_notifier With the initiation of the COVID vaccination drive across India for all individuals above the age of 18, I wrote a python script which
A very basic ciphering/deciphering tool
ckrett-python-library This is an useful python library for people who care about privacy, this library is useful to cipher and decipher text using 4 s
Werkzeug has a debug console that requires a pin. It's possible to bypass this with an LFI vulnerability or use it as a local privilege escalation vector.
Werkzeug Debug Console Pin Bypass Werkzeug has a debug console that requires a pin by default. It's possible to bypass this with an LFI vulnerability
A frontend to ease the use of pulseaudio's routing capabilities, mimicking voicemeeter's workflow
Pulsemeeter A frontend to ease the use of pulseaudio's routing capabilities, mimicking voicemeeter's workflow Features Create virtual inputs and outpu
Pyjiting is a experimental Python-JIT compiler, which is the product of my undergraduate thesis
Pyjiting is a experimental Python-JIT compiler, which is the product of my undergraduate thesis. The goal is to implement a light-weight miniature general-purpose Python JIT compiler.
Biohacking con Python honeycon21
biohacking-honeycon21 This repository includes the slides of the public presentation 'Biohacking con Python' in the Hack&Beers of HoneyCON21 (PPTX and
For radiometrically calibrating and PSF deconvolving IRIS data
irispreppy For radiometrically calibrating and PSF deconvolving IRIS data. I dislike how I need to own proprietary software (IDL) just to simply prepa
A Regex based linter tool that works for any language and works exclusively with custom linting rules.
renag Documentation Available Here Short for Regex (re) Nag (like "one who complains"). Now also PEGs (Parsing Expression Grammars) compatible with py
Pequenos programas variados que estou praticando e implementando, leia o Read.me!
my-small-programs Pequenos programas variados que estou praticando e implementando! Arquivo: automacao Automacao de processos de rotina com código Pyt
A refresher for PowerBI Desktop documents
PowerBI_Refresher-NPP Informació Per executar el programa s'ha de tenir instalat el python versio 3 o mes. Requeriments a requirements.txt. El fitxer
Integer sets where all subsets have unique sums
Evil Sums Generation of sets of numbers where all constituents are recoverable from a partial sum.
A simple weather app.
keather A simple weather app. This is currently not finished. Dependencies: yay -S python-beautifulsoup4 tk
my own python useful functions
PythonToolKit Motivation This Repo should help save time for data scientists' daily work regarding the Time Series regression task by providing functi
Homed - Light-weight, easily configurable, dockerized homepage
homed GitHub Repo Docker Hub homed is a light-weight customizable portal primari
Mengzhan (John) code for Closed Loop Control system of Sharp Wave Ripples in Hippocampus CA3 region
ClosedLoopControl_Yu Mengzhan (John) code for Closed Loop Control system of Sharp Wave Ripples in Hippocampus CA3 region Creating Python Virtual Envir
Windows symbol tables for Volatility 3
Windows Symbol Tables for Volatility 3 This repository is the Windows Symbol Table storage for Volatility 3. How to Use $ git clone https://github.com
Python library for parsing Godot scene files
Godot Parser This is a python library for parsing Godot scene (.tscn) and resource (.tres) files. It's intended to make it easier to automate certain
A python package to manage the stored receiver-side Strain Green's Tensor (SGT) database of 3D background models and able to generate Green's function and synthetic waveform
A python package to manage the stored receiver-side Strain Green's Tensor (SGT) database of 3D background models and able to generate Green's function and synthetic waveform