Avoid PyCharm Overloading
=========================
This speeds up PyCharm's package index processes and avoids CPU & memory
overloading.
On my machine (4c/32GB) with 5 conda environments and 10 projects, this reduces
PyCharm's package scanning from 15 minutes to 45 seconds.
Why?
----
When PyCharm has multiple conda environments, it will sometimes launch concurrent
package scanning across those environments. To do so, it uses the `conda` cli,
and a tool called conda_packaging_tool.py, available from the JetBrains
intellij-community github repository (see below).
Unfortunately, these tools do not behave well when launched in parallel, as
they tend to overload CPU & memory in this case.
How?
----
To avoid stalling these processes, we modify their sources as follows:
1. modify the conda cli
$ conda activate base
$ pip install runfast
$ nano `which conda`
Modify the main section:
if __name__ == '__main__':
from conda.cli import main
from runfast import cached
cached(main)
2. modify the conda_packaging_tool.py
if __name__ == '__main__':
from runfast import cached
cached(main)
How does this work?
-------------------
Two simple steps:
1. When called in parallel, only one process is allowed to proceed.
2. `runfast.cached` caches the output of these tools (stdout, stderr) for 1 minute,
given the same command line parameters
That is, when PyCharm launches 5 package scanning (`conda list`) commands, only
one of them will run immediately. If some of the scans are for the same environment,
only one of them will actually run, while the others simply return the same output.
How to clear the cache or avoid caching at all?
$ export RUNFAST_NOCACHE=1
Testing
-------
a) conda packaging tool
# first time
$ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
real 0m13.271s
user 0m9.631s
sys 0m1.936s
# second time
$ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
real 0m3.159s
user 0m0.139s
sys 0m1.298s
b) conda cli
# first time
$ time conda list -p /path/to/env
real 0m7.985s
user 0m7.883s
sys 0m0.087s
# second time
real 0m0.152s
user 0m0.116s
sys 0m0.032s
References
----------
* https://github.com/JetBrains/intellij-community/blob/master/python/helpers/conda_packaging_tool.py
* https://github.com/conda/conda/blob/33a142c16530fcdada6c377486f1c1a385738a96/conda/core/index.py#L53
This speeds up PyCharm's package index processes and avoids CPU & memory overloading
Overview
The functions we created are included in a script. The necessary parts for pre-processing were taken. Analysis complete.
Feature-Engineering The functions we created are included in a script. The necessary parts for pre-processing were taken. Analysis complete. Business
Code for the manim-generated scenes used in 3blue1brown videos
This project contains the code used to generate the explanatory math videos found on 3Blue1Brown. This almost entirely consists of scenes generated us
Find the remote website version based on a git repository
versionshaker Versionshaker is a tool to find a remote website version based on a git repository This tool will help you to find the website version o
Automated GitHub profile content using the USGS API, Plotly and GitHub Actions.
Top 20 Largest Earthquakes in the Past 24 Hours Location Mag Date and Time (UTC) 92 km SW of Sechura, Peru 5.2 11-05-2021 23:19:50 113 km NNE of Lobuj
UFDR2DIR - A script to convert a Cellebrite UFDR to the original file structure
UFDR2DIR A script to convert a Cellebrite UFDR to it's original file and directo
This is a project for socks card label validation where the socks card is validated comparing with the correct socks card whose coordinates are stored in the database. When the test socks card is compared with the correct socks card(master socks card) the software checks whether both test and master socks card mathches or not.
Automation_in_socks_label_validation THEME: MACHINE LEARNING This is a project for socks card label validation where the socks card is validated compa
March-madness - March Madness results 1985-2021
march-madness Results for all 2,268 NCAA Division I Men's Basketball Tournament games since the modern format was introduced in 1985. Includes years,
Wagtail + Lottie is a Wagtail package for playing Adobe After Effects animations exported as json with Bodymovin.
Wagtail Lottie Wagtail + Lottie is a Wagtail package for playing Adobe After Effects animations exported as json with Bodymovin. Usage Export your ani
Project in which we modelise an Among Us problem using graph theories.
Python-AmongUsProblem Project in which we modelise an Among Us problem using graph theories. The rules are as following: Total of 100 players 10 playe
Creates infinite amount of guilded accounts in seconds.
Guilded Cookie Creator [fuck guilded i quit working on this, they patch like every fucking method after 2/3 days i release shit] Optimizations Asynchr
The ROS publisher/subscriber example packaged as a snap
publisher-subscriber The ROS publisher/subscriber example packaged as a snap, based on ROS Noetic and Ubuntu Core 20. Strictly confined. This example
Data Orchestration Platform
Table of contents What is DOP Design Concept A Typical DOP Orchestration Flow Prerequisites - Run in Docker For DOP Native Features For DBT Instructio
Programmatic interface to Synapse services for Python
A Python client for Sage Bionetworks' Synapse, a collaborative, open-source research platform that allows teams to share data, track analyses, and collaborate
Calculatrix is a project where I'll create plenty of calculators in a lot of differents languages
Calculatrix What is Calculatrix ? Calculatrix is a project where I'll create plenty of calculators in a lot of differents languages. I know this sound
🌍💉 Global COVID-19 vaccination data at the regional level.
COVID-19 vaccination data at subnational level. To ensure its officiality, the source data is carefully verified.
Tie together `drf-spectacular` and `djangorestframework-dataclasses` for easy-to-use apis and openapi schemas.
Speccify Tie together drf-spectacular and djangorestframework-dataclasses for easy-to-use apis and openapi schemas. Usage @dataclass class MyQ
This is a small Panel applet for the Budgie Desktop to display the battery charge of a connected Bluetooth device.
BudgieBluetoothBattery This is a small Panel applet for the Budgie Desktop to display the battery charge of a connected Bluetooth device. It uses the
Extend the maya channel box with searchability and colour
channel-box-plus will add search-ability over its attributes, and it will colour user defined attributes, making them easier to distinguish.
Learn Python Regular Expressions step by step from beginner to advanced levels
Python re(gex)? Learn Python Regular Expressions step by step from beginner to advanced levels with hundreds of examples and exercises The book also i
Object-oriented programming (OOP) is a method of structuring a program by bundling related properties and behaviors into individual objects. In this tutorial, you’ll learn the basics of object-oriented programming in Python.
06_Python_Object_Class Introduction 👋 Objected oriented programming as a discipline has gained a universal following among developers. Python, an in-