Plux - A dynamic code loading framework for building plugable Python distributions

Related tags

Miscellaneousplux
Overview

Plux

CI badge PyPI Version PyPI License Code style: black

plux is the dynamic code loading framework used in LocalStack.

Overview

The plux builds a higher-level plugin mechanism around Python's entry point mechanism. It provides tools to load plugins from entry points at run time, and to discover entry points from plugins at build time (so you don't have to declare entry points statically in your setup.py).

Core concepts

  • PluginSpec: describes a Plugin. Each plugin has a namespace, a unique name in that namespace, and a PluginFactory (something that creates Plugin the spec is describing. In the simplest case, that can just be the Plugin's class).
  • Plugin: an object that exposes a should_load and load method. Note that it does not function as a domain object (it does not hold the plugins lifecycle state, like initialized, loaded, etc..., or other metadata of the Plugin)
  • PluginFinder: finds plugins, either at build time (by scanning the modules using pkgutil and setuptools) or at run time (reading entrypoints of the distribution using stevedore)
  • PluginManager: manages the run time lifecycle of a Plugin, which has three states:
    • resolved: the entrypoint pointing to the PluginSpec was imported and the PluginSpec instance was created
    • init: the PluginFactory of the PluginSpec was successfully invoked
    • loaded: the load method of the Plugin was successfully invoked

architecture

Loading Plugins

At run time, a PluginManager uses a PluginFinder that in turn uses stevedore to scan the available entrypoints for things that look like a PluginSpec. With PluginManager.load(name: str) or PluginManager.load_all(), plugins within the namespace that are discoverable in entrypoints can be loaded. If an error occurs at any state of the lifecycle, the PluginManager informs the PluginLifecycleListener about it, but continues operating.

Discovering entrypoints

At build time (e.g., with python setup.py develop/install/sdist), a special PluginFinder collects anything that can be interpreted as a PluginSpec, and creates from it setuptools entrypoints. In the setup.py we can use the plugin.setuptools.load_entry_points method to collect a dictionary for the entry_points value of setup().

from plugin.setuptools import load_entry_points

setup(
    entry_points=load_entry_points(exclude=("tests", "tests.*",))
)

Note that load_entry_points will try to resolve a cached version of entry_points.txt from the .egg-info directory, to avoid resolving the entry points when building the package from a source distribution.

Examples

To build something using the plugin framework, you will first want to introduce a Plugin that does something when it is loaded. And then, at runtime, you need a component that uses the PluginManager to get those plugins.

One class per plugin

This is the way we went with LocalstackCliPlugin. Every plugin class (e.g., ProCliPlugin) is essentially a singleton. This is easy, as the classes are discoverable as plugins. Simply create a Plugin class with a name and namespace and it will be discovered by the build time PluginFinder.

# abstract case (not discovered at build time, missing name)
class CliPlugin(Plugin):
    namespace = "my.plugins.cli"

    def load(self, cli):
        self.attach(cli)

    def attach(self, cli):
        raise NotImplementedError

# discovered at build time (has a namespace, name, and is a Plugin)
class MyCliPlugin(CliPlugin):
    name = "my"

    def attach(self, cli):
        # ... attach commands to cli object

now we need a PluginManager (which has a generic type) to load the plugins for us:

cli = # ... needs to come from somewhere

manager: PluginManager[CliPlugin] = PluginManager("my.plugins.cli", load_args=(cli,))

plugins: List[CliPlugin] = manager.load_all()

# todo: do stuff with the plugins, if you want/need
#  in this example, we simply use the plugin mechanism to run a one-shot function (attach) on a load argument

Re-usable plugins

When you have lots of plugins that are structured in a similar way, we may not want to create a separate Plugin class for each plugin. Instead we want to use the same Plugin class to do the same thing, but use several instances of it. The PluginFactory, and the fact that PluginSpec instances defined at module level are discoverable (inpired by pluggy), can be used to achieve that.

class ServicePlugin(Plugin):

    def __init__(self, service_name):
        self.service_name = service_name
        self.service = None

    def should_load(self):
        return self.service_name in config.SERVICES

    def load(self):
        module = importlib.import_module("localstack.services.%s" % self.service_name)
        # suppose we define a convention that each service module has a Service class, like moto's `Backend`
        self.service = module.Service()

def service_plugin_factory(name) -> PluginFactory:
    def create():
        return ServicePlugin(name)

    return create

# discoverable
s3 = PluginSpec("localstack.plugins.services", "s3", service_plugin_factory("s3"))

# discoverable
dynamodb = PluginSpec("localstack.plugins.services", "dynamodb", service_plugin_factory("dynamodb"))

# ... could be simplified with convenience framework code, but the principle will stay the same

Then we could use the PluginManager to build a Supervisor

class Supervisor:
    manager: PluginManager[ServicePlugin]

    def start(self, service_name):
        plugin = manager.load(service_name)
        service = plugin.service
        service.start()

Functions as plugins

with the @plugin decorator, you can expose functions as plugins. They will be wrapped by the framework into FunctionPlugin instances, which satisfy both the contract of a Plugin, and that of the function.

from plugin import plugin


@plugin(namespace="localstack.configurators")
def configure_logging(runtime):
    logging.basicConfig(level=runtime.config.loglevel)

    
@plugin(namespace="localstack.configurators")
def configure_somethingelse(runtime):
    # do other stuff with the runtime object
    pass

With a PluginManager via load_all, you receive the FunctionPlugin instances, that you can call like the functions

runtime = LocalstackRuntime()

for configurator in PluginManager("localstack.configurators").load_all():
    configurator(runtime)

Install

pip install plux

Develop

Create the virtual environment, install dependencies, and run tests

make venv
make test

Run the code formatter

make format

Upload the pypi package using twine

make upload
Owner
LocalStack
Enabling efficient local dev&test loops for Cloud applications
LocalStack
FantasyBballHelper - Espn Fantasy Basketball Helper

ESPN FANTASY BASKETBALL HELPER The simple goal of this project is to allow fanta

1 Jan 18, 2022
Different steganography methods with examples and my own small image database

literally-the-most-useless-project [Different steganography methods with examples and my own small image database] This project currently contains thr

Kamyishka 1 Dec 09, 2022
HomeAssistant Linux Companion

Application to run on linux desktop computer to provide sensors data to homeasssistant, and get notifications as if it was a mobile device.

Javier Lopez 10 Dec 27, 2022
creates a batch file that uses adb to auto-install apks into the Windows Subsystem for Android and registers it as the default application to open apks.

wsa-apktool creates a batch file that uses adb to auto-install apks into the Windows Subsystem for Android and registers it as the default application

Aditya Vikram 3 Apr 05, 2022
A simple armature retargeting tool for Blender

Simple-Retarget-Tool-Blender A simple armature retargeting tool for Blender Update V2: Set Rest Pose to easily apply rest pose. Preset Import/Export.

Fahad Hasan Pathik 74 Jan 04, 2023
Nesse repositório serão armazenados os conteúdos de aula

Lets_Code_DS_Degree_Alunos Nesse repositório serão armazenados os conteúdos de aula Formato das aulas: Notebook de aula já vem comentado para reduzir

Patricia Bongiovanni Catandi 6 Jan 21, 2022
GA SEI Unit 4 project backend for Bloom.

Grow Your OpportunitiesTM Background Watch the Bloom Intro Video At Bloom, we believe every job seeker deserves an opportunity to find meaningful work

Jonathan Herman 3 Sep 20, 2021
Holographic Declarative Memory for Python ACT-R

HDM This is the repository for the Holographic Declarative Memory (HDM) module for Python ACT-R. This repository contains: documentation: a paper, con

Carleton Cognitive Modeling Lab 1 Jan 17, 2022
python for windows extensions

This is the readme for the Python for Win32 (pywin32) extensions source code. See CHANGES.txt for recent changes. 'setup.py' is a standard distutils

27 Dec 08, 2022
A basic tool to generate Hydrogen drum machine kits.

Generate Hydrogen Kit A basic tool to generate drumkit.xml files for Hydrogen drum machine. Saves a bit of time when making kits. Supply it with a nam

Luna Langton 2 Nov 28, 2021
Standalone PyQGIS application for executing custom scripts without a QGIS GUI.

PyQGIS Standalone Script Executer Standalone PyQGIS application that is able to run a custom script, in this case Proximity.py without the need of a G

6 Sep 23, 2022
Repository to store sample python programs for python learning

py Repository to store sample Python programs. This repository is meant for beginners to assist them in their learning of Python. The repository cover

codebasics 5.8k Dec 30, 2022
Fork of pathlib aiming to support the full stdlib Python API.

pathlib2 Fork of pathlib aiming to support the full stdlib Python API. The old pathlib module on bitbucket is in bugfix-only mode. The goal of pathlib

Jazzband 73 Dec 23, 2022
Something like Asteroids but not really, done in CircuitPython

CircuitPython Staroids Something like Asteroids, done in CircuitPython. Works with FunHouse, MacroPad, Pybadge, EdgeBadge, CLUE, and Pygamer. circuitp

Tod E. Kurt 14 May 31, 2022
This tool for beginner and help those people they gather information about Email Header Analysis, Instagram Information, Instagram Username Check, Ip Information, Phone Number Information, Port Scan

This tool for beginner and help those people they gather information about Email Header Analysis, Instagram Information, Instagram Username Check, Ip Information, Phone Number Information, Port Scan.

cb-kali 5 Feb 18, 2022
Make pack up python files easier.

python-easy-pack make pack up python files easier. 目前只提供了中文环境 如何使用? 将index.py复制到你的项目文件夹,或者把.py文件拷贝到这个文件夹。 打开你的cmd或者powershell 切换到程序所在目录,输入python index

2 Dec 15, 2021
Python pyside2 kütüphanesi ile oluşturduğum drone için yer kontrol istasyonu yazılımı.

Ground Control Station (Yer Kontrol İstasyonu) Teknofest yarışmasında yerlilik kısmında Yer Kontrol İstasyonu yazılımı seçeneği bulunuyordu. Bu yüzden

Emirhan Bülbül 4 May 14, 2022
Python Monopoly Simulator

Monopoly simulator Original creator: Games Computer Play YouTube: https://www.youtube.com/channel/UCTrp88f-QJ1SqKX8o5IDhWQ Config file (optional) conf

Games Computers Play 37 Jan 03, 2023
Simple python bot, that notifies about new manga chapters through Telegram.

Simple python bot, that notifies about new manga chapters through Telegram.

Dmitry Kopturov 1 Dec 05, 2021
200 LeetCode problems

LeetCode I classify 200 leetcode problems into some categories and upload my code to who concern WEEK 1 # Title Difficulty Array 15 3Sum Medium 1324 P

Hoang Cao Bao 108 Dec 08, 2022