Plux - A dynamic code loading framework for building plugable Python distributions

Related tags

Miscellaneousplux
Overview

Plux

CI badge PyPI Version PyPI License Code style: black

plux is the dynamic code loading framework used in LocalStack.

Overview

The plux builds a higher-level plugin mechanism around Python's entry point mechanism. It provides tools to load plugins from entry points at run time, and to discover entry points from plugins at build time (so you don't have to declare entry points statically in your setup.py).

Core concepts

  • PluginSpec: describes a Plugin. Each plugin has a namespace, a unique name in that namespace, and a PluginFactory (something that creates Plugin the spec is describing. In the simplest case, that can just be the Plugin's class).
  • Plugin: an object that exposes a should_load and load method. Note that it does not function as a domain object (it does not hold the plugins lifecycle state, like initialized, loaded, etc..., or other metadata of the Plugin)
  • PluginFinder: finds plugins, either at build time (by scanning the modules using pkgutil and setuptools) or at run time (reading entrypoints of the distribution using stevedore)
  • PluginManager: manages the run time lifecycle of a Plugin, which has three states:
    • resolved: the entrypoint pointing to the PluginSpec was imported and the PluginSpec instance was created
    • init: the PluginFactory of the PluginSpec was successfully invoked
    • loaded: the load method of the Plugin was successfully invoked

architecture

Loading Plugins

At run time, a PluginManager uses a PluginFinder that in turn uses stevedore to scan the available entrypoints for things that look like a PluginSpec. With PluginManager.load(name: str) or PluginManager.load_all(), plugins within the namespace that are discoverable in entrypoints can be loaded. If an error occurs at any state of the lifecycle, the PluginManager informs the PluginLifecycleListener about it, but continues operating.

Discovering entrypoints

At build time (e.g., with python setup.py develop/install/sdist), a special PluginFinder collects anything that can be interpreted as a PluginSpec, and creates from it setuptools entrypoints. In the setup.py we can use the plugin.setuptools.load_entry_points method to collect a dictionary for the entry_points value of setup().

from plugin.setuptools import load_entry_points

setup(
    entry_points=load_entry_points(exclude=("tests", "tests.*",))
)

Note that load_entry_points will try to resolve a cached version of entry_points.txt from the .egg-info directory, to avoid resolving the entry points when building the package from a source distribution.

Examples

To build something using the plugin framework, you will first want to introduce a Plugin that does something when it is loaded. And then, at runtime, you need a component that uses the PluginManager to get those plugins.

One class per plugin

This is the way we went with LocalstackCliPlugin. Every plugin class (e.g., ProCliPlugin) is essentially a singleton. This is easy, as the classes are discoverable as plugins. Simply create a Plugin class with a name and namespace and it will be discovered by the build time PluginFinder.

# abstract case (not discovered at build time, missing name)
class CliPlugin(Plugin):
    namespace = "my.plugins.cli"

    def load(self, cli):
        self.attach(cli)

    def attach(self, cli):
        raise NotImplementedError

# discovered at build time (has a namespace, name, and is a Plugin)
class MyCliPlugin(CliPlugin):
    name = "my"

    def attach(self, cli):
        # ... attach commands to cli object

now we need a PluginManager (which has a generic type) to load the plugins for us:

cli = # ... needs to come from somewhere

manager: PluginManager[CliPlugin] = PluginManager("my.plugins.cli", load_args=(cli,))

plugins: List[CliPlugin] = manager.load_all()

# todo: do stuff with the plugins, if you want/need
#  in this example, we simply use the plugin mechanism to run a one-shot function (attach) on a load argument

Re-usable plugins

When you have lots of plugins that are structured in a similar way, we may not want to create a separate Plugin class for each plugin. Instead we want to use the same Plugin class to do the same thing, but use several instances of it. The PluginFactory, and the fact that PluginSpec instances defined at module level are discoverable (inpired by pluggy), can be used to achieve that.

class ServicePlugin(Plugin):

    def __init__(self, service_name):
        self.service_name = service_name
        self.service = None

    def should_load(self):
        return self.service_name in config.SERVICES

    def load(self):
        module = importlib.import_module("localstack.services.%s" % self.service_name)
        # suppose we define a convention that each service module has a Service class, like moto's `Backend`
        self.service = module.Service()

def service_plugin_factory(name) -> PluginFactory:
    def create():
        return ServicePlugin(name)

    return create

# discoverable
s3 = PluginSpec("localstack.plugins.services", "s3", service_plugin_factory("s3"))

# discoverable
dynamodb = PluginSpec("localstack.plugins.services", "dynamodb", service_plugin_factory("dynamodb"))

# ... could be simplified with convenience framework code, but the principle will stay the same

Then we could use the PluginManager to build a Supervisor

class Supervisor:
    manager: PluginManager[ServicePlugin]

    def start(self, service_name):
        plugin = manager.load(service_name)
        service = plugin.service
        service.start()

Functions as plugins

with the @plugin decorator, you can expose functions as plugins. They will be wrapped by the framework into FunctionPlugin instances, which satisfy both the contract of a Plugin, and that of the function.

from plugin import plugin


@plugin(namespace="localstack.configurators")
def configure_logging(runtime):
    logging.basicConfig(level=runtime.config.loglevel)

    
@plugin(namespace="localstack.configurators")
def configure_somethingelse(runtime):
    # do other stuff with the runtime object
    pass

With a PluginManager via load_all, you receive the FunctionPlugin instances, that you can call like the functions

runtime = LocalstackRuntime()

for configurator in PluginManager("localstack.configurators").load_all():
    configurator(runtime)

Install

pip install plux

Develop

Create the virtual environment, install dependencies, and run tests

make venv
make test

Run the code formatter

make format

Upload the pypi package using twine

make upload
Owner
LocalStack
Enabling efficient local dev&test loops for Cloud applications
LocalStack
Wordle Solver

Wordle Solver Installation Install the following onto your computer: Python 3.10.x Download Page Run pip install -r requirements.txt Instructions To r

John Bucknam 1 Feb 15, 2022
A 3-line lisp implementation

Nanolisp The download page of many a language harbors deep senses of forboding, of evil lurking in its native lair. You feel that the language is not

5 Jun 17, 2022
MIXLAB_NASA_TICKET mixlab 灵感来源于NASA的火星船票

MIXLAB_NASA_TICKET mixlab 灵感来源于NASA的火星船票,我们想要使用开源的代码来定制化这一设计。 其中photo_to_cartoon 是paddle的开源代码:https://github.com/minivision-ai/photo2cartoon-paddle 也借

tongji_cy 38 Feb 20, 2022
Stack BOF Protection Bypass Techniques

Stack Buffer Overflow - Protection Bypass Techniques

ommadawn46 18 Dec 28, 2022
Run python scripts and pass data between multiple python and node processes using this npm module

Run python scripts and pass data between multiple python and node processes using this npm module. process-communication has a event based architecture for interacting with python data and errors ins

Tyler Laceby 2 Aug 06, 2021
30DaysOfCode-PhoenixClub - Solution of everyday coding problem given in 30DaysofCode contest held on Hackerrank

30DaysOfCode-PhoenixClub 👨‍💻 Every day problems solution given in 30DaysOfCode

Urveshkumar 8 Jan 30, 2022
Randomly distribute members by groups making sure that every sector is represented

Generate Groups Randomly distribute members by groups making sure that every sector is represented The Scenario Imagine that you have a large group of

Jorge Gomes 1 Oct 22, 2021
HairCLIP: Design Your Hair by Text and Reference Image

Overview This repository hosts the official PyTorch implementation of the paper: "HairCLIP: Design Your Hair by Text and Reference Image". Our single

322 Dec 30, 2022
An OBS script to fuze files together

OBS TEXT FUZE Fuze text files and inject the output into a text source. The Index file directory should be a list of file directorys for the text file

SuperZooper3 1 Dec 27, 2021
A simple tool made in Python language

Simple tool Uma simples ferramenta feita 100% em linguagem Python 💻 Requisitos: Python3 instalado em seu dispositivo Clonagem e acesso 📳 git clone h

josh washington 4 Dec 07, 2021
Your copilot to studies and work (Pomodoro-timer, Translate and Notes app)

Copylot Your copilot to studies and work (Pomodoro-timer, Translate and Notes app) Copylot are three applications in one: Pomodoro Translate Notes Cop

Eduardo Mendes 20 Dec 16, 2022
A basic DIY-project made using Python and MySQL

Banking-Using-Python-MySQL This is a basic DIY-project made using Python and MySQL. Pre-Requisite needed:-- MySQL command Line:- creating a database

ABHISHEK 0 Jul 03, 2022
MuMMI Core is the underlying infrastructure and generalizable component of the MuMMI framework

MuMMI Core is the underlying infrastructure and generalizable component of the MuMMI framework, which facilitates the coordination of massively parallel multiscale simulations.

4 Aug 17, 2022
Subcert is an subdomain enumeration tool, that finds all the subdomains from certificate transparency logs.

Subcert Subcert is a subdomain enumeration tool, that finds all the valid subdomains from certificate transparency logs. Table of contents Setup Demo

A3h1nt 59 Dec 16, 2022
Superset custom path for python

It is a common requirement to have superset running under a base url, (https://mydomain.at/analytics/ instead of https://mydomain.at/). I created the

9 Dec 14, 2022
Webcash is an experimental e-cash (electronic cash)

Webcash Webcash is an experimental new electronic cash ("e-cash") that enables decentralized and instant payments to anyone, anywhere in the world. Us

Mark Friedenbach 0 Feb 26, 2022
Curses frontend for Canto daemon

Canto Curses The curses (text) client for canto-daemon. Canto-daemon is required to work and is found at: http://github.com/themoken/canto-next Requir

Jack Miller 86 Dec 28, 2022
An electron application to check battery of bluetooth devices connected to linux devices.

bluetooth-battery-electron An electron application to check battery of bluetooth devices connected to linux devices. This project provides an electron

Vasu Sharma 15 Dec 03, 2022
Transform your boring distro into a hacking powerhouse.

Pentizer Transform your boring distro into a hacking powerhouse. Pentizer is a personal project that imports Kali and Parrot repositories in any Debia

Michail Tsimpliarakis 2 Nov 05, 2021
GUI tool to manage the contents of chests in Botw

Botw chest manager is a small gui tool allowing to easily manage chests. Sometimes Ice Spear can be very time consuming when adding a simple chest. The purpose of this light tool is to add a new ches

3 Aug 25, 2022