Plux - A dynamic code loading framework for building plugable Python distributions

Related tags

Miscellaneousplux
Overview

Plux

CI badge PyPI Version PyPI License Code style: black

plux is the dynamic code loading framework used in LocalStack.

Overview

The plux builds a higher-level plugin mechanism around Python's entry point mechanism. It provides tools to load plugins from entry points at run time, and to discover entry points from plugins at build time (so you don't have to declare entry points statically in your setup.py).

Core concepts

  • PluginSpec: describes a Plugin. Each plugin has a namespace, a unique name in that namespace, and a PluginFactory (something that creates Plugin the spec is describing. In the simplest case, that can just be the Plugin's class).
  • Plugin: an object that exposes a should_load and load method. Note that it does not function as a domain object (it does not hold the plugins lifecycle state, like initialized, loaded, etc..., or other metadata of the Plugin)
  • PluginFinder: finds plugins, either at build time (by scanning the modules using pkgutil and setuptools) or at run time (reading entrypoints of the distribution using stevedore)
  • PluginManager: manages the run time lifecycle of a Plugin, which has three states:
    • resolved: the entrypoint pointing to the PluginSpec was imported and the PluginSpec instance was created
    • init: the PluginFactory of the PluginSpec was successfully invoked
    • loaded: the load method of the Plugin was successfully invoked

architecture

Loading Plugins

At run time, a PluginManager uses a PluginFinder that in turn uses stevedore to scan the available entrypoints for things that look like a PluginSpec. With PluginManager.load(name: str) or PluginManager.load_all(), plugins within the namespace that are discoverable in entrypoints can be loaded. If an error occurs at any state of the lifecycle, the PluginManager informs the PluginLifecycleListener about it, but continues operating.

Discovering entrypoints

At build time (e.g., with python setup.py develop/install/sdist), a special PluginFinder collects anything that can be interpreted as a PluginSpec, and creates from it setuptools entrypoints. In the setup.py we can use the plugin.setuptools.load_entry_points method to collect a dictionary for the entry_points value of setup().

from plugin.setuptools import load_entry_points

setup(
    entry_points=load_entry_points(exclude=("tests", "tests.*",))
)

Note that load_entry_points will try to resolve a cached version of entry_points.txt from the .egg-info directory, to avoid resolving the entry points when building the package from a source distribution.

Examples

To build something using the plugin framework, you will first want to introduce a Plugin that does something when it is loaded. And then, at runtime, you need a component that uses the PluginManager to get those plugins.

One class per plugin

This is the way we went with LocalstackCliPlugin. Every plugin class (e.g., ProCliPlugin) is essentially a singleton. This is easy, as the classes are discoverable as plugins. Simply create a Plugin class with a name and namespace and it will be discovered by the build time PluginFinder.

# abstract case (not discovered at build time, missing name)
class CliPlugin(Plugin):
    namespace = "my.plugins.cli"

    def load(self, cli):
        self.attach(cli)

    def attach(self, cli):
        raise NotImplementedError

# discovered at build time (has a namespace, name, and is a Plugin)
class MyCliPlugin(CliPlugin):
    name = "my"

    def attach(self, cli):
        # ... attach commands to cli object

now we need a PluginManager (which has a generic type) to load the plugins for us:

cli = # ... needs to come from somewhere

manager: PluginManager[CliPlugin] = PluginManager("my.plugins.cli", load_args=(cli,))

plugins: List[CliPlugin] = manager.load_all()

# todo: do stuff with the plugins, if you want/need
#  in this example, we simply use the plugin mechanism to run a one-shot function (attach) on a load argument

Re-usable plugins

When you have lots of plugins that are structured in a similar way, we may not want to create a separate Plugin class for each plugin. Instead we want to use the same Plugin class to do the same thing, but use several instances of it. The PluginFactory, and the fact that PluginSpec instances defined at module level are discoverable (inpired by pluggy), can be used to achieve that.

class ServicePlugin(Plugin):

    def __init__(self, service_name):
        self.service_name = service_name
        self.service = None

    def should_load(self):
        return self.service_name in config.SERVICES

    def load(self):
        module = importlib.import_module("localstack.services.%s" % self.service_name)
        # suppose we define a convention that each service module has a Service class, like moto's `Backend`
        self.service = module.Service()

def service_plugin_factory(name) -> PluginFactory:
    def create():
        return ServicePlugin(name)

    return create

# discoverable
s3 = PluginSpec("localstack.plugins.services", "s3", service_plugin_factory("s3"))

# discoverable
dynamodb = PluginSpec("localstack.plugins.services", "dynamodb", service_plugin_factory("dynamodb"))

# ... could be simplified with convenience framework code, but the principle will stay the same

Then we could use the PluginManager to build a Supervisor

class Supervisor:
    manager: PluginManager[ServicePlugin]

    def start(self, service_name):
        plugin = manager.load(service_name)
        service = plugin.service
        service.start()

Functions as plugins

with the @plugin decorator, you can expose functions as plugins. They will be wrapped by the framework into FunctionPlugin instances, which satisfy both the contract of a Plugin, and that of the function.

from plugin import plugin


@plugin(namespace="localstack.configurators")
def configure_logging(runtime):
    logging.basicConfig(level=runtime.config.loglevel)

    
@plugin(namespace="localstack.configurators")
def configure_somethingelse(runtime):
    # do other stuff with the runtime object
    pass

With a PluginManager via load_all, you receive the FunctionPlugin instances, that you can call like the functions

runtime = LocalstackRuntime()

for configurator in PluginManager("localstack.configurators").load_all():
    configurator(runtime)

Install

pip install plux

Develop

Create the virtual environment, install dependencies, and run tests

make venv
make test

Run the code formatter

make format

Upload the pypi package using twine

make upload
Owner
LocalStack
Enabling efficient local dev&test loops for Cloud applications
LocalStack
Simple Wayland HotKey Daemon

swhkd Simple Wayland HotKey Daemon This project is still very new and I'm making new decisions everyday as to where I should drive this project. I'm u

Aakash Sen Sharma 407 Dec 30, 2022
Get a list of all offline/online members in a discord server

Discord server insights Get a list of all offline/online members in a discord server. Uses Selenium to crawl invite links. Config Download Chrome driv

Prakhar Gurunani 3 Oct 21, 2022
Tool for running a high throughput data ingestion/transformation workload with MongoDB

Mongo Mangler The mongo-mangler tool is a lightweight Python utility, which you can run from a low-powered machine to execute a high throughput data i

Paul Done 9 Jan 02, 2023
A tutorial presents several practical examples of how to build DAGs in Apache Airflow

Apache Airflow - Python Brasil 2021 Este tutorial apresenta vários exemplos práticos de como construir DAGs no Apache Airflow. Background Apache Airfl

Jusbrasil 14 Jun 03, 2022
This repository contains code for building education startup.

Learning Management System Overview It's the code for EssayBrain, a tool for teacher that automatically grades and validates essays. In order to valid

Shyam Das Shrestha 1 Nov 21, 2021
BlackIP-Rep is a tool designed to gather the reputation and information of Bulk IP's.

BlackIP-Rep is a tool designed to gather the reputation and information of Bulk IP's. Focused on increasing the workflow of Security Operations(SOC) team during investigation.

0LiVEr 6 Dec 12, 2022
The code behind sqlfmt.com, a web UI for sqlfmt

The code behind sqlfmt.com, a web UI for sqlfmt

Ted Conbeer 2 Dec 14, 2022
Performance monitoring and testing of OpenStack

Browbeat Browbeat is a performance tuning and analysis tool for OpenStack. Browbeat is free, Open Source software. Analyze and tune your Cloud for opt

cloud-bulldozer 83 Dec 14, 2022
A prototype COG-based tile server for sparse Mars datasets

Mars tiler Mars Tiler is a prototype web application that serves tiles from cloud-optimized GeoTIFFs, with an emphasis on supporting planetary dataset

Daven Quinn 3 Mar 23, 2022
Minterpy - Multidimensional interpolation in Python.

minterpy is an open-source Python package for a multivariate generalization of the classical Newton and Lagrange interpolation schemes as well as related tasks.

Center for Advanced Systems Understanding 18 Jan 06, 2023
Transpiles some Python into human-readable Golang.

pytago Transpiles some Python into human-readable Golang. Try out the web demo Installation and usage There are two "officially" supported ways to use

Michael Phelps 318 Jan 03, 2023
NewsBlur is a personal news reader bringing people together to talk about the world.

NewsBlur NewsBlur is a personal news reader bringing people together to talk about the world.

Samuel Clay 6.2k Dec 29, 2022
Recreating my first CRUD in python, but now more professional

Recreating my first CRUD in python, but now more professional

Ricardo Deo Sipione Augusto 2 Nov 27, 2021
Simple Python Gemini browser with nice formatting

gg I wasn't satisfied with any of the other available Gemini clients, so I wrote my own. Requires Python 3.9 (maybe older, I haven't checked) and opti

Sarah Taube 2 Nov 21, 2021
Projeto de Jogo de dados em Python 3 onde é definido o lado a ser apostado e número de jogadas, pontuando os acertos e exibindo se ganhou ou perdeu.

Jogo de DadoX Projeto de script que simula um Jogo de dados em Python 3 onde é definido o lado a ser apostado (1, 2, 3, 4, 5 e 6) ou se vai ser um núm

Estênio Mariano 1 Jul 10, 2021
serological measurements from multiplexed ELISA assays

pysero pysero enables serological measurements with multiplexed and standard ELISA assays. The project automates estimation of antibody titers from da

Chan Zuckerberg Biohub 5 Aug 06, 2022
Blender Add-on to Add Metal Materials to Your Scene

Blender QMM (Quick Metal Materials) Blender Addon to Add Metal Materials to Your Scene Installation Download the latest ZIP from Releases. Usage This

Don Schnitzius 27 Dec 26, 2022
For when you really need to rank things

Comparisonator For when you really need to rank things. Do you know that feeling when there's this urge deep within you that tells you to compare thin

Maciej Wilczyński 1 Nov 01, 2021
Convert Photoshop curves (acv) to xmp presets for Lightroom

acv2xmp Convert Photoshop curves (acv) to Lightroom preset (xmp) acv2xmp.py Basic command prompt that relies on standard library only and can be used

5 Feb 06, 2022
Pomodoro timer by the Algodrip team!

PomoDrip 🍅 Pomodoro timer by the Algo Drip team! To-do: Create the script for the pomodoro timer Design the front-end of the program (Flask or Javasc

Algodrip 3 Sep 12, 2021