Macro recording and metaprogramming in Python

Overview

macro-kit

macro-kit is a package for efficient macro recording and metaprogramming in Python using abstract syntax tree (AST).

The design of AST in this package is strongly inspired by Julia metaprogramming. Similar methods are also implemented in builtin ast module but macro-kit is more focused on the macro generation and customization.

Installation

  • use pip
pip install macro-kit
  • from source
pip install git+https://github.com/hanjinliu/macro-kit

Examples

  1. Define a macro-recordable function
from macrokit import Macro, Expr, Symbol
macro = Macro()

@macro.record
def str_add(a, b):
    return str(a) + str(b)

val0 = str_add(1, 2)
val1 = str_add(val0, "xyz")
macro
[Out]
var0x24fdc2d1530 = str_add(1, 2)
var0x24fdc211df0 = str_add(var0x24fdc2d1530, 'xyz')
# substitute identifiers of variables
# var0x24fdc2d1530 -> x
macro.format([(val0, "x")]) 
[Out]
x = str_add(1, 2)
var0x24fdc211df0 = str_add(x, 'xyz')
# substitute to _dict["key"], or _dict.__getitem__("key")
expr = Expr(head="getitem", args=[Symbol("_dict"), "key"])
macro.format([(val0, expr)])
[Out]
_dict['key'] = str_add(1, 2)
var0x24fdc211df0 = str_add(_dict['key'], 'xyz')
  1. Record class
macro = Macro()

@macro.record
class C:
    def __init__(self, val: int):
        self.value = val
    
    @property
    def value(self):
        return self._value
    
    @value.setter
    def value(self, new_value: int):
        if not isinstance(new_value, int):
            raise TypeError("new_value must be an integer.")
        self._value = new_value
    
    def show(self):
        print(self._value)

c = C(1)
c.value = 5
c.value = -10
c.show()
[Out]
-10
macro.format([(c, "ins")])
[Out]
ins = C(1)
ins.value = -10     # setattr (and setitem) will not be recorded in duplicate
var0x7ffed09d2cd8 = ins.show()
macro.eval({"C": C})
[Out]
-10
  1. Record module
import numpy as np
macro = Macro()
np = macro.record(np) # macro-recordable numpy

arr = np.random.random(30)
mean = np.mean(arr)

macro
[Out]
var0x2a0a2864090 = numpy.random.random(30)
var0x2a0a40daef0 = numpy.mean(var0x2a0a2864090)
from dask import array as da
dask_macro = macro.format([(np, "da")])
dask_macro
[Out]
var0x2a0a2864090 = da.random.random(30)
var0x2a0a40daef0 = da.mean(var0x2a0a2864090)
output = {}
dask_macro.eval({"da": da}, output)
output
[Out]
{:da: 
   
    ,
 :var0x2a0a2864090: dask.array
    
     ,
 :var0x2a0a40daef0: dask.array
     
      }

     
    
   
You might also like...
Find dependent python scripts of a python script in a project directory.

Find dependent python scripts of a python script in a project directory.

SysInfo is an app developed in python which gives Basic System Info , and some detailed graphs of system performance .
SysInfo is an app developed in python which gives Basic System Info , and some detailed graphs of system performance .

SysInfo SysInfo is an app developed in python which gives Basic System Info , and some detailed graphs of system performance . Installation Download t

An OData v4 query parser and transpiler for Python

odata-query is a library that parses OData v4 filter strings, and can convert them to other forms such as Django Queries, SQLAlchemy Queries, or just plain SQL.

Python program to do with percentages and chances, random generation.

Chances and Percentages Python program to do with percentages and chances, random generation. What is this? This small program will generate a list wi

A Python library for reading, writing and visualizing the OMEGA Format
A Python library for reading, writing and visualizing the OMEGA Format

A Python library for reading, writing and visualizing the OMEGA Format, targeted towards storing reference and perception data in the automotive context on an object list basis with a focus on an urban use case.

A simple and easy to use Spam Bot made in Python!

This is a simple spam bot made in python. You can use to to spam anyone with anything on any platform.

RapidFuzz is a fast string matching library for Python and C++

RapidFuzz is a fast string matching library for Python and C++, which is using the string similarity calculations from FuzzyWuzzy

pydsinternals - A Python native library containing necessary classes, functions and structures to interact with Windows Active Directory.
pydsinternals - A Python native library containing necessary classes, functions and structures to interact with Windows Active Directory.

pydsinternals - Directory Services Internals Library A Python native library containing necessary classes, functions and structures to interact with W

A Python package implementing various colour checker detection algorithms and related utilities.
A Python package implementing various colour checker detection algorithms and related utilities.

A Python package implementing various colour checker detection algorithms and related utilities.

Releases(v0.3.8)
  • v0.3.8(Mar 16, 2022)

    Bug fixes

    • Length 1 tuple (1,) and length 0 set set() was not recorded correctly.
    • Import and __all__ did not match.
    • The __getitem__ was not correctly defined when a slice is used as indices.
    Source code(tar.gz)
    Source code(zip)
  • v0.3.7(Jan 22, 2022)

  • v0.3.6(Jan 13, 2022)

    Changes

    • Mock object for easier Expr construction. Instead of writing like Expr("getattr", [Expr("getattr", ...), b]), you can use m = Mock("m") and create Expr object by m.a(0, "s").
    • Full support of Python 3.7, and add tox test.

    Bug Fixes

    • The __eq__ method will return False instead of raise Exception to make macro recording safer.
    Source code(tar.gz)
    Source code(zip)
  • v0.3.4(Dec 12, 2021)

  • v0.3.3(Dec 10, 2021)

    New Features

    • New ast type supports: break, continue and raise.
    • Modules will be registered to Symbol whenever it is converted into a Symbol so that you'll not have to pass module object when you call eval. For instance, instead of macro.eval({"np", np}) you only have to call macro.eval().

    Changes

    • The "type" argument in Symbol is removed because it is never used.

    Bug Fixes

    • Hash of Symbol was not completely safe.
    • mModule did not return constant values correctly (such as np.pi).
    Source code(tar.gz)
    Source code(zip)
  • v0.3.2(Dec 8, 2021)

    New Features

    • register_type and Symbol.register_type can be used as a decorator.
    • Symbol.var("x") can create unique variable x.
    • Many missing AST types are supported, such as % (mod operation), += (in-place operations) and unary operations.
    • Validators are implemented in Expr to assert correct grammar for certain header (although they are not perfect). For instance, Expr("getattr", [Symbol("x"), "name"]) used to return :(x.'name') but now it returns :(x.name).
    • Many special methods, such as __int__ and __hash__, are correctly converted to int(X) in macro.
    • Macro can used as a field of a class. Unique Macro object will be created for different instances.

    Changes

    • Attribute _last_setval is moved to Macro.
    • Method type definition is changed to delayed definition.
    • Type mapping of symbol now uses dictionary as much as possible.
    • The type keyword argument in Symbol constructor is deleted because it was not used.

    Bug Fixes

    • Boolean operations (and and or) could not be parsed.
    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Dec 5, 2021)

    New Features

    • Support function definition, if, for, and many other expressions.
    • Macro.callbacks attribute for expression-added callbacks.
    • flags keyword argument to specify what kind of expressions will be recorded.

    Changes

    • Macro inherits Expr but head is fixed to Head.block to simplify module.
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Oct 24, 2021)

    New Features

    • Support string parsing using ast.
    • optimize method of Macro can remove unused returned variables. It makes macro looks better.

    Changes

    • Head is different from that in v0.1.0, thus may be inconsistent.
    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Oct 23, 2021)

Owner
Department of Biological Sciences, School of Science, The University of Tokyo. My major is biophysics, especially microtubule and motor proteins.
A library for interacting with Path of Exile game and economy data, and a unique loot filter generation framework.

wraeblast A library for interfacing with Path of Exile game and economy data, and a set of item filters geared towards trade league players. Filter Ge

David Gidwani 29 Aug 28, 2022
A tiny Python library for generating public IDs from integers

pids Create short public identifiers based on integer IDs. Installation pip install pids Usage from pids import pid public_id = pid.from_int(1234) #

Simon Willison 7 Nov 11, 2021
Give you a better view of your Docker registry disk usage.

registry-du Give you a better view of your Docker registry disk usage. This small tool will analysis your Docker registry(vanilla or Harbor both work)

Nova Kwok 16 Jan 07, 2023
Keval allows you to call arbitrary Windows kernel-mode functions from user mode, even (and primarily) on another machine.

Keval Keval allows you to call arbitrary Windows kernel-mode functions from user mode, even (and primarily) on another machine. The user mode portion

42 Dec 17, 2022
A meme error handler for python

Pwython OwO what's this? Pwython is project aiming to fill in one of the biggest problems with python, which is that it is slow lacks owoified text. N

SystematicError 23 Jan 15, 2022
Finds price floor for every single attribute in a given collection

Solana Solanart Scanner Enjoy the Free Code Steps to run Download VS Code

Dalton Nisbett 19 Oct 20, 2022
✨ Voici un code en Python par moi, et en français qui permet de générer du texte Lorem.

Lorem Gen ❗ Voici un code en Python par moi, et en français qui permet de générer du texte Lorem. Dépendences : pip install lorem_text 💖 Enjoy 🎫 Mon

MrGabin 3 Jun 07, 2021
Utility to extract Fantasy Grounds Unity Line-of-sight and lighting files from a Univeral VTT file exported from Dungeondraft

uvtt2fgu Utility to extract Fantasy Grounds Unity Line-of-sight and lighting files from a Univeral VTT file exported from Dungeondraft This program wo

Andre Kostur 29 Dec 05, 2022
Software to help automate collecting crowdsourced annotations using Mechanical Turk.

Video Crowdsourcing Software to help automate collecting crowdsourced annotations using Mechanical Turk. The goal of this project is to enable crowdso

Mike Peven 1 Oct 25, 2021
Numbers-parser - Python module for parsing Apple Numbers .numbers files

numbers-parser numbers-parser is a Python module for parsing Apple Numbers .numbers files. It supports Numbers files generated by Numbers version 10.3

Jon Connell 154 Jan 05, 2023
A hashtag from string extract python module

A hashtag from string extract python module

Fayas Noushad 3 Aug 10, 2022
This repository contains scripts that help you validate QR codes.

Validation tools This repository contains scripts that help you validate QR codes. It's hacky, and a warning for Apple Silicon users: the dependencies

Ryan Barrett 8 Mar 01, 2022
An extremely simple package with a single utillity class used for gracefully handling POSIX shutdown signals.

graceful-killer An extremely simple package with a single utillity class used for gracefully handling POSIX shutdown signals. Installation Use pip to

Sven Ćurković 1 Dec 09, 2021
The producer-consumer problem implemented with threads in Python

This was developed using a Python virtual environment, I would strongly recommend to do the same if you want to clone this repository. How to run this

Omar Beltran 1 Oct 30, 2021
A simple Python app that generates semi-random chord progressions.

chords-generator A simple Python app that generates semi-random chord progressions.

53 Sep 07, 2022
A work in progress box containing various Python utilities

python-wipbox A set of modern Python libraries under development to simplify the execution of reusable routines by different projects. Table of Conten

Deepnox 2 Jan 20, 2022
A small python tool to get relevant values from SRI invoices

SriInvoiceProcessing A small python tool to get relevant values from SRI invoices Some useful info to run the tool Login into your SRI account and ret

Wladymir Brborich 2 Jan 07, 2022
A python module to validate input.

A python module to validate input.

Matthias 6 Sep 13, 2022
Airspy-Utils is a small software collection to help with firmware related operations on Airspy HF+ devices.

Airspy-Utils Airspy-Utils is a small software collection to help with firmware related operations on Airspy HF+ devices on Linux (and other free syste

Dhiru Kholia 11 Oct 04, 2022
A library to easily convert climbing route grades between different grading systems.

pyclimb A library to easily convert climbing route grades between different grading systems. In rock climbing, mountaineering, and other climbing disc

Ilias Antonopoulos 4 Jan 26, 2022