A toolkit for developing and deploying serverless Python code in AWS Lambda.

Overview

python-lambda logo

pypi pypi

Python-lambda is a toolset for developing and deploying serverless Python code in AWS Lambda.

A call for contributors

With python-lambda and pytube both continuing to gain momentum, I'm calling for contributors to help build out new features, review pull requests, fix bugs, and maintain overall code quality. If you're interested, please email me at nficano[at]gmail.com.

Description

AWS Lambda is a service that allows you to write Python, Java, or Node.js code that gets executed in response to events like http requests or files uploaded to S3.

Working with Lambda is relatively easy, but the process of bundling and deploying your code is not as simple as it could be.

The Python-Lambda library takes away the guess work of developing your Python-Lambda services by providing you a toolset to streamline the annoying parts.

Requirements

  • Python 2.7, >= 3.6 (At the time of writing this, these are the Python runtimes supported by AWS Lambda).
  • Pip (~8.1.1)
  • Virtualenv (~15.0.0)
  • Virtualenvwrapper (~4.7.1)

Getting Started

First, you must create an IAM Role on your AWS account called lambda_basic_execution with the LambdaBasicExecution policy attached.

On your computer, create a new virtualenv and project folder.

$ mkvirtualenv pylambda
(pylambda) $ mkdir pylambda

Next, download Python-Lambda using pip via pypi.

(pylambda) $ pip install python-lambda

From your pylambda directory, run the following to bootstrap your project.

(pylambda) $ lambda init

This will create the following files: event.json, __init__.py, service.py, and config.yaml.

Let's begin by opening config.yaml in the text editor of your choice. For the purpose of this tutorial, the only required information is aws_access_key_id and aws_secret_access_key. You can find these by logging into the AWS management console.

Next let's open service.py, in here you'll find the following function:

def handler(event, context):
    # Your code goes here!
    e = event.get('e')
    pi = event.get('pi')
    return e + pi

This is the handler function; this is the function AWS Lambda will invoke in response to an event. You will notice that in the sample code e and pi are values in a dict. AWS Lambda uses the event parameter to pass in event data to the handler.

So if, for example, your function is responding to an http request, event will be the POST JSON data and if your function returns something, the contents will be in your http response payload.

Next let's open the event.json file:

{
  "pi": 3.14,
  "e": 2.718
}

Here you'll find the values of e and pi that are being referenced in the sample code.

If you now try and run:

(pylambda) $ lambda invoke -v

You will get:

# 5.858
# execution time: 0.00000310s
# function execution timeout: 15s

As you probably put together, the lambda invoke command grabs the values stored in the event.json file and passes them to your function.

The event.json file should help you develop your Lambda service locally. You can specify an alternate event.json file by passing the --event-file=<filename>.json argument to lambda invoke.

When you're ready to deploy your code to Lambda simply run:

(pylambda) $ lambda deploy

The deploy script will evaluate your virtualenv and identify your project dependencies. It will package these up along with your handler function to a zip file that it then uploads to AWS Lambda.

You can now log into the AWS Lambda management console to verify the code deployed successfully.

Wiring to an API endpoint

If you're looking to develop a simple microservice you can easily wire your function up to an http endpoint.

Begin by navigating to your AWS Lambda management console and clicking on your function. Click the API Endpoints tab and click "Add API endpoint".

Under API endpoint type select "API Gateway".

Next change Method to POST and Security to "Open" and click submit (NOTE: you should secure this for use in production, open security is used for demo purposes).

At last you need to change the return value of the function to comply with the standard defined for the API Gateway endpoint, the function should now look like this:

def handler(event, context):
    # Your code goes here!
    e = event.get('e')
    pi = event.get('pi')
    return {
        "statusCode": 200,
        "headers": { "Content-Type": "application/json"},
        "body": e + pi
    }

Now try and run:

$ curl --header "Content-Type:application/json" \
       --request POST \
       --data '{"pi": 3.14, "e": 2.718}' \
       https://<API endpoint URL>
# 5.8580000000000005

Environment Variables

Lambda functions support environment variables. In order to set environment variables for your deployed code to use, you can configure them in config.yaml. To load the value for the environment variable at the time of deployment (instead of hard coding them in your configuration file), you can use local environment values (see 'env3' in example code below).

environment_variables:
  env1: foo
  env2: baz
  env3: ${LOCAL_ENVIRONMENT_VARIABLE_NAME}

This would create environment variables in the lambda instance upon deploy. If your functions don't need environment variables, simply leave this section out of your config.

Uploading to S3

You may find that you do not need the toolkit to fully deploy your Lambda or that your code bundle is too large to upload via the API. You can use the upload command to send the bundle to an S3 bucket of your choosing. Before doing this, you will need to set the following variables in config.yaml:

role: basic_s3_upload
bucket_name: 'example-bucket'
s3_key_prefix: 'path/to/file/'

Your role must have s3:PutObject permission on the bucket/key that you specify for the upload to work properly. Once you have that set, you can execute lambda upload to initiate the transfer.

Deploying via S3

You can also choose to use S3 as your source for Lambda deployments. This can be done by issuing lambda deploy-s3 with the same variables/AWS permissions you'd set for executing the upload command.

Development

Development of "python-lambda" is facilitated exclusively on GitHub. Contributions in the form of patches, tests and feature creation and/or requests are very welcome and highly encouraged. Please open an issue if this tool does not function as you'd expect.

Environment Setup

  1. Install pipenv
  2. Install direnv
  3. Install Precommit (optional but preferred)
  4. cd into the project and enter "direnv allow" when prompted. This will begin installing all the development dependancies.
  5. If you installed pre-commit, run pre-commit install inside the project directory to setup the githooks.

Releasing to Pypi

Once you pushed your chances to master, run one of the following:

# If you're installing a major release:
make deploy-major

# If you're installing a minor release:
make deploy-minor

# If you're installing a patch release:
make deploy-patch
Owner
Nick Ficano
Hi, I'm Nick! I develop software and live in Smithtown, New York.
Nick Ficano
Navigate to your directory of choice the proceed as follows

Installation πŸš€ Navigate to your directory of choice the proceed as follows; 1 .Clone the git repo and create a virtual environment Depending on your

Ondiek Elijah Ochieng 2 Jan 31, 2022
Tensorboard plugin 3d with python

tensorboard-plugin-3d Overview In this example, we render a run selector dropdown component. When the user selects a run, it shows a preview of all sc

KitwareMedical 26 Nov 14, 2022
Fastest python library for making asynchronous group requests.

FGrequests: Fastest Asynchronous Group Requests Installation Install using pip: pip install fgrequests Documentation Pretty easy to use. import fgrequ

Farid Chowdhury 14 Nov 22, 2022
A Regex based linter tool that works for any language and works exclusively with custom linting rules.

renag Documentation Available Here Short for Regex (re) Nag (like "one who complains"). Now also PEGs (Parsing Expression Grammars) compatible with py

Ryan Peach 12 Oct 20, 2022
A Python library to simulate a Zoom H6 recorder remote control

H6 A Python library to emulate a Zoom H6 recorder remote control Introduction This library allows you to control your Zoom H6 recorder from your compu

Matias Godoy 68 Nov 02, 2022
Python with braces. Because Python is awesome, but whitespace is awful.

Bython Python with braces. Because Python is awesome, but whitespace is awful. Bython is a Python preprosessor which translates curly brackets into in

1 Nov 04, 2021
A repository for all ZenML projects that are specific production use-cases.

ZenFiles Original Image source: https://www.goodfon.com/wallpaper/x-files-sekretnye-materialy.html And naturally, all credits to the awesome X-Files s

ZenML 66 Jan 06, 2023
Procedural 3D data generation pipeline for architecture

Synthetic Dataset Generator Authors: Stanislava Fedorova Alberto Tono Meher Shashwat Nigam Jiayao Zhang Amirhossein Ahmadnia Cecilia bolognesi Dominik

Computational Design Institute 49 Nov 25, 2022
ASVspoof 2021 Baseline Systems

ASVspoof 2021 Baseline Systems Baseline systems are grouped by task: Speech Deepfake (DF) Logical Access (LA) Physical Access (PA) Please find more de

91 Dec 28, 2022
Pre-crisis Risk Management for Personal Finance

Антикризисный риск-ΠΌΠ΅Π½Π΅Π΄ΠΆΠΌΠ΅Π½Ρ‚ Π»ΠΈΡ‡Π½Ρ‹Ρ… финансов Риск-ΠΌΠ΅Π½Π΅Π΄ΠΆΠΌΠ΅Π½Ρ‚ Π»ΠΈΡ‡Π½Ρ‹Ρ… финансов условиях санкций ΠΈ/ΠΈΠ»ΠΈ финансового кризиса: Π΄Π΅Π»Π°Π΅ΠΌ сСгодня всС, Ρ‡Ρ‚ΠΎΠ±Ρ‹ Π·Π°

Dmitry Petukhov 593 Jan 09, 2023
NeurIPS'19: Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting (Pytorch implementation for noisy labels).

Meta-Weight-Net NeurIPS'19: Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting (Official Pytorch implementation for noisy labels). The

243 Jan 03, 2023
A tool for light-duty persistent memoization of API calls

JSON Memoize What is this? json_memoize is a straightforward tool for light-duty persistent memoization, created with API calls in mind. It stores the

1 Dec 11, 2021
ELF file deserializer and serializer library

elfo ELF file deserializer and serializer library. import elfo elf = elfo.ELF.from_path('main') elf ELF( header=ELFHeader( e_ident=e

Filipe LaΓ­ns 3 Aug 23, 2021
Simple python code for compile brainfuck program.

py-brainf*ck Just a basic compiled that compiles your brainf*ck codes and gives you informations about memory, used cells, dumped version, logs etc...

4 Jun 13, 2021
Hashcrack: Hash Bruteforse tool using python

HashCrack Hash Bruteforse tool Usage hashcrack.py -n 6 -c lower -l 5 -a md5 -t 3

Lev 1 May 04, 2022
CupScript is a simple programing language made with python

CupScript CupScript is a simple programming language made with python It includes some basic functions, variables, loops, and some other built in func

FUSEN 23 Dec 29, 2022
Expression interpreter written in Python

Calc Interpreter An interpreter modeled after a calculator implemented in Python 3. The program currently only supports basic mathematical expressions

1 Oct 17, 2021
Think DSP: Digital Signal Processing in Python, by Allen B. Downey.

ThinkDSP LaTeX source and Python code for Think DSP: Digital Signal Processing in Python, by Allen B. Downey. The premise of this book (and the other

Allen Downey 3.2k Jan 08, 2023
A reproduction repo for a Scheduling bug in AirFlow 2.2.3

A reproduction repo for a Scheduling bug in AirFlow 2.2.3

Ilya Strelnikov 1 Feb 09, 2022
Understanding the field usage of any object in Salesforce

Understanding the field usage of any object in Salesforce One of the biggest problems that I have addressed while working with Salesforce is to unders

Sebastian Undurraga 1 Dec 14, 2021