Repository for Research Paper: Serverless Thumbnail Generation

Overview

Repository for Research Paper: Serverless Thumbnail Generation

Serverless deployment on IBM Cloud Code Engine that uses the Lithops Framework to process images in parallel

English Title: Juxtaposition of thumbnail generation on IBM Cloud Code Engine with multiprocessing through the Lithops Framework and the traditional deployment on a virtual machine

German Title: Gegenüberstellung eines serverless Batch Jobs basierend auf dem Lithops Framework zur parallelen Generierung von Thumbnails auf IBM Cloud Code Engine und der Bereitstellung auf einem Virtual Server im Hinblick auf Performance, Skalierbarkeit und Kosten

(susceptible to change)

This repository contains:

  • The code for the thumbnail generator as used for the research paper
    • server_thumbnails_sequential.py (Sequentially generate thumbnails on local machine)
    • server_thumbnails_multi.py (Use the standard python multiprocessing library to generate thumbnails in parallel on local machine)
    • serverless_thumbnails.py (Uses Lithops to create a Batch Job that will spawn one worker per image to be processed. Benefits from principle of unlimited capacity in the cloud)
  • Documentation including charts and architecture
  • The dataset of images I used for the research paper
  • Logs&Stats of all the experiments I conducted for transparency

Recreating the results from the research paper:

  • Closely follow the following steps to set everything up
  • Run server_thumbnails_sequential.py and server_thumbnails_multi.py on a virtual machine with the specs:
    • 2 vCPU
    • 8 GiB Memory
    • 4 Gbps Network-Bandwidth
  • Even when following the instructions the results might vary from the ones in the paper because of:
    • Your used images (My dataset of images is in the /dataset directory)
    • Varying response time of IBM Cloud COS, IBM Cloud Code Engine
    • OS-specific configuration of the virtual machine

RUNNING MY SCRIPT WILL CREATE COSTS due to high bandwidth usage of COS and usage of Code Engine or a Virtual Server

Running the Thumbnail Generator

Setting up requirements

  • Create an IBM Cloud IAM Key here
  • Create an IBM Cloud COS Bucket and store all images to be processed in a directory called images
  • Create an IBM Cloud Code Engine Namespace
  • Add these keys with your values to the .lithops_config:
lithops:
    storage: ibm_cos
    storage_bucket: 
   
    
    backend: code_engine
    mode: serverless
    
ibm:
    iam_api_key: 
    
     
   
code_engine:
    namespace: 
     
      
    region: 
      
       
    runtime: ibmfunctions/lithops-ce-v385:235

      
     
    
   

For more info, have a look here

Sequential Thumbnail Generation

  1. Run server_thumbnails_sequential.py
  2. The script will read the available images in the bucket
  3. Specify the number of images to be turned into thumbnails in the console
  4. The script will generate the thumbnails and will save them to the /thumbnails directory in your bucket root

Multiprocessing Thumbnail Generation

  1. Run server_thumbnails_multi.py
  2. The script will read the available images in the bucket
  3. Specify the number of images to be turned into thumbnails in the console
  4. The script will generate the thumbnails and will save them to the /thumbnails directory in your bucket root

Serverless Thumbnail Generation (Lithops)

  1. Run serverless_thumbnails.py
  2. The script will read the available images in the bucket
  3. Specify the number of images to be turned into thumbnails in the console
  4. The script will generate the thumbnails and will save them to the /thumbnails directory in your bucket root

Architecture

Thumbnail Generator System Context

Thumbnail Generator Architecture

Results

Dataset images

All images are taken from Unsplash and do not require attribution. To find the author of each image, add the image name to the following url: https://unsplash.com/photos/

Owner
Luca Mueller
Working at IBM
Luca Mueller
Serverless console based chat program made with PyDrive2.

Nautilus Serverless console based chat program made with PyDrive2. How To Use: After installing the program files from GitHub, they can be put anywher

legitimately_the_1984th_year_of_the_common_era_&_anno_domini_designations 0 Jul 30, 2022
Zappa makes it super easy to build and deploy server-less, event-driven Python applications on AWS Lambda + API Gateway.

Zappa makes it super easy to build and deploy server-less, event-driven Python applications (including, but not limited to, WSGI web apps) on AWS Lambda + API Gateway. Think of it as "serverless" web

Zappa 2.2k Jan 09, 2023
Chainlink Python Serverless External Adapter Template

This template shows a basic usecase of an external adapter written in Python for the CryptoCompare API. It can be ran locally, in Docker, AWS Lambda, or GCP Functions.

Chainlink Hackathon - Insureblox 1 Nov 17, 2021
⚡ Serverless Framework – Build web, mobile and IoT applications with serverless architectures using AWS Lambda, Azure Functions, Google CloudFunctions

⚡ Serverless Framework – Build web, mobile and IoT applications with serverless architectures using AWS Lambda, Azure Functions, Google CloudFunctions & more! –

Serverless 44k Jan 03, 2023
Serverless Public Key Infrastructure Framework

Ottr: Serverless Public Key Infrastructure Framework Ottr is a serverless framework for Public Key Infrastructure (PKI) that aims to provide a robust

Airbnb 259 Dec 28, 2022
Serverless function for replicating weather underground data to an influxDB database

Weather Underground → Influx DB 🌤 Serverless function for replicating Weather U

Ben Meier 1 Dec 30, 2021
Repository for Research Paper: Serverless Thumbnail Generation

Serverless deployment on IBM Cloud Code Engine that uses the Lithops Framework to process images in parallel

Luca Mueller 2 Jan 12, 2022
Python Serverless Microframework for AWS

AWS Chalice Chalice is a framework for writing serverless apps in python. It allows you to quickly create and deploy applications that use AWS Lambda.

Amazon Web Services 9.4k Jan 08, 2023
Portfolio-tracker - This serverless application let's you keep track of your investment portfolios

Portfolio-tracker - This serverless application let's you keep track of your investment portfolios

José Coelho 1 Jan 23, 2022