Microservice example with Python, Faust-Streaming and Kafka (Redpanda)

Overview

Microservices Orchestration with Python, Faust-Streaming and Kafka (Redpanda)

Example project for PythonBenin meetup. It demonstrates how to use Faust-Streaming to build a microservice using Python and Kafka/Redpanda.

Installation

This project requires poetry. Once poetry is installed, launch the following commands:

git clone [email protected]:lerela/orchestration-python-kafka.git
cd orchestration-python-kafka
poetry install

We will use Redpanda as a replacement for Kafka, but Kafka can also be used. Start a development instance of Redpanda with (do not use this in production!):

docker run -d --pull=always --name=redpanda-1 --rm \
  -p 9092:9092 \
  -p 9644:9644 \
  docker.vectorized.io/vectorized/redpanda:latest \
  redpanda start \
  --overprovisioned \
  --smp 1  \
  --memory 1G \
  --reserve-memory 0M \
  --node-id 0 \
  --check=false

Usage

Start a worker

poetry run faust --datadir=worker-1/ -A orchestration worker --without-web -l info

The parameter --datadir is useful to start multiple workers on the same host (the folder name must be unique for each worker). --without-web disables the Faust web interface since we do not need it.

Send a message on the topic

Using Faust-Streaming: poetry run faust -A orchestration send @topic '{"origin": "Haie Vive", "destination": "Adogleta", "duration": "940"}'

Directly using Redpanda (would work with any Kafka client):

  1. Open a producer with: docker exec -it redpanda-1 rpk topic produce zem-ride --brokers=localhost:9092
  2. Directly send the messages as JSON (press enter to send a message): {"origin": "Calavi", "destination": "Troisième Pont", "duration": 2560}

Troubleshooting

PartitionsMismatch

If you see the following error:

faust.exceptions.PartitionsMismatch: The source topic 'zem-ride' for table 'durations'
has 1 partitions, but the changelog
topic 'zem-app-durations-changelog' has 8 partitions.

Please make sure the topics have the same number of partitions
by configuring Kafka correctly.

Then Faust has wrongly created 8 partitions for the durations changelog (the topic that synchronizes the table "durations"). Just delete the topic while the worker is running with docker exec -it redpanda-1 rpk topic delete zem-app-durations-changelog, and it will be properly re-created. Alternatively, you can delete it when the worker is stopped and manually re-create the topic with rpk topic create zem-app-durations-changelog -p 1 -r 1. Of course, increase the partition number if you wish to use multiple workers.

License

This micro-project is an example of a talk I gave at PythonBenin and is licensed under the MIT License.

CherryPy is a pythonic, object-oriented HTTP framework. https://docs.cherrypy.org/

Welcome to the GitHub repository of CherryPy! CherryPy is a pythonic, object-oriented HTTP framework. It allows building web applications in much the

CherryPy 1.6k Dec 29, 2022
An effective, simple, and async security library for the Sanic framework.

Sanic Security An effective, simple, and async security library for the Sanic framework. Table of Contents About the Project Getting Started Prerequis

Sunset Dev 72 Nov 30, 2022
The core of a service layer that integrates with the Pyramid Web Framework.

pyramid_services The core of a service layer that integrates with the Pyramid Web Framework. pyramid_services defines a pattern and helper methods for

Michael Merickel 78 Apr 15, 2022
A proof-of-concept CherryPy inspired Python micro framework

Varmkorv Varmkorv is a CherryPy inspired micro framework using Werkzeug. This is just a proof of concept. You are free to use it if you like, or find

Magnus Karlsson 1 Nov 22, 2021
PipeLayer is a lightweight Python pipeline framework

PipeLayer is a lightweight Python pipeline framework. Define a series of steps, and chain them together to create modular applications

greaterthan 64 Jul 21, 2022
Swagger/OpenAPI First framework for Python on top of Flask with automatic endpoint validation & OAuth2 support

Connexion Connexion is a framework that automagically handles HTTP requests based on OpenAPI Specification (formerly known as Swagger Spec) of your AP

Zalando SE 4.2k Jan 07, 2023
An easy-to-use high-performance asynchronous web framework.

中文 | English 一个易用的高性能异步 web 框架。 Index.py 文档 Index.py 实现了 ASGI3 接口,并使用 Radix Tree 进行路由查找。是最快的 Python web 框架之一。一切特性都服务于快速开发高性能的 Web 服务。 大量正确的类型注释 灵活且高效的

Index.py 264 Dec 31, 2022
A public API written in Python using the Flask web framework to determine the direction of a road sign using AI

python-public-API This repository is a public API for solving the problem of the final of the AIIJC competition. The task is to create an AI for the c

Lev 1 Nov 08, 2021
Fast⚡, simple and light💡weight ASGI micro🔬 web🌏-framework for Python🐍.

NanoASGI Asynchronous Python Web Framework NanoASGI is a fast ⚡ , simple and light 💡 weight ASGI micro 🔬 web 🌏 -framework for Python 🐍 . It is dis

Kavindu Santhusa 8 Jun 16, 2022
A very simple asynchronous wrapper that allows you to get access to the Oracle database in asyncio programs.

cx_Oracle_async A very simple asynchronous wrapper that allows you to get access to the Oracle database in asyncio programs. Easy to use , buy may not

36 Dec 21, 2022
Full duplex RESTful API for your asyncio web apps

TBone TBone makes it easy to develop full-duplex RESTful APIs on top of your asyncio web application or webservice. It uses a nonblocking asynchronous

TBone Framework 37 Aug 07, 2022
Fast, asynchronous and elegant Python web framework.

Warning: This project is being completely re-written. If you're curious about the progress, reach me on Slack. Vibora is a fast, asynchronous and eleg

vibora.io 5.7k Jan 08, 2023
Official mirror of https://gitlab.com/pgjones/quart

Quart Quart is an async Python web microframework. Using Quart you can, render and serve HTML templates, write (RESTful) JSON APIs, serve WebSockets,

Phil Jones 2 Oct 05, 2022
Web framework based on type hint。

Hint API 中文 | English 基于 Type hint 的 Web 框架 hintapi 文档 hintapi 实现了 WSGI 接口,并使用 Radix Tree 进行路由查找。是最快的 Python web 框架之一。一切特性都服务于快速开发高性能的 Web 服务。 大量正确的类型

Aber 19 Dec 02, 2022
The no-nonsense, minimalist REST and app backend framework for Python developers, with a focus on reliability, correctness, and performance at scale.

The Falcon Web Framework Falcon is a reliable, high-performance Python web framework for building large-scale app backends and microservices. It encou

Falconry 9k Jan 01, 2023
Mini Web Framework on MicroPython (Esp8266)

dupgee Dupgee is a mini web framework developed for micro-python(Tested on esp8266). Installation pip install dupgee Create Project dupgee create newp

ahmet kotan 38 Jul 25, 2022
Flask + Docker + Nginx + Gunicorn + MySQL + Factory Method Pattern

This Flask project is reusable and also an example of how to merge Flask, Docker, Nginx, Gunicorn, MySQL, new: Flask-RESTX, Factory Method design pattern, and other optional dependencies such as Dyna

Facundo Padilla 19 Jul 23, 2022
The Modern And Developer Centric Python Web Framework. Be sure to read the documentation and join the Slack channel questions: http://slack.masoniteproject.com

NOTE: Masonite 2.3 is no longer compatible with the masonite-cli tool. Please uninstall that by running pip uninstall masonite-cli. If you do not unin

Masonite 1.9k Jan 04, 2023
Otter is framework for creating microservices in Flask like fassion using RPC communication via message queue.

Otter Framework for microservices. Overview Otter is framework for creating microservices in Flask like fassion using RPC communication via message qu

Volodymyr Biloshytskyi 4 Mar 23, 2022
FPS, fast pluggable server, is a framework designed to compose and run a web-server based on plugins.

FPS, fast pluggable server, is a framework designed to compose and run a web-server based on plugins. It is based on top of fastAPI, uvicorn, typer, and pluggy.

Adrien Delsalle 1 Nov 16, 2021