Microservice example with Python, Faust-Streaming and Kafka (Redpanda)

Overview

Microservices Orchestration with Python, Faust-Streaming and Kafka (Redpanda)

Example project for PythonBenin meetup. It demonstrates how to use Faust-Streaming to build a microservice using Python and Kafka/Redpanda.

Installation

This project requires poetry. Once poetry is installed, launch the following commands:

git clone [email protected]:lerela/orchestration-python-kafka.git
cd orchestration-python-kafka
poetry install

We will use Redpanda as a replacement for Kafka, but Kafka can also be used. Start a development instance of Redpanda with (do not use this in production!):

docker run -d --pull=always --name=redpanda-1 --rm \
  -p 9092:9092 \
  -p 9644:9644 \
  docker.vectorized.io/vectorized/redpanda:latest \
  redpanda start \
  --overprovisioned \
  --smp 1  \
  --memory 1G \
  --reserve-memory 0M \
  --node-id 0 \
  --check=false

Usage

Start a worker

poetry run faust --datadir=worker-1/ -A orchestration worker --without-web -l info

The parameter --datadir is useful to start multiple workers on the same host (the folder name must be unique for each worker). --without-web disables the Faust web interface since we do not need it.

Send a message on the topic

Using Faust-Streaming: poetry run faust -A orchestration send @topic '{"origin": "Haie Vive", "destination": "Adogleta", "duration": "940"}'

Directly using Redpanda (would work with any Kafka client):

  1. Open a producer with: docker exec -it redpanda-1 rpk topic produce zem-ride --brokers=localhost:9092
  2. Directly send the messages as JSON (press enter to send a message): {"origin": "Calavi", "destination": "Troisième Pont", "duration": 2560}

Troubleshooting

PartitionsMismatch

If you see the following error:

faust.exceptions.PartitionsMismatch: The source topic 'zem-ride' for table 'durations'
has 1 partitions, but the changelog
topic 'zem-app-durations-changelog' has 8 partitions.

Please make sure the topics have the same number of partitions
by configuring Kafka correctly.

Then Faust has wrongly created 8 partitions for the durations changelog (the topic that synchronizes the table "durations"). Just delete the topic while the worker is running with docker exec -it redpanda-1 rpk topic delete zem-app-durations-changelog, and it will be properly re-created. Alternatively, you can delete it when the worker is stopped and manually re-create the topic with rpk topic create zem-app-durations-changelog -p 1 -r 1. Of course, increase the partition number if you wish to use multiple workers.

License

This micro-project is an example of a talk I gave at PythonBenin and is licensed under the MIT License.

Cses2humio - CrowdStrike Falcon Event Stream to Humio

CrowdStrike Falcon Event Stream to Humio This project intend to provide a simple

Trifork.Security 6 Aug 02, 2022
A Simple Kivy Greeting App

SimpleGreetingApp A Simple Kivy Greeting App This is a very simple GUI App that receives a name text input from the user and returns a "Hello" greetin

Mariya 40 Dec 02, 2022
WebSocket and WAMP in Python for Twisted and asyncio

Autobahn|Python WebSocket & WAMP for Python on Twisted and asyncio. Quick Links: Source Code - Documentation - WebSocket Examples - WAMP Examples Comm

Crossbar.io 2.4k Jan 06, 2023
Phoenix LiveView but for Django

Reactor, a LiveView library for Django Reactor enables you to do something similar to Phoenix framework LiveView using Django Channels. What's in the

Eddy Ernesto del Valle Pino 526 Jan 02, 2023
Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed.

Tornado Web Server Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed. By using non-blocking ne

20.9k Jan 01, 2023
The Modern And Developer Centric Python Web Framework. Be sure to read the documentation and join the Slack channel questions: http://slack.masoniteproject.com

NOTE: Masonite 2.3 is no longer compatible with the masonite-cli tool. Please uninstall that by running pip uninstall masonite-cli. If you do not unin

Masonite 1.9k Jan 04, 2023
The comprehensive WSGI web application library.

Werkzeug werkzeug German noun: "tool". Etymology: werk ("work"), zeug ("stuff") Werkzeug is a comprehensive WSGI web application library. It began as

The Pallets Projects 6.2k Jan 01, 2023
Low code web framework for real world applications, in Python and Javascript

Full-stack web application framework that uses Python and MariaDB on the server side and a tightly integrated client side library.

Frappe 4.3k Dec 30, 2022
An abstract and extensible framework in python for building client SDKs and CLI tools for a RESTful API.

django-rest-client An abstract and extensible framework in python for building client SDKs and CLI tools for a RESTful API. Suitable for APIs made wit

Certego 4 Aug 25, 2022
Pulumi-checkly - Checkly Pulumi Provider With Python

🚨 This project is still in very early stages and is not stable, use at your own

Checkly 16 Dec 15, 2022
O SnakeG é um WSGI feito para suprir necessidadades de perfomance e segurança.

SnakeG O SnakeG é um WSGI feito para suprir necessidadades de perfomance e segurança. Veja o que o SnakeG possui: Multiprocessamento de requisições HT

Jaedson Silva 1 Jul 02, 2022
Persistent remote applications for X11; screen sharing for X11, MacOS and MSWindows.

Table of Contents About Installation Usage Help About Xpra is known as "screen for X" : its seamless mode allows you to run X11 programs, usually on a

xpra.org 785 Dec 30, 2022
Goblet is an easy-to-use framework that enables developers to quickly spin up fully featured REST APIs with python on GCP

GOBLET Goblet is a framework for writing serverless rest apis in python in google cloud. It allows you to quickly create and deploy python apis backed

Austen 78 Dec 27, 2022
A proof-of-concept CherryPy inspired Python micro framework

Varmkorv Varmkorv is a CherryPy inspired micro framework using Werkzeug. This is just a proof of concept. You are free to use it if you like, or find

Magnus Karlsson 1 Nov 22, 2021
Library for building WebSocket servers and clients in Python

What is websockets? websockets is a library for building WebSocket servers and clients in Python with a focus on correctness and simplicity. Built on

Aymeric Augustin 4.3k Dec 31, 2022
Free and open source full-stack enterprise framework for agile development of secure database-driven web-based applications, written and programmable in Python.

Readme web2py is a free open source full-stack framework for rapid development of fast, scalable, secure and portable database-driven web-based applic

2k Dec 31, 2022
The web framework for inventors

Emmett is a full-stack Python web framework designed with simplicity in mind. The aim of Emmett is to be clearly understandable, easy to be learned an

Emmett 796 Dec 26, 2022
The no-nonsense, minimalist REST and app backend framework for Python developers, with a focus on reliability, correctness, and performance at scale.

The Falcon Web Framework Falcon is a reliable, high-performance Python web framework for building large-scale app backends and microservices. It encou

Falconry 9k Jan 01, 2023
bottle.py is a fast and simple micro-framework for python web-applications.

Bottle: Python Web Framework Bottle is a fast, simple and lightweight WSGI micro web-framework for Python. It is distributed as a single file module a

Bottle Micro Web Framework 7.8k Dec 31, 2022
A comprehensive reference for all topics related to building and maintaining microservices

This pandect (πανδέκτης is Ancient Greek for encyclopedia) was created to help you find and understand almost anything related to Microservices that i

Ivan Bilan 64 Dec 09, 2022