A minimal configuration for a dockerized kafka project.

Related tags

Miscellaneousdockerkafkaapachezookeeper
Overview

Docker Kafka Quickstart

A minimal configuration for a dockerized kafka project.

Usage:

  • Run this command to build kafka and zookeeper containers, and create the network kafka-network, where your containers can access the endpoint kafka:29092.
# creates the network `kafka-network` and build and
# runs the containers for zookeeper and kafka
docker-compose -f "docker-compose.kafka.yml" up -d --build # remove -d if you want to see logs in the same shell
  • You can then run as many broker/consumer containers on your kafka-network, in this case: generator is a broker, and indicator is a consumer.
# runs the generator and indicator containers
docker-compose up -d --build
  • If you check the logs in your generator container you should be seeing the messages sent to the queuing.sentences.test topic:
### KAFKA_BROKER_URL kafka:29092
09:35:09.135969 {'time': '09:35:08.107419', 'player': 'player2', 'sentence': 'sDZRJk nwENFG I sR VOOZNWll aGDdZV'}
09:35:10.138498 {'time': '09:35:10.138207', 'player': 'player2', 'sentence': 'vgIXxE  sa'}
09:35:11.141023 {'time': '09:35:11.140690', 'player': 'player1', 'sentence': 'E raH jzIeKHqXZF JGGB   tx  M oLusi '}
09:35:12.142630 {'time': '09:35:12.142360', 'player': 'player2', 'sentence': 'WxsPlteC  yxh  ogME dvbtFn l b TxVB  gaMFZe'}
09:35:13.144693 {'time': '09:35:13.144454', 'player': 'player1', 'sentence': ' aIRVm W TeIcKuigy  HPLOvJrQSXSl ZlLHlL FRH'}
09:35:14.146182 {'time': '09:35:14.145966', 'player': 'player1', 'sentence': 'cmMH k phZ LapRdGv C geTDzIT'}
09:35:15.147609 {'time': '09:35:15.147378', 'player': 'player2', 'sentence': 'QihJPzjFM WmkmDzMLl DkyydRVeyEI Rm'}
09:35:16.149716 {'time': '09:35:16.149502', 'player': 'player1', 'sentence': '  C '}
09:35:17.151371 {'time': '09:35:17.151155', 'player': 'player2', 'sentence': ''}
09:35:18.152736 {'time': '09:35:18.152508', 'player': 'player1', 'sentence': 'Oh tgfSl tY iBrxMHf aNX h'}
...
  • And the logs for indicator show the messages recieved on the queuing.sentences.test topic:
### KAFKA_BROKER_URL kafka:29092
Connecting...
Connected to queuing.sentences.test topic!
09:35:13.320414 {'time': '09:35:13.144454', 'player': 'player1', 'sentence': ' aIRVm W TeIcKuigy  HPLOvJrQSXSl ZlLHlL FRH'}
09:35:14.150851 {'time': '09:35:14.145966', 'player': 'player1', 'sentence': 'cmMH k phZ LapRdGv C geTDzIT'}
09:35:15.279800 {'time': '09:35:15.147378', 'player': 'player2', 'sentence': 'QihJPzjFM WmkmDzMLl DkyydRVeyEI Rm'}
09:35:16.153846 {'time': '09:35:16.149502', 'player': 'player1', 'sentence': '  C '}
09:35:17.156841 {'time': '09:35:17.151155', 'player': 'player2', 'sentence': ''}
09:35:18.156372 {'time': '09:35:18.152508', 'player': 'player1', 'sentence': 'Oh tgfSl tY iBrxMHf aNX h'}
09:35:19.160985 {'time': '09:35:19.154674', 'player': 'player2', 'sentence': 'qgSAzkUNm  fBQyruw S U   PvLnharO '}
09:35:20.183376 {'time': '09:35:20.157269', 'player': 'player1', 'sentence': 'OlX SW  a gcpXVGHxW'}

Refs:

For more information about using kafka with Docker: https://hub.docker.com/r/wurstmeister/kafka/

Owner
Nouamane Tazi
Software Engineer | AI Passionate. Seeking end-of-studies 6-month internship.
Nouamane Tazi
GitHub Repository
Wunderland desktop wallpaper and Microsoft Teams background.

Wunderland Professional Impress your colleagues, friends and family with this edition of the "Wunderland" wallpaper. With the nostalgic feel of the or

3 Dec 14, 2022
Logging-monitoring-instrumentation - A brief repository on logging monitoring and instrumentation in Python

logging-monitoring-instrumentation A brief repository on logging monitoring and

Noah Gift 6 Feb 17, 2022
A script to add issues to a project in Github based on label or status.

Add Github Issues to Project (Beta) A python script to move Github issues to a next-gen (beta) Github Project Getting Started These instructions will

Kate Donaldson 3 Jan 16, 2022
Simple script with AminoLab to send ghost messages

Simple script with AminoLab to send ghost messages

Moleey 1 Nov 22, 2021
Datargsing is a data management and manipulation Python library

Datargsing What is It? Datargsing is a data management and manipulation Python library which is currently in deving Why this library is good? This Pyt

CHOSSY Lucas 10 Oct 24, 2022
A webapp that timestamps key moments in a football clip

A look into what we're building Demo.mp4 Prerequisites Python 3 Node v16+ Steps to run Create a virtual environment. Activate the virtual environment.

Pranav 1 Dec 10, 2021
A simple program to run through inputs for a 3n+1 problem

Author Tyler Windemuth Collatz_Conjecture A simple program to run through inputs for a 3n+1 problem Purpose: doesn't really have a purpose, did this t

0 Apr 22, 2022
thonny plugin for gitonic

thonny-gitonic thonny plugin for gitonic open gitonic in thonny by pressing Control+Shift+g, or via tools menu press ESC key to minimize gitonic windo

karl 1 Apr 12, 2022
Animation retargeting tool for Autodesk Maya. Retargets mocap to a custom rig with a few clicks.

Animation Retargeting Tool for Maya A tool for transferring animation data between rigs or transfer raw mocap from a skeleton to a custom rig. (The sc

Joaen 62 Dec 19, 2022
"Hacking" the (Telekom) Zyxel GPON SFP module (PMG3000-D20B)

"Hacking" the (Telekom) Zyxel GPON SFP module (PMG3000-D20B) The SFP can be sour

Matthias Riegler 52 Jan 03, 2023
An interactive course to git

OperatorEquals' Sandbox Git Course! Preface This Git course is an ongoing project containing use cases that I've met (and still meet) while working in

John Torakis 62 Sep 19, 2022
UF3: a python library for generating ultra-fast interatomic potentials

Ultra-Fast Force Fields (UF3) S. R. Xie, M. Rupp, and R. G. Hennig, "Ultra-fast interpretable machine-learning potentials", preprint arXiv:2110.00624

Ultra-Fast Force Fields 24 Nov 13, 2022
A lightweight and unlocked launcher for Lunar Client made in Python.

LCLPy LCL's Python Port of Lunar Client Lite. Releases: https://github.com/Aetopia/LCLPy/releases Build Install PyInstaller. pip install PyInstaller

21 Aug 03, 2022
Demo scripts for the Kubernetes Security Webinar

Kubernetes Security Webinar [in Russian] YouTube video (October 13, 2021) Authors: Artem Yushkovsky (LinkedIn, GitHub) Maxim Mosharov @ Whitespots.io

Slurm 34 Dec 06, 2022
🦕 Compile Deno executables and compress them for all platforms easily

Denoc Compile Deno executables and compress them for all platforms easily. Install You can install denoc from PyPI like any other package: pip install

Eliaz Bobadilla 8 Apr 04, 2022
This application demonstrates IoTVAS device discovery and security assessment API integration with the Rapid7 InsightVM.

Introduction This repository hosts a sample application that demonstrates integrating Firmalyzer's IoTVAS API with the Rapid7 InsightVM platform. This

Firmalyzer BV 4 Nov 09, 2022
A collection of examples of using cocotb for functional verification of VHDL designs with GHDL.

At the moment, this repo is in an early state and serves as a learning tool for me. So it contains a a lot of quirks and code which can be done much better by cocotb-professionals.

T. Meissner 7 Mar 10, 2022
Async-first dependency injection library based on python type hints

Dependency Depression Async-first dependency injection library based on python type hints Quickstart First let's create a class we would be injecting:

Doctor 8 Oct 10, 2022
A python script that will automate the boring task of login to the captive portal again and again

A python script that will automate the boring task of login to the captive portal again and again

Rakib Hasan 2 Feb 09, 2022
Refer'd Resume Scanner

Refer'd Resume Scanner I wanted to share a free resource we built to assist applicants with resume building. Our resume scanner identifies potential s

Refer'd 74 Mar 07, 2022