A python tool for synchronizing the messages from different threads, processes, or hosts.

Related tags

Miscellaneouspythonsynchronizationstreammultiprocessingpython-librarystdoutpython3stdout-redirection
Overview

Sync-stream

This project is designed for providing the synchoronization of the stdout / stderr among different threads, processes, devices or hosts. The package could be used for the following cases:

  1. Use syncstream.LineBuffer: Multiple threads are created. The messages (stdout) from different threads are required to be collected.
  2. Use syncstream.LineProcBuffer in the main process, and syncstream.LineProcMirror in the sub-processes: Multiple sub-processes are created on the same device. The stdout / stderr of each process is redirected to a LineProcMirror, and the results are collected by LineProcBuffer.
  3. Use syncstream.LineFileBuffer: Multiple processes are created. These processes may be deployed on different devices (even with different platforms), but all devices could get accessed to the same shared disk. In this case, the message could be shared by locked files. Each process would hold an independent LineFileBuffer pointing to the same log files.
  4. Use syncstream.LineHostBuffer on the server side, and syncstream.LineHostMirror on the client side: Multiple processes are deployed on different devices, and they could not share the same disk. In this case, the message are synchronized by the web service. Each process would hold a LineHostMirror, and the results would be collected by LineHostBuffer.

The basic package would not contain the file and host modules. To install the package, please use the following options:

pip install syncstream[option1,option2...]
Option Supports
file Install dependencies for the file module. The module provides syncstream.LineFileBuffer.
host Install dependencies for the host module. The module provides syncstream.LineHostBuffer, and syncstream.LineHostMirror.

Documentation

View the documentation here: 📘 https://cainmagi.github.io/sync-stream/

Update reports

0.3.3 @ 6/29/2021

  1. Fix small typos.
  2. Bump the dependencies to the newest versions.

0.3.2 @ 6/14/2021

  1. Fix a bug caused by stopping the mirrors.
  2. Format the meta-data defined in setup.py.
  3. Add the documentation. Currently only the tutorial is finished.

0.3.0 @ 6/4/2021

  1. Support the stop signal for mproc and host modules.
  2. Fix some bugs in the testing script.
  3. Fix typos.

0.2.2 @ 5/25/2021

  1. Add clear() methods for all buffers and mirrors.
  2. Fix typos in the package setup and info file.
  3. Fix a bug caused by writing data to the host in the testing scripts for Linux.

0.2.1 @ 5/24/2021

  1. Add the PyPI publish workflow.

0.2.0 @ 5/24/2021

  1. Finish the synchronization based on the file lock package fasteners.
  2. Finish the synchronization based on the web service packages flask, flask-restful and urllib3.
  3. Fix the compatibility of the testing scripts for py36, py37.

0.1.0 @ 5/22/2021

  1. Finish the synchronization based on the stdlib.
  2. Create this project.
You might also like...
Provide error messages for Python exceptions, even if the original message is empty

errortext is a Python package to provide error messages for Python exceptions, even if the original message is empty.

Thomas Aglassinger 0 Dec 7, 2021
Telegram bot to remove the forwarded tag from messages.
Telegram bot to remove the forwarded tag from messages.

Anonymous Sender Bot @AnonySendBot Telegram bot to remove the forwarded tag from messages. Table of Contents Usage Deploy To Heroku Local Deploying En

Stark Bots 26 Nov 24, 2022
Automatically remove user join messages when the user leaves the server.

CleanLeave Automatically remove user join messages when the user leaves the server. Installation You will need to install poetry to run this bot local

null 11 Sep 19, 2022
Simple script with AminoLab to send ghost messages

Simple script with AminoLab to send ghost messages

Moleey 1 Nov 22, 2021
A Github Action for sending messages to a Matrix Room.
A Github Action for sending messages to a Matrix Room.

matrix-commit A Github Action for sending messages to a Matrix Room. Screenshot: Example Usage: # .github/workflows/matrix-commit.yml on: push:

null 3 Sep 11, 2022
ViberExport - Export messages from Viber messenger using viber.db file

📲 ViberExport Export messages from Viber messenger using viber.db file ⚡ Usage:

null 7 Nov 23, 2022
Telop - Encode and decode messages using an interpretation of the telegraphic code devised by José María Mathé

telop Telop (TELégrafoÓPtico) - Utilidad para codificar y descodificar mensajes de texto empleando una interpretación del código telegráfico ideado po

Ricardo F. 4 Nov 1, 2022
Automatically unpin old messages so you can always pin more!

PinRotate Automatically unpin old messages so you can always pin more! Installation You will need to install poetry to run this bot locally for develo

null 3 Sep 18, 2022
A tool to flash .ofp files in bootloader mode without needing MSM Tool, an alternative to official realme tool
A tool to flash .ofp files in bootloader mode without needing MSM Tool, an alternative to official realme tool

Oppo/Realme Flash .OFP File on Bootloader A tool to flash .ofp files in bootloader mode without needing MSM Tool, an alternative to official realme to

Italo Almeida 70 Jan 2, 2023
Releases(0.3.3)

  • 0.3.3(Jun 29, 2021)

    Introduction

    Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.3.3 @ 6/29/2021

    1. Fix small typos.
    2. Bump the dependencies to the newest versions.
    Source code(tar.gz)
    Source code(zip)

  • 0.3.2(Jun 14, 2021)

    Introduction

    Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.3.2 @ 6/14/2021

    1. Fix a bug caused by stopping the mirrors.
    2. Format the meta-data defined in setup.py.
    3. Add the documentation. Currently, only the tutorial is finished.
    Source code(tar.gz)
    Source code(zip)

  • 0.3.0(Jun 4, 2021)

    Introduction

    Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.3.0 @ 6/4/2021

    1. Support the stop signal for mproc and host modules.
    2. Fix some bugs in the testing script.
    3. Fix typos.
    Source code(tar.gz)
    Source code(zip)

  • 0.2.2(May 25, 2021)

    Introduction

    Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.2.2 @ 5/25/2021

    1. Add clear() methods for all buffers and mirrors.
    2. Fix typos in the package setup and info file.
    3. Fix a bug caused by writing data to the host in the testing scripts for Linux.
    Source code(tar.gz)
    Source code(zip)

  • 0.2.1(May 24, 2021)

    Introduction

    This is the first release of this project. Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.2.1 @ 5/24/2021

    1. Add the PyPI publish workflow.

    0.2.0 @ 5/24/2021

    1. Finish the synchronization based on the file lock package fasteners.
    2. Finish the synchronization based on the web service packages flask, flask-restful and urllib3.
    3. Fix the compatibility of the testing scripts for py36, py37.

    0.1.0 @ 5/22/2021

    1. Finish the synchronization based on the stdlib.
    2. Create this project.
    Source code(tar.gz)
    Source code(zip)
Owner
Yuchen Jin
A Ph.D student in University of Houston (UH). Interested area includes: machine learning, programming and religion.
Yuchen Jin
GitHub Repository https://cainmagi.github.io/sync-stream/
Tools I'm building in order to help my investments decisions

b3-tools Tools I'm building in order to help my investments decisions. Based in the REITs I've in my personal portifolio I ran a script that scrapy th

Rafael Cassau 2 Jan 21, 2022
Convert-Decimal-to-Binary-Octal-and-Hexadecimal

Convert-Decimal-to-Binary-Octal-and-Hexadecimal We have a number in a decimal number, and we have to convert it into a binary, octal, and hexadecimal

Maanyu M 2 Oct 08, 2021
addons to the turtle package that help you drew stuff more quickly

TurtlePlus addons to the turtle package that help you drew stuff more quickly --------------

1 Nov 18, 2021
Streamlit apps done following data professor's course on YouTube

streamlit-twelve-apps Streamlit apps done following data professor's course on YouTube Español Curso de apps de data science hecho por Data Professor

Federico Bravin 1 Jan 10, 2022
Terrible python code from the "bubble that breaks maths" video.

Terrible python code from the "bubble that breaks maths" video.

Stand-up Maths 12 Oct 25, 2022
Social reading and reviewing, decentralized with ActivityPub

BookWyrm Social reading and reviewing, decentralized with ActivityPub Contents Joining BookWyrm Contributing About BookWyrm What it is and isn't The r

BookWyrm 1.4k Jan 08, 2023
A Python Based Utility for Processing GST-Return JSON Files to Multiple Formats

GSTR 1/2A Utility by Shan.tk Open Source GSTR 1/GSTR 2A JSON to Excel utility based on Python. Useful for Auditors in Verifying GSTR 1 Return Invoices

Sudharshan TK 1 Oct 08, 2022
Originally used during Marketplace.tf's open period, this program was used to get the profit of items bought with keys and sold for dollars.

Originally used during Marketplace.tf's open period, this program was used to get the profit of items bought with keys and sold for dollars. Practically useless for me now, but can be used as an exam

BoggoTV 1 Dec 11, 2021
GDSC UIET KUK 📍 , welcomes you all to this amazing event where you will be introduced to the world of coding 💻 .

GDSC UIET KUK 📍 , welcomes you all to this amazing event where you will be introduced to the world of coding 💻 .

Google Developer Student Club UIET KUK 9 Mar 24, 2022
Display your data in an attractive way in your notebook!

Bloxs Bloxs is a simple python package that helps you display information in an attractive way (formed in blocks). Perfect for building dashboards, re

MLJAR 192 Dec 28, 2022
Proyectos de ejercicios básicos y avanzados hecho en python

Proyectos Básicos y Avanzados hecho en python Instalación: Tener instalado python 3.x o superior. Tener pip instalado. Tener virtualenv o venv instala

Karlo Xavier Chok 1 Dec 27, 2021
Library for mocking AsyncIOMotorClient built on top of mongomock.

mongomock-motor Best effort mock for AsyncIOMotorClient (Database, Collection, e.t.c) built on top of mongomock library. Example / Showcase from mongo

Michael Kryukov 43 Jan 04, 2023
A python script to turn tabs into spaces the right way.

detab A python script to turn tabs into spaces the right way. detab turns all tabs into spaces, not just leading tabs. Not all tabs have the same leng

1 Jan 26, 2022
Assembly example for CadQuery

Spindle and vacuum attachment This is a model of the vacuum attachment for my Workbee CNC router. There is a mist spray coming from the left hand side

Marcus Boyd 20 Sep 16, 2022
I³ Tracker for Essential Open Innovation Datasets

I³ Tracker for Essential Open Innovation Datasets This repository is set up to track, version, and contribute updates to the I³ Essential Open Innovat

1 Feb 08, 2022
Python script which synchronizes the replica-directoty with the original-one.

directories_synchronizer Python script which synchronizes the replica-directoty with the original-one. Automatically detects all changes when script i

0 Feb 13, 2022
Powerful virtual assistant in python

Virtual assistant in python Powerful virtual assistant in python Set up Step 1: download repo and unzip Step 2: pip install requirements.txt (if py au

Arkal 3 Jan 23, 2022
The goal of this program was to find the most common color in my living room.

The goal of this program was to find the most common color in my living room. I found a dataset online with colors names and their corr

1 Nov 09, 2021
Osintgram by Datalux but i fixed some errors i found and made it look cleaner

OSINTgram-V2 OSINTgram-V2 is made from Osintgram which is made by Datalux originally but i took the script and fixed some errors i found and made the

2 Feb 02, 2022
Is a polybar module that will show you your progress in Hack The Box

HTB-Status for Polybar Is a polybar module that will show you your progress in Hack The Box indicating your current rank, global rank, points and resp

bitc0de 8 Jan 14, 2022