A python tool for synchronizing the messages from different threads, processes, or hosts.

Related tags

Miscellaneouspythonsynchronizationstreammultiprocessingpython-librarystdoutpython3stdout-redirection
Overview

Sync-stream

This project is designed for providing the synchoronization of the stdout / stderr among different threads, processes, devices or hosts. The package could be used for the following cases:

  1. Use syncstream.LineBuffer: Multiple threads are created. The messages (stdout) from different threads are required to be collected.
  2. Use syncstream.LineProcBuffer in the main process, and syncstream.LineProcMirror in the sub-processes: Multiple sub-processes are created on the same device. The stdout / stderr of each process is redirected to a LineProcMirror, and the results are collected by LineProcBuffer.
  3. Use syncstream.LineFileBuffer: Multiple processes are created. These processes may be deployed on different devices (even with different platforms), but all devices could get accessed to the same shared disk. In this case, the message could be shared by locked files. Each process would hold an independent LineFileBuffer pointing to the same log files.
  4. Use syncstream.LineHostBuffer on the server side, and syncstream.LineHostMirror on the client side: Multiple processes are deployed on different devices, and they could not share the same disk. In this case, the message are synchronized by the web service. Each process would hold a LineHostMirror, and the results would be collected by LineHostBuffer.

The basic package would not contain the file and host modules. To install the package, please use the following options:

pip install syncstream[option1,option2...]
Option Supports
file Install dependencies for the file module. The module provides syncstream.LineFileBuffer.
host Install dependencies for the host module. The module provides syncstream.LineHostBuffer, and syncstream.LineHostMirror.

Documentation

View the documentation here: 📘 https://cainmagi.github.io/sync-stream/

Update reports

0.3.3 @ 6/29/2021

  1. Fix small typos.
  2. Bump the dependencies to the newest versions.

0.3.2 @ 6/14/2021

  1. Fix a bug caused by stopping the mirrors.
  2. Format the meta-data defined in setup.py.
  3. Add the documentation. Currently only the tutorial is finished.

0.3.0 @ 6/4/2021

  1. Support the stop signal for mproc and host modules.
  2. Fix some bugs in the testing script.
  3. Fix typos.

0.2.2 @ 5/25/2021

  1. Add clear() methods for all buffers and mirrors.
  2. Fix typos in the package setup and info file.
  3. Fix a bug caused by writing data to the host in the testing scripts for Linux.

0.2.1 @ 5/24/2021

  1. Add the PyPI publish workflow.

0.2.0 @ 5/24/2021

  1. Finish the synchronization based on the file lock package fasteners.
  2. Finish the synchronization based on the web service packages flask, flask-restful and urllib3.
  3. Fix the compatibility of the testing scripts for py36, py37.

0.1.0 @ 5/22/2021

  1. Finish the synchronization based on the stdlib.
  2. Create this project.
You might also like...
Provide error messages for Python exceptions, even if the original message is empty

errortext is a Python package to provide error messages for Python exceptions, even if the original message is empty.

Thomas Aglassinger 0 Dec 7, 2021
Telegram bot to remove the forwarded tag from messages.
Telegram bot to remove the forwarded tag from messages.

Anonymous Sender Bot @AnonySendBot Telegram bot to remove the forwarded tag from messages. Table of Contents Usage Deploy To Heroku Local Deploying En

Stark Bots 26 Nov 24, 2022
Automatically remove user join messages when the user leaves the server.

CleanLeave Automatically remove user join messages when the user leaves the server. Installation You will need to install poetry to run this bot local

null 11 Sep 19, 2022
Simple script with AminoLab to send ghost messages

Simple script with AminoLab to send ghost messages

Moleey 1 Nov 22, 2021
A Github Action for sending messages to a Matrix Room.
A Github Action for sending messages to a Matrix Room.

matrix-commit A Github Action for sending messages to a Matrix Room. Screenshot: Example Usage: # .github/workflows/matrix-commit.yml on: push:

null 3 Sep 11, 2022
ViberExport - Export messages from Viber messenger using viber.db file

📲 ViberExport Export messages from Viber messenger using viber.db file ⚡ Usage:

null 7 Nov 23, 2022
Telop - Encode and decode messages using an interpretation of the telegraphic code devised by José María Mathé

telop Telop (TELégrafoÓPtico) - Utilidad para codificar y descodificar mensajes de texto empleando una interpretación del código telegráfico ideado po

Ricardo F. 4 Nov 1, 2022
Automatically unpin old messages so you can always pin more!

PinRotate Automatically unpin old messages so you can always pin more! Installation You will need to install poetry to run this bot locally for develo

null 3 Sep 18, 2022
A tool to flash .ofp files in bootloader mode without needing MSM Tool, an alternative to official realme tool
A tool to flash .ofp files in bootloader mode without needing MSM Tool, an alternative to official realme tool

Oppo/Realme Flash .OFP File on Bootloader A tool to flash .ofp files in bootloader mode without needing MSM Tool, an alternative to official realme to

Italo Almeida 70 Jan 2, 2023
Releases(0.3.3)

  • 0.3.3(Jun 29, 2021)

    Introduction

    Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.3.3 @ 6/29/2021

    1. Fix small typos.
    2. Bump the dependencies to the newest versions.
    Source code(tar.gz)
    Source code(zip)

  • 0.3.2(Jun 14, 2021)

    Introduction

    Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.3.2 @ 6/14/2021

    1. Fix a bug caused by stopping the mirrors.
    2. Format the meta-data defined in setup.py.
    3. Add the documentation. Currently, only the tutorial is finished.
    Source code(tar.gz)
    Source code(zip)

  • 0.3.0(Jun 4, 2021)

    Introduction

    Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.3.0 @ 6/4/2021

    1. Support the stop signal for mproc and host modules.
    2. Fix some bugs in the testing script.
    3. Fix typos.
    Source code(tar.gz)
    Source code(zip)

  • 0.2.2(May 25, 2021)

    Introduction

    Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.2.2 @ 5/25/2021

    1. Add clear() methods for all buffers and mirrors.
    2. Fix typos in the package setup and info file.
    3. Fix a bug caused by writing data to the host in the testing scripts for Linux.
    Source code(tar.gz)
    Source code(zip)

  • 0.2.1(May 24, 2021)

    Introduction

    This is the first release of this project. Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.2.1 @ 5/24/2021

    1. Add the PyPI publish workflow.

    0.2.0 @ 5/24/2021

    1. Finish the synchronization based on the file lock package fasteners.
    2. Finish the synchronization based on the web service packages flask, flask-restful and urllib3.
    3. Fix the compatibility of the testing scripts for py36, py37.

    0.1.0 @ 5/22/2021

    1. Finish the synchronization based on the stdlib.
    2. Create this project.
    Source code(tar.gz)
    Source code(zip)
Owner
Yuchen Jin
A Ph.D student in University of Houston (UH). Interested area includes: machine learning, programming and religion.
Yuchen Jin
GitHub Repository https://cainmagi.github.io/sync-stream/
Trackthis - This library can be used to track USPS and UPS shipments.

Trackthis - This library can be used to track USPS and UPS shipments. It has the option of returning the raw API response, or optionally, it can be used to standardize the USPS and UPS responses so t

Aaron Guzman 0 Mar 29, 2022
Shows VRML team stats of all players in your pubs

VRML Team Stat Searcher Displays Team Name, Team Rank (Worldwide), and tier of all the players in your pubs. GUI WIP: Only username search works (for

Hamish Burke 2 Dec 22, 2022
A python script to search for k-uniform Euclidean tilings.

k-uniform-solver A python script to search for k-uniform Euclidean tilings. This project's aim is to replicate and extend the list of k-uniform Euclid

3 Dec 06, 2022
Tools for collecting social media data around focal events

Social Media Focal Events The focalevents codebase provides tools for organizing data collected around focal events on social media. It is often diffi

Ryan Gallagher 80 Nov 28, 2022
Data derived from the OpenType specification

This package currently provides the opentypespec.tags module, which exports FEATURE_TAGS, SCRIPT_TAGS, LANGUAGE_TAGS and BASELINE_TAGS dictionaries, representing data from the Layout Tag Registry

Simon Cozens 4 Dec 01, 2022
Saturne best tools pour baiser tout le système de discord

Installation | Important | Discord 🌟 Comme Saturne est gratuit, les dons sont vraiment appréciables et maintiennent le développement! Caractéristique

GalackQSM 8 Oct 02, 2022
Sardana integration into the Jupyter ecosystem.

sardana-jupyter Sardana integration into the Jupyter ecosystem.

Marc Espín 1 Dec 23, 2021
fast_bss_eval is a fast implementation of the bss_eval metrics for the evaluation of blind source separation.

fast_bss_eval Do you have a zillion BSS audio files to process and it is taking days ? Is your simulation never ending ? Fear no more! fast_bss_eval i

Robin Scheibler 99 Dec 13, 2022
Roman numeral conversion with python

Roman numeral conversion Discipline: Programming Languages Student: Paulo Henrique Diniz de Lima Alencar. Language: Python Description Responsible for

Paulo Alencar 1 Jul 11, 2022
New multi tool im making adding features currently

Emera Multi Tool New multi tool im making adding features currently Current List of Planned Features - Linkvertise Bypasser - Discord Auto Bump - Gith

Lamp 3 Dec 03, 2021
Repository to store sample python programs for python learning

py Repository to store sample Python programs. This repository is meant for beginners to assist them in their learning of Python. The repository cover

codebasics 5.8k Dec 30, 2022
A TODO-list tool written in Python

PyTD A TODO-list tool written in Python. Its goal is to provide a stable posibility to get a good view over all your TODOs motivate you to actually fi

1 Feb 12, 2022
A python package template that can be adapted for RAP projects

Warning - this repository is a snapshot of a repository internal to NHS Digital. This means that links to videos and some URLs may not work. Repositor

NHS Digital 3 Nov 08, 2022
Automatically load and dump your dataclasses 📂🙋

file dataclasses Installation By default, filedataclasses comes with support for JSON files only. To support other formats like YAML and TOML, filedat

Alon 1 Dec 30, 2021
This is a working model for which I have used python.

Jarvis_voiceAssistance This is a working model for which I have used python. This model can: 1)Play a video or song on youtube. 2)Tell us time. 3)Tell

Hardik Jain 1 Jan 30, 2022
Traffic flow test platform, especially for reinforcement learning

Traffic Flow Test Platform Traffic flow test platform, especially for reinforcement learning, named TFTP. A traffic signal control framework that can

4 Nov 07, 2022
urlwatch is intended to help you watch changes in webpages and get notified of any changes.

urlwatch is intended to help you watch changes in webpages and get notified (via e-mail, in your terminal or through various third party services) of any changes.

Thomas Perl 2.5k Jan 08, 2023
It is convenient to quickly import Python packages from the network.

It is convenient to quickly import Python packages from the network.

zmaplex 1 Jan 18, 2022
BDD base project: Python + Behave

BDD base project: Python + Behave Basic example of using Python with Behave (BDD). This Gherkin example includes: Basic Scenario Scenario Outline Tagg

eccanto 1 Dec 08, 2021
pydock - Docker-based environment manager for Python

pydock - Docker-based environment manager for Python ⚠️ pydock is still in beta mode, and very unstable. It is not recommended for anything serious. p

Alejandro Piad 16 Sep 18, 2021