A python tool for synchronizing the messages from different threads, processes, or hosts.

Related tags

Miscellaneouspythonsynchronizationstreammultiprocessingpython-librarystdoutpython3stdout-redirection
Overview

Sync-stream

This project is designed for providing the synchoronization of the stdout / stderr among different threads, processes, devices or hosts. The package could be used for the following cases:

  1. Use syncstream.LineBuffer: Multiple threads are created. The messages (stdout) from different threads are required to be collected.
  2. Use syncstream.LineProcBuffer in the main process, and syncstream.LineProcMirror in the sub-processes: Multiple sub-processes are created on the same device. The stdout / stderr of each process is redirected to a LineProcMirror, and the results are collected by LineProcBuffer.
  3. Use syncstream.LineFileBuffer: Multiple processes are created. These processes may be deployed on different devices (even with different platforms), but all devices could get accessed to the same shared disk. In this case, the message could be shared by locked files. Each process would hold an independent LineFileBuffer pointing to the same log files.
  4. Use syncstream.LineHostBuffer on the server side, and syncstream.LineHostMirror on the client side: Multiple processes are deployed on different devices, and they could not share the same disk. In this case, the message are synchronized by the web service. Each process would hold a LineHostMirror, and the results would be collected by LineHostBuffer.

The basic package would not contain the file and host modules. To install the package, please use the following options:

pip install syncstream[option1,option2...]
Option Supports
file Install dependencies for the file module. The module provides syncstream.LineFileBuffer.
host Install dependencies for the host module. The module provides syncstream.LineHostBuffer, and syncstream.LineHostMirror.

Documentation

View the documentation here: 📘 https://cainmagi.github.io/sync-stream/

Update reports

0.3.3 @ 6/29/2021

  1. Fix small typos.
  2. Bump the dependencies to the newest versions.

0.3.2 @ 6/14/2021

  1. Fix a bug caused by stopping the mirrors.
  2. Format the meta-data defined in setup.py.
  3. Add the documentation. Currently only the tutorial is finished.

0.3.0 @ 6/4/2021

  1. Support the stop signal for mproc and host modules.
  2. Fix some bugs in the testing script.
  3. Fix typos.

0.2.2 @ 5/25/2021

  1. Add clear() methods for all buffers and mirrors.
  2. Fix typos in the package setup and info file.
  3. Fix a bug caused by writing data to the host in the testing scripts for Linux.

0.2.1 @ 5/24/2021

  1. Add the PyPI publish workflow.

0.2.0 @ 5/24/2021

  1. Finish the synchronization based on the file lock package fasteners.
  2. Finish the synchronization based on the web service packages flask, flask-restful and urllib3.
  3. Fix the compatibility of the testing scripts for py36, py37.

0.1.0 @ 5/22/2021

  1. Finish the synchronization based on the stdlib.
  2. Create this project.
You might also like...
Provide error messages for Python exceptions, even if the original message is empty

errortext is a Python package to provide error messages for Python exceptions, even if the original message is empty.

Thomas Aglassinger 0 Dec 7, 2021
Telegram bot to remove the forwarded tag from messages.
Telegram bot to remove the forwarded tag from messages.

Anonymous Sender Bot @AnonySendBot Telegram bot to remove the forwarded tag from messages. Table of Contents Usage Deploy To Heroku Local Deploying En

Stark Bots 26 Nov 24, 2022
Automatically remove user join messages when the user leaves the server.

CleanLeave Automatically remove user join messages when the user leaves the server. Installation You will need to install poetry to run this bot local

null 11 Sep 19, 2022
Simple script with AminoLab to send ghost messages

Simple script with AminoLab to send ghost messages

Moleey 1 Nov 22, 2021
A Github Action for sending messages to a Matrix Room.
A Github Action for sending messages to a Matrix Room.

matrix-commit A Github Action for sending messages to a Matrix Room. Screenshot: Example Usage: # .github/workflows/matrix-commit.yml on: push:

null 3 Sep 11, 2022
ViberExport - Export messages from Viber messenger using viber.db file

📲 ViberExport Export messages from Viber messenger using viber.db file ⚡ Usage:

null 7 Nov 23, 2022
Telop - Encode and decode messages using an interpretation of the telegraphic code devised by José María Mathé

telop Telop (TELégrafoÓPtico) - Utilidad para codificar y descodificar mensajes de texto empleando una interpretación del código telegráfico ideado po

Ricardo F. 4 Nov 1, 2022
Automatically unpin old messages so you can always pin more!

PinRotate Automatically unpin old messages so you can always pin more! Installation You will need to install poetry to run this bot locally for develo

null 3 Sep 18, 2022
A tool to flash .ofp files in bootloader mode without needing MSM Tool, an alternative to official realme tool
A tool to flash .ofp files in bootloader mode without needing MSM Tool, an alternative to official realme tool

Oppo/Realme Flash .OFP File on Bootloader A tool to flash .ofp files in bootloader mode without needing MSM Tool, an alternative to official realme to

Italo Almeida 70 Jan 2, 2023
Releases(0.3.3)

  • 0.3.3(Jun 29, 2021)

    Introduction

    Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.3.3 @ 6/29/2021

    1. Fix small typos.
    2. Bump the dependencies to the newest versions.
    Source code(tar.gz)
    Source code(zip)

  • 0.3.2(Jun 14, 2021)

    Introduction

    Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.3.2 @ 6/14/2021

    1. Fix a bug caused by stopping the mirrors.
    2. Format the meta-data defined in setup.py.
    3. Add the documentation. Currently, only the tutorial is finished.
    Source code(tar.gz)
    Source code(zip)

  • 0.3.0(Jun 4, 2021)

    Introduction

    Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.3.0 @ 6/4/2021

    1. Support the stop signal for mproc and host modules.
    2. Fix some bugs in the testing script.
    3. Fix typos.
    Source code(tar.gz)
    Source code(zip)

  • 0.2.2(May 25, 2021)

    Introduction

    Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.2.2 @ 5/25/2021

    1. Add clear() methods for all buffers and mirrors.
    2. Fix typos in the package setup and info file.
    3. Fix a bug caused by writing data to the host in the testing scripts for Linux.
    Source code(tar.gz)
    Source code(zip)

  • 0.2.1(May 24, 2021)

    Introduction

    This is the first release of this project. Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.2.1 @ 5/24/2021

    1. Add the PyPI publish workflow.

    0.2.0 @ 5/24/2021

    1. Finish the synchronization based on the file lock package fasteners.
    2. Finish the synchronization based on the web service packages flask, flask-restful and urllib3.
    3. Fix the compatibility of the testing scripts for py36, py37.

    0.1.0 @ 5/22/2021

    1. Finish the synchronization based on the stdlib.
    2. Create this project.
    Source code(tar.gz)
    Source code(zip)
Owner
Yuchen Jin
A Ph.D student in University of Houston (UH). Interested area includes: machine learning, programming and religion.
Yuchen Jin
GitHub Repository https://cainmagi.github.io/sync-stream/
《practical python programming》的中文翻译

欢迎光临 大约 25 年前，当我第一次学习 Python 时，发现 Python 竟然可以被高效地应用到各种混乱的工作项目上，我立即被震惊了。15 年前，我自己也将这种乐趣教授给别人。教学的结果就是本课程——一门实用的学习 Python的课程。

编程人 125 Dec 17, 2022
Demo content - Automate your automation!

Automate-AAP2 Demo Content - Automate your automation! A fully automated Ansible Automation Platform. Context Installing and configuring Ansible Autom

0 Oct 27, 2022
Companion Web site for Fluent Python, Second Edition

Fluent Python, the site Source code and content for fluentpython.com. The site complements Fluent Python, Second Edition with extra content that did n

Fluent Python 49 Dec 08, 2022
2021华为软件精英挑战赛 程序输出分析器

AutoGrader 0.2.0更新：加入资源分配溢出检测，如果发生资源溢出会输出溢出发生的位置。 如果通过检测，会显示通过符号 如果没有通过检测，会显示警告，并输出溢出发生的位置和操作

54 Aug 14, 2022
A program to calculate the are of a triangle. made with Python.

Area-Calculator What is Area-Calculator? Area-Calculator is a program to find out the area of a triangle easily. fully made with Python. Needed a pyth

Chandula Janith 0 Nov 27, 2021
Project for viewing the cheapest flight deals from Netherlands to other countries.

Flight_Deals_AMS Project for viewing the cheapest flight deals from Netherlands to other countries.

2 Dec 17, 2022
Another Provably Rare Gem Miner 💎 (for Raritygems)

Provably Rare Gem Miner Go (for Rarity) Pull Request is strongly welcome as I don't know anything about Golang/Python/Web3. Usage Install Python 3.x i

朱里 6 Apr 22, 2022
Repository for 2021 Computer Vision Class @ Chulalongkorn University

2110443 - Computer Vision (2021/2) Computer Vision @ Chulalongkorn University Anaconda Download Link https://www.anaconda.com/download/ Miniconda and

Chula PIC Lab 5 Jul 19, 2022
Implemented Exploratory Data Analysis (EDA) using Python.Built a dashboard in Tableau and found that 45.87% of People suffer from heart disease.

Heart_Disease_Diagnostic_Analysis Objective 🎯 The aim of this project is to use the given data and perform ETL and data analysis to infer key metrics

Sultan Shaikh 4 Jan 28, 2022
Build your own Etherscan with web3.py

Build your own Etherscan with web3.py Video Tutorial: Run it pip3 install -r requirements.txt export FLASK_APP=app export FLASK_ENV=development flask

35 Jan 02, 2023
Student Enrollment Analysis System

SEAS Student Enrollment Analysis System Steps to start working: create a user name "seas", host name: local, password: seas, mark all checkbox - go C

Md. Zakaria Kabir 3 Jul 12, 2022
A simple but fully functional calculator that will take multiple operations.

Functional-Calculator A simple but fully functional calculator that will take multiple operations. Usage Run the following command through terminal: p

Uzziel Ariel 1 Dec 22, 2022
Small tool to use hero .json files created with Optolith for The Dark Eye/ Das Schwarze Auge 5 to perform talent probes.

DSA5-ProbeMaker A little tool for The Dark Eye 5th Edition (Das Schwarze Auge 5) to load .json from Optolith character generation and easily perform t

2 Jan 06, 2022
Scientific color maps and standardization tools

Scicomap is a package that provides scientific color maps and tools to standardize your favourite color maps if you don't like the built-in ones. Scicomap currently provides sequential, bi-sequential

Thomas Bury 14 Nov 30, 2022
A Red Team tool for exfiltrating sensitive data from Jira tickets.

Jir-thief This Module will connect to Jira's API using an access token, export to a word .doc, and download the Jira issues that the target has access

Antonio Piazza 82 Dec 12, 2022
A collection of full-stack resources for programmers.

A collection of full-stack resources for programmers.

Charles-Axel Dein 22.3k Dec 30, 2022
Remote Worker

Remote Worker Separation of Responsibilities There are several reasons to move some processing out of the main code base for security or performance:

V2EX 69 Dec 05, 2022
JLC2KICAD_lib is a python script that generate a component library for KiCad from the JLCPCB/easyEDA library.

JLC2KiCad_lib is a python script that generate a component library (schematic, footprint and 3D model) for KiCad from the JLCPCB/easyEDA library. This script requires Python 3.6 or higher.

Nicolas Toussaint 73 Dec 26, 2022
A community based economy bot with python works only with python 3.7.8 as web3 requires cytoolz

A community based economy bot with python works only with python 3.7.8 as web3 requires cytoolz has some issues building with python 3.10

4 Jan 01, 2022
Decipher using Markov Chain Monte Carlo

Decipher using Markov Chain Monte Carlo

Science étonnante 43 Dec 24, 2022