ETL flow framework based on Yaml configs in Python

Last update: Jul 06, 2022

Related tags

Overview

ETL framework based on Yaml configs in Python

A light framework for creating data streams. Setting up streams through configuration in the Yaml file. There is a schedule, task pools, concurrency limitation. Works quickly, does not require a lot of resources. Runs on Windows and Linux. Flow run in parallel via threading library. Internally SQLite Database. Native data transformation. There is a web interface.

At the moment there are connectors to sources

CSV file
SQLite
Postgres
MySQL
Yandex Metrika Management API
Yandex Metrika Stats API
Yandex Metrika Logs API
Yandex Direct API
Yandex Direct Report API
Criteo
Google Sheets

Storages

Save to csv file
Clickhouse

Documentation

Requirements

python >=3.9
virtual environment

Settings

It is highly recommended to install in a virtual environment.

Flowmaster needs a home, '{HOME}/FlowMaster' is the default,
but you can lay foundation somewhere else if you prefer
(optional)

For Windows

setx FLOWMASTER_HOME "{YOUR_PATH}"

For Linux

export FLOWMASTER_HOME={YOUR_PATH}

Installing

pip install flowmaster==0.7.1

# For install web UI.
pip install flowmaster[webui]==0.7.1

# Optional libraries.
pip install flowmaster[clickhouse,postgres,mysql,yandexdirect,yandexmetrika,criteo,googlesheets]==0.7.1

Run

flowmaster run --help
flowmaster run

WEB UI

http://localhost:8822

CHANGELOG

Support

Telegram support chat

Author

Pavel Maksimov

My contacts Telegram, Facebook

Удачи тебе, друг! Поставь звездочку ;)

You might also like...

signac-flow - manage workflows with signac

signac-flow - manage workflows with signac The signac framework helps users manage and scale file-based workflows, facilitating data reuse, sharing, a

44 Oct 14, 2022

Elementary is an open-source data reliability framework for modern data teams. The first module of the framework is data lineage.

Data lineage made simple, reliable, and automated. Effortlessly track the flow of data, understand dependencies and analyze impact. Features Visualiza

898 Jan 9, 2023

Randomisation-based inference in Python based on data resampling and permutation.

67 Dec 27, 2022

Karate Club: An API Oriented Open-source Python Framework for Unsupervised Learning on Graphs (CIKM 2020)

Karate Club is an unsupervised machine learning extension library for NetworkX. Please look at the Documentation, relevant Paper, Promo Video, and Ext

1.8k Jan 9, 2023

Tuplex is a parallel big data processing framework that runs data science pipelines written in Python at the speed of compiled code

Tuplex is a parallel big data processing framework that runs data science pipelines written in Python at the speed of compiled code. Tuplex has similar Python APIs to Apache Spark or Dask, but rather than invoking the Python interpreter, Tuplex generates optimized LLVM bytecode for the given pipeline and input data set.

791 Jan 4, 2023

BioMASS - A Python Framework for Modeling and Analysis of Signaling Systems

Mathematical modeling is a powerful method for the analysis of complex biological systems. Although there are many researches devoted on produ

22 Dec 27, 2022

PyChemia, Python Framework for Materials Discovery and Design

PyChemia, Python Framework for Materials Discovery and Design PyChemia is an open-source Python Library for materials structural search. The purpose o

61 Oct 2, 2022

wikirepo is a Python package that provides a framework to easily source and leverage standardized Wikidata information

Python based Wikidata framework for easy dataframe extraction wikirepo is a Python package that provides a framework to easily source and leverage sta

35 Jan 4, 2023

PLStream: A Framework for Fast Polarity Labelling of Massive Data Streams

PLStream: A Framework for Fast Polarity Labelling of Massive Data Streams Motivation When dataset freshness is critical, the annotating of high speed

4 Aug 2, 2022

Comments

No such file or directory: '/home/ubuntu/FlowMaster/pools.yaml'

Привет, очень хороший проект, однако столкнулся со следующей проблемой при устанвоке библиотеки

с ванильным python pip такого пакета вообще не видно
при установке через conda установка проходит замечательно, однако при запуске получаю

(base) [email protected]:~/FlowMaster$ flowmaster run
Traceback (most recent call last):
  File "/home/ubuntu/miniforge3/bin/flowmaster", line 5, in <module>
    from flowmaster.__main__ import app
  File "/home/ubuntu/miniforge3/lib/python3.9/site-packages/flowmaster/__main__.py", line 9, in <module>
    import flowmaster.cli.notebook
  File "/home/ubuntu/miniforge3/lib/python3.9/site-packages/flowmaster/cli/notebook.py", line 5, in <module>
    from flowmaster.service import (
  File "/home/ubuntu/miniforge3/lib/python3.9/site-packages/flowmaster/service.py", line 11, in <module>
    from flowmaster.operators.etl.policy import ETLNotebook
  File "/home/ubuntu/miniforge3/lib/python3.9/site-packages/flowmaster/operators/etl/__init__.py", line 3, in <module>
    from flowmaster.operators.etl.providers.abstract import ProviderAbstract, ExportAbstract
  File "/home/ubuntu/miniforge3/lib/python3.9/site-packages/flowmaster/operators/etl/providers/__init__.py", line 4, in <module>
    from flowmaster.operators.etl.providers.criteo import CriteoProvider
  File "/home/ubuntu/miniforge3/lib/python3.9/site-packages/flowmaster/operators/etl/providers/criteo/__init__.py", line 2, in <module>
    from flowmaster.operators.etl.providers.criteo.export import (
  File "/home/ubuntu/miniforge3/lib/python3.9/site-packages/flowmaster/operators/etl/providers/criteo/export.py", line 8, in <module>
    from flowmaster.executors import SleepIteration
  File "/home/ubuntu/miniforge3/lib/python3.9/site-packages/flowmaster/executors/__init__.py", line 16, in <module>
    from flowmaster.pool import pools
  File "/home/ubuntu/miniforge3/lib/python3.9/site-packages/flowmaster/pool.py", line 106, in <module>
    pools_dict = YamlHelper.parse_file(str(Settings.POOL_CONFIG_FILEPATH))
  File "/home/ubuntu/miniforge3/lib/python3.9/site-packages/flowmaster/utils/yaml_helper.py", line 14, in parse_file
    with open(path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/FlowMaster/pools.yaml'

Что я делаю не так?(

opened by micweeks 1

Releases(0.7.1)

0.7.1(Aug 29, 2021)
prevented planned of tasks from one instance of the operator class

fixed error GeneratorExit

fixed transform array type for Clickhouse loader

Source code(tar.gz)
Source code(zip)
0.6.1(Jun 22, 2021)
Redesigned executor

New

add politics 'time_limit_seconds_from_worktime', 'soft_time_limit_seconds'.

add provider 'flowmaster'

Fixing

fix schedule (interval seconds mode)

add logging 'loguru'

fix clear_statuses_of_lost_items

fix allow_execute_flow

change command 'db reset'

There are backward incompatible changes

new field 'expires_utc' in FlowItem

rename command 'run' to 'run_local' and rename command 'run_thread' to 'run'

add new class ExecutorIterationTask.

change, moving and rename class ThreadExecutor to ThreadAsyncExecutor.

change and rename class SleepTask to SleepIteration.

change and rename class TaskPool to NextIterationInPools.

ETLOperator return ExecutorIterationTask.

rename func order_flow to ordering_flow_tasks.

rename func start_executor to sync_executor.

rename field FlowItem.config_hash to FlowItem.notebook_hash

change FLOW_CONFIGS_DIR and rename FLOW_CONFIGS_DIR to NOTEBOOKS_DIR

rename objects config to notebook

add class Settings

Source code(tar.gz)
Source code(zip)
0.5.0(May 25, 2021)

Source code(tar.gz)
Source code(zip)
0.3.1(May 15, 2021)
There are backward incompatible changes

Add local executor

Fix Yandex Direct provider

Refactoring

Source code(tar.gz)
Source code(zip)
0.2.2(May 13, 2021)

Add provider Yandex Direct Refactoring

Incompatible changes
Source code(tar.gz)
Source code(zip)
0.1.3(May 2, 2021)

Source code(tar.gz)
Source code(zip)
0.1.0(May 1, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Павел Максимов

Python Data Engineer, Python Developer, ETL, Разработчик рекомендательных систем

GitHub Repository

Exploratory data analysis

Exploratory data analysis An Exploratory data analysis APP TAPIWA CHAMBOKO 🚀 About Me I'm a full stack developer experienced in deploying artificial

1 Nov 07, 2021

DefAP is a program developed to facilitate the exploration of a material's defect chemistry

DefAP is a program developed to facilitate the exploration of a material's defect chemistry. A large number of features are provided and rapid exploration is supported through the use of autoplotting

6 Oct 25, 2022

International Space Station data with Python research 🌎

International Space Station data with Python research 🌎 Plotting ISS trajectory, calculating the velocity over the earth and more. Plotting trajector

41 Jun 16, 2022

A notebook to analyze Amazon Recommendation Review Dataset.

Amazon Recommendation Review Dataset Analyzer A notebook to analyze Amazon Recommendation Review Dataset. Features Calculates distinct user count, dis

3 Aug 22, 2022

Finding project directories in Python (data science) projects, just like there R rprojroot and here packages

Find relative paths from a project root directory Finding project directories in Python (data science) projects, just like there R here and rprojroot

102 Nov 16, 2022

Display the behaviour of a realtime program with a scope or logic analyser.

1. A monitor for realtime MicroPython code This library provides a means of examining the behaviour of a running system. It was initially designed to

17 Dec 05, 2022

Developed for analyzing the covariance for OrcVIO

about This repo is developed for analyzing the covariance for OrcVIO environment setup platform ubuntu 18.04 using conda conda env create --file envir

1 Dec 08, 2021

A neural-based binary analysis tool

A neural-based binary analysis tool Introduction This directory contains the demo of a neural-based binary analysis tool. We test the framework using

208 Dec 22, 2022

X-news - Pipeline data use scrapy, kafka, spark streaming, spark ML and elasticsearch, Kibana

5 Sep 28, 2022

MoRecon - A tool for reconstructing missing frames in motion capture data.

38 Dec 03, 2022

Projeto para realizar o RPA Challenge . Utilizando Python e as bibliotecas Selenium e Pandas.

RPA Challenge in Python Projeto para realizar o RPA Challenge (www.rpachallenge.com), utilizando Python. O objetivo deste desafio é criar um fluxo de

1 Apr 12, 2022

Pandas and Dask test helper methods with beautiful error messages.

beavis Pandas and Dask test helper methods with beautiful error messages. test helpers These test helper methods are meant to be used in test suites.

18 Nov 28, 2022

PyEmits, a python package for easy manipulation in time-series data.

PyEmits, a python package for easy manipulation in time-series data. Time-series data is very common in real life. Engineering FSI industry (Financial

5 Sep 23, 2022

Automatic earthquake catalog building workflow: EQTransformer + Siamese EQTransformer + PickNet + REAL + HypoInverse

Automatic regional-scale earthquake catalog building workflow: EQTransformer + Siamese EQTransforme

9 Nov 27, 2022

Techdegree Data Analysis Project 2

Basketball Team Stats Tool In this project you will be writing a program that reads from the "constants" data (PLAYERS and TEAMS) in constants.py. Thi

2 Oct 23, 2021

Stitch together Nanopore tiled amplicon data without polishing a reference

Stitch together Nanopore tiled amplicon data using a reference guided approach Tiled amplicon data, like those produced from primers designed with pri

14 Aug 30, 2022

Example Of Splunk Search Query With Python And Splunk Python SDK

SSQAuto (Splunk Search Query Automation) Example Of Splunk Search Query With Python And Splunk Python SDK installation: ➜ ~ git clone https://github.c

1 Nov 14, 2021

Desafio proposto pela IGTI em seu bootcamp de Cloud Data Engineer

Desafio Modulo 4 - Cloud Data Engineer Bootcamp - IGTI Objetivos Criar infraestrutura como código Utuilizando um cluster Kubernetes na Azure Ingestão

4 Jan 23, 2022

DaCe is a parallel programming framework that takes code in Python/NumPy and other programming languages

aCe - Data-Centric Parallel Programming Decoupling domain science from performance optimization. DaCe is a parallel programming framework that takes c

330 Dec 30, 2022

A data structure that extends pyspark.sql.DataFrame with metadata information.

MetaFrame A data structure that extends pyspark.sql.DataFrame with metadata info

8 Feb 15, 2022