Simple and flexible ML workflow engine.

Overview

Katana ML Skipper

PyPI - Python GitHub Stars GitHub Issues Current Version

This is a simple and flexible ML workflow engine. It helps to orchestrate events across a set of microservices and create executable flow to handle requests. Engine is designed to be configurable with any microservices. Enjoy!

Skipper

Author

Katana ML, Andrej Baranovskij

Instructions

Start/Stop

Docker Compose

Start:

docker-compose up --build -d

Stop:

docker-compose down

This will start RabbitMQ container. To run engine and services, navigate to related folders and follow instructions.

Web API FastAPI endpoint:

http://127.0.0.1:8080/api/v1/skipper/tasks/docs

Kubernetes

NGINX Ingress Controller:

If you are using local Kubernetes setup, install NGINX Ingress Controller

Build Docker images:

docker-compose -f docker-compose-kubernetes.yml build

Setup Kubernetes services:

./kubectl-setup.sh

Skipper API endpoint published through NGINX Ingress (you can setup your own host in /etc/hosts):

http://kubernetes.docker.internal/api/v1/skipper/tasks/docs

Check NGINX Ingress Controller pod name:

kubectl get pods -n ingress-nginx

Sample response, copy the name of 'Running' pod:

NAME                                       READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-dhtcm       0/1     Completed   0          14m
ingress-nginx-admission-patch-x8zvw        0/1     Completed   0          14m
ingress-nginx-controller-fd7bb8d66-tnb9t   1/1     Running     0          14m

NGINX Ingress Controller logs:

kubectl logs -n ingress-nginx -f 
   

   

Skipper API logs:

kubectl logs -n katana-skipper -f -l app=skipper-api

Remove Kubernetes services:

./kubectl-remove.sh

Components

  • api - Web API implementation
  • workflow - workflow logic
  • services - a set of sample microservices, you should replace this with your own services. Update references in docker-compose.yml
  • rabbitmq - service for RabbitMQ broker
  • skipper-lib - reusable Python library to streamline event communication through RabbitMQ
  • logger - logger service

URLs

  • Web API
http://127.0.0.1:8080/api/v1/skipper/tasks/docs

If running on local Kubernetes with Docker Desktop:

http://kubernetes.docker.internal/api/v1/skipper/tasks/docs
  • RabbitMQ:
http://localhost:15672/ (skipper/welcome1)

If running on local Kubernets, make sure port forwarding is enabled:

kubectl -n rabbits port-forward rabbitmq-0 15672:15672
  • PyPI
https://pypi.org/project/skipper-lib/
  • OCI - deployment guide for Oracle Cloud

Usage

You can use Skipper engine to run Web API, workflow and communicate with a group of ML microservices implemented under services package.

Skipper can be deployed to any Cloud vendor with Kubernetes or Docker support. You can scale Skipper runtime on Cloud using Kubernetes commands.

License

Licensed under the Apache License, Version 2.0. Copyright 2020-2021 Katana ML, Andrej Baranovskij. Copy of the license.

Comments
  • Cache EventProducer

    Cache EventProducer

    I found that cache the EventProducer can improve performace 40%. I tried but it block may request when increase the speed test. Do you have suggest to fix that

    opened by manhtd98 7
  • Docker-compose up not working

    Docker-compose up not working

    Hi

    Thank you for the wonderful katana-skipper. I am trying to digest the library and execute the docker-compose.yml. But it seems like it is not working.

    Would appreciate it if you could take a look

    good first issue 
    opened by jamesee 6
  • Doc: How to add a new service with a new queue

    Doc: How to add a new service with a new queue

    How do we add a new service with a new queue called translator?

    1. I add a new router adding a new path for my new service defining a new prefix and tag named translator.
    2. I create a new request model for my new service in models.py containing task_type and expect a type translator and a payload
    3. I define a new service container with the correct variables and set my SERVICE=translator and QUEUE_NAME=skipper_translator

    I am able to call the new endpoint and it returns:

    task_id: "-", 
    task_status: "Success", 
    outcome: "<starlette.responses.JSONResponse object at 0x7ff2672dbed0>"
    

    However the container is never triggered.

    What am I missing?

    opened by ladrua 4
  • The difference between event_producer and exchange_producer

    The difference between event_producer and exchange_producer

    Hello, Thanks for sharing your ML workflow. I appreciate if you could explain the difference between event_producer and exchange_producer. event_producer is used to produce an event to rabbitmq, but exchange_producer is not clear to me. Can't we use event_producer in place of exchange_producer?

    good first issue 
    opened by fadishaar84 4
  • Encountering Authentication Issues

    Encountering Authentication Issues

    When I run the start command on docker I get the following error in the data-service container. Would greatly appreciate guidance on how to fix this issue. ` data-service katanaml/data-service RUNNING

    Traceback (most recent call last):

    File "main.py", line 19, in

    main()
    

    File "main.py", line 15, in main

    'http://127.0.0.1:5001/api/v1/skipper/logger/log_receiver'))
    

    File "/usr/local/lib/python3.7/site-packages/skipper_lib/events/event_receiver.py", line 16, in init

    credentials=credentials))
    

    File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 360, in init

    self._impl = self._create_connection(parameters, _impl_class)
    

    File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection

    raise self._reap_last_connection_workflow_error(error)
    

    pika.exceptions.AMQPConnectionError

    Traceback (most recent call last):

    File "main.py", line 19, in

    main()
    

    File "main.py", line 15, in main

    'http://127.0.0.1:5001/api/v1/skipper/logger/log_receiver'))
    

    File "/usr/local/lib/python3.7/site-packages/skipper_lib/events/event_receiver.py", line 16, in init

    credentials=credentials))
    

    File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 360, in init

    self._impl = self._create_connection(parameters, _impl_class)
    

    File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection

    raise self._reap_last_connection_workflow_error(error)
    

    pika.exceptions.ProbableAuthenticationError: ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfi`

    opened by LM-01 3
  • How can we move from docker compose to kubernetes?

    How can we move from docker compose to kubernetes?

    Hello Andrej, I would like to ask about how to move from docker-compose to Kubernetes, do we have to use some tools like kompose or other tools, I appreciate if you could guide me a little bit about how to perform this conversion to run our services on Skipper not using docker compose but kubernetes. Thank you.

    opened by fadishaar84 2
Releases(v1.1.0)
  • v1.1.0(Dec 11, 2021)

    This release of Katana ML Skipper includes:

    • Skipper Lib JS - support for Node.js containers
    • Error handling
    • Configurable FastAPI endpoints
    • Various improvements and bug fixes

    What's Changed

    • (README.md) Adding Andrej's profile url by @xandrade in https://github.com/katanaml/katana-skipper/pull/3

    New Contributors

    • @xandrade made their first contribution in https://github.com/katanaml/katana-skipper/pull/3

    Full Changelog: https://github.com/katanaml/katana-skipper/compare/v1.0.0...v1.1.0

    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Oct 9, 2021)

    First production release of Katana ML Skipper.

    Included:

    • Logger
    • Workflow
    • API async and sync
    • Services
    • Docker support
    • Kubernetes support
    • Tested on OCI Cloud

    Full Changelog: https://github.com/katanaml/katana-skipper/commits/v1.0.0

    Source code(tar.gz)
    Source code(zip)
Owner
Katana ML
Machine Learning for Business Automation
Katana ML
This is a Machine Learning model which predicts the presence of Diabetes in Patients

Diabetes Disease Prediction This is a machine Learning mode which tries to determine if a person has a diabetes or not. Data The dataset is in comma s

Edem Gold 4 Mar 16, 2022
Automated Machine Learning with scikit-learn

auto-sklearn auto-sklearn is an automated machine learning toolkit and a drop-in replacement for a scikit-learn estimator. Find the documentation here

AutoML-Freiburg-Hannover 6.7k Jan 07, 2023
Backtesting an algorithmic trading strategy using Machine Learning and Sentiment Analysis.

Trading Tesla with Machine Learning and Sentiment Analysis An interactive program to train a Random Forest Classifier to predict Tesla daily prices us

Renato Votto 31 Nov 17, 2022
My capstone project for Udacity's Machine Learning Nanodegree

MLND-Capstone My capstone project for Udacity's Machine Learning Nanodegree Lane Detection with Deep Learning In this project, I use a deep learning-b

Michael Virgo 407 Dec 12, 2022
CrayLabs and user contibuted examples of using SmartSim for various simulation and machine learning applications.

SmartSim Example Zoo This repository contains CrayLabs and user contibuted examples of using SmartSim for various simulation and machine learning appl

Cray Labs 14 Mar 30, 2022
Python Automated Machine Learning library for tabular data.

Simple but powerful Automated Machine Learning library for tabular data. It uses efficient in-memory SAP HANA algorithms to automate routine Data Scie

Daniel Khromov 47 Dec 17, 2022
A Python package for time series classification

pyts: a Python package for time series classification pyts is a Python package for time series classification. It aims to make time series classificat

Johann Faouzi 1.4k Jan 01, 2023
Model search (MS) is a framework that implements AutoML algorithms for model architecture search at scale.

Model Search Model search (MS) is a framework that implements AutoML algorithms for model architecture search at scale. It aims to help researchers sp

AriesTriputranto 1 Dec 13, 2021
Little Ball of Fur - A graph sampling extension library for NetworKit and NetworkX (CIKM 2020)

Little Ball of Fur is a graph sampling extension library for Python. Please look at the Documentation, relevant Paper, Promo video and External Resour

Benedek Rozemberczki 619 Dec 14, 2022
A Python library for detecting patterns and anomalies in massive datasets using the Matrix Profile

matrixprofile-ts matrixprofile-ts is a Python 2 and 3 library for evaluating time series data using the Matrix Profile algorithms developed by the Keo

Target 696 Dec 26, 2022
DirectML is a high-performance, hardware-accelerated DirectX 12 library for machine learning.

DirectML is a high-performance, hardware-accelerated DirectX 12 library for machine learning. DirectML provides GPU acceleration for common machine learning tasks across a broad range of supported ha

Microsoft 1.1k Jan 04, 2023
Tangram makes it easy for programmers to train, deploy, and monitor machine learning models.

Tangram Website | Discord Tangram makes it easy for programmers to train, deploy, and monitor machine learning models. Run tangram train to train a mo

Tangram 1.4k Jan 05, 2023
Python Research Framework

Python Research Framework

EleutherAI 106 Dec 13, 2022
A simple application that calculates the probability distribution of a normal distribution

probability-density-function General info An application that calculates the probability density and cumulative distribution of a normal distribution

1 Oct 25, 2022
LILLIE: Information Extraction and Database Integration Using Linguistics and Learning-Based Algorithms

LILLIE: Information Extraction and Database Integration Using Linguistics and Learning-Based Algorithms Based on the work by Smith et al. (2021) Query

5 Aug 06, 2022
Turning images into '9-pan' palettes using KMeans clustering from sklearn.

img2palette Turning images into '9-pan' palettes using KMeans clustering from sklearn. Requirements We require: Pillow, for opening and processing ima

Samuel Vidovich 2 Jan 01, 2022
Python package for causal inference using Bayesian structural time-series models.

Python Causal Impact Causal inference using Bayesian structural time-series models. This package aims at defining a python equivalent of the R CausalI

Thomas Cassou 219 Dec 11, 2022
MBTR is a python package for multivariate boosted tree regressors trained in parameter space.

MBTR is a python package for multivariate boosted tree regressors trained in parameter space.

SUPSI-DACD-ISAAC 61 Dec 19, 2022
Diabetes Prediction with Logistic Regression

Diabetes Prediction with Logistic Regression Exploratory Data Analysis Data Preprocessing Model & Prediction Model Evaluation Model Validation: Holdou

AZİZE SULTAN PALALI 2 Oct 23, 2021
LinearRegression2 Tvads and CarSales

LinearRegression2_Tvads_and_CarSales This project infers the insight that how the TV ads for cars and car Sales are being linked with each other. It i

Ashish Kumar Yadav 1 Dec 29, 2021