Simple and flexible ML workflow engine.

Overview

Katana ML Skipper

PyPI - Python GitHub Stars GitHub Issues Current Version

This is a simple and flexible ML workflow engine. It helps to orchestrate events across a set of microservices and create executable flow to handle requests. Engine is designed to be configurable with any microservices. Enjoy!

Skipper

Author

Katana ML, Andrej Baranovskij

Instructions

Start/Stop

Docker Compose

Start:

docker-compose up --build -d

Stop:

docker-compose down

This will start RabbitMQ container. To run engine and services, navigate to related folders and follow instructions.

Web API FastAPI endpoint:

http://127.0.0.1:8080/api/v1/skipper/tasks/docs

Kubernetes

NGINX Ingress Controller:

If you are using local Kubernetes setup, install NGINX Ingress Controller

Build Docker images:

docker-compose -f docker-compose-kubernetes.yml build

Setup Kubernetes services:

./kubectl-setup.sh

Skipper API endpoint published through NGINX Ingress (you can setup your own host in /etc/hosts):

http://kubernetes.docker.internal/api/v1/skipper/tasks/docs

Check NGINX Ingress Controller pod name:

kubectl get pods -n ingress-nginx

Sample response, copy the name of 'Running' pod:

NAME                                       READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-dhtcm       0/1     Completed   0          14m
ingress-nginx-admission-patch-x8zvw        0/1     Completed   0          14m
ingress-nginx-controller-fd7bb8d66-tnb9t   1/1     Running     0          14m

NGINX Ingress Controller logs:

kubectl logs -n ingress-nginx -f 
   

   

Skipper API logs:

kubectl logs -n katana-skipper -f -l app=skipper-api

Remove Kubernetes services:

./kubectl-remove.sh

Components

  • api - Web API implementation
  • workflow - workflow logic
  • services - a set of sample microservices, you should replace this with your own services. Update references in docker-compose.yml
  • rabbitmq - service for RabbitMQ broker
  • skipper-lib - reusable Python library to streamline event communication through RabbitMQ
  • logger - logger service

URLs

  • Web API
http://127.0.0.1:8080/api/v1/skipper/tasks/docs

If running on local Kubernetes with Docker Desktop:

http://kubernetes.docker.internal/api/v1/skipper/tasks/docs
  • RabbitMQ:
http://localhost:15672/ (skipper/welcome1)

If running on local Kubernets, make sure port forwarding is enabled:

kubectl -n rabbits port-forward rabbitmq-0 15672:15672
  • PyPI
https://pypi.org/project/skipper-lib/
  • OCI - deployment guide for Oracle Cloud

Usage

You can use Skipper engine to run Web API, workflow and communicate with a group of ML microservices implemented under services package.

Skipper can be deployed to any Cloud vendor with Kubernetes or Docker support. You can scale Skipper runtime on Cloud using Kubernetes commands.

License

Licensed under the Apache License, Version 2.0. Copyright 2020-2021 Katana ML, Andrej Baranovskij. Copy of the license.

Comments
  • Cache EventProducer

    Cache EventProducer

    I found that cache the EventProducer can improve performace 40%. I tried but it block may request when increase the speed test. Do you have suggest to fix that

    opened by manhtd98 7
  • Docker-compose up not working

    Docker-compose up not working

    Hi

    Thank you for the wonderful katana-skipper. I am trying to digest the library and execute the docker-compose.yml. But it seems like it is not working.

    Would appreciate it if you could take a look

    good first issue 
    opened by jamesee 6
  • Doc: How to add a new service with a new queue

    Doc: How to add a new service with a new queue

    How do we add a new service with a new queue called translator?

    1. I add a new router adding a new path for my new service defining a new prefix and tag named translator.
    2. I create a new request model for my new service in models.py containing task_type and expect a type translator and a payload
    3. I define a new service container with the correct variables and set my SERVICE=translator and QUEUE_NAME=skipper_translator

    I am able to call the new endpoint and it returns:

    task_id: "-", 
    task_status: "Success", 
    outcome: "<starlette.responses.JSONResponse object at 0x7ff2672dbed0>"
    

    However the container is never triggered.

    What am I missing?

    opened by ladrua 4
  • The difference between event_producer and exchange_producer

    The difference between event_producer and exchange_producer

    Hello, Thanks for sharing your ML workflow. I appreciate if you could explain the difference between event_producer and exchange_producer. event_producer is used to produce an event to rabbitmq, but exchange_producer is not clear to me. Can't we use event_producer in place of exchange_producer?

    good first issue 
    opened by fadishaar84 4
  • Encountering Authentication Issues

    Encountering Authentication Issues

    When I run the start command on docker I get the following error in the data-service container. Would greatly appreciate guidance on how to fix this issue. ` data-service katanaml/data-service RUNNING

    Traceback (most recent call last):

    File "main.py", line 19, in

    main()
    

    File "main.py", line 15, in main

    'http://127.0.0.1:5001/api/v1/skipper/logger/log_receiver'))
    

    File "/usr/local/lib/python3.7/site-packages/skipper_lib/events/event_receiver.py", line 16, in init

    credentials=credentials))
    

    File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 360, in init

    self._impl = self._create_connection(parameters, _impl_class)
    

    File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection

    raise self._reap_last_connection_workflow_error(error)
    

    pika.exceptions.AMQPConnectionError

    Traceback (most recent call last):

    File "main.py", line 19, in

    main()
    

    File "main.py", line 15, in main

    'http://127.0.0.1:5001/api/v1/skipper/logger/log_receiver'))
    

    File "/usr/local/lib/python3.7/site-packages/skipper_lib/events/event_receiver.py", line 16, in init

    credentials=credentials))
    

    File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 360, in init

    self._impl = self._create_connection(parameters, _impl_class)
    

    File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection

    raise self._reap_last_connection_workflow_error(error)
    

    pika.exceptions.ProbableAuthenticationError: ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfi`

    opened by LM-01 3
  • How can we move from docker compose to kubernetes?

    How can we move from docker compose to kubernetes?

    Hello Andrej, I would like to ask about how to move from docker-compose to Kubernetes, do we have to use some tools like kompose or other tools, I appreciate if you could guide me a little bit about how to perform this conversion to run our services on Skipper not using docker compose but kubernetes. Thank you.

    opened by fadishaar84 2
Releases(v1.1.0)
  • v1.1.0(Dec 11, 2021)

    This release of Katana ML Skipper includes:

    • Skipper Lib JS - support for Node.js containers
    • Error handling
    • Configurable FastAPI endpoints
    • Various improvements and bug fixes

    What's Changed

    • (README.md) Adding Andrej's profile url by @xandrade in https://github.com/katanaml/katana-skipper/pull/3

    New Contributors

    • @xandrade made their first contribution in https://github.com/katanaml/katana-skipper/pull/3

    Full Changelog: https://github.com/katanaml/katana-skipper/compare/v1.0.0...v1.1.0

    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Oct 9, 2021)

    First production release of Katana ML Skipper.

    Included:

    • Logger
    • Workflow
    • API async and sync
    • Services
    • Docker support
    • Kubernetes support
    • Tested on OCI Cloud

    Full Changelog: https://github.com/katanaml/katana-skipper/commits/v1.0.0

    Source code(tar.gz)
    Source code(zip)
Owner
Katana ML
Machine Learning for Business Automation
Katana ML
Applied Machine Learning for Graduate Program in Computer Science (PPGCC)

Applied Machine Learning for Graduate Program in Computer Science (PPGCC) - Federal University of Santa Catarina

Jônatas Negri Grandini 1 Dec 22, 2021
Pragmatic AI Labs 421 Dec 31, 2022
slim-python is a package to learn customized scoring systems for decision-making problems.

slim-python is a package to learn customized scoring systems for decision-making problems. These are simple decision aids that let users make yes-no p

Berk Ustun 37 Nov 02, 2022
Simplify stop motion animation with machine learning.

Simplify stop motion animation with machine learning.

Nick Bild 25 Sep 15, 2022
This repo includes some graph-based CTR prediction models and other representative baselines.

Graph-based CTR prediction This is a repository designed for graph-based CTR prediction methods, it includes our graph-based CTR prediction methods: F

Big Data and Multi-modal Computing Group, CRIPAC 47 Dec 30, 2022
A statistical library designed to fill the void in Python's time series analysis capabilities, including the equivalent of R's auto.arima function.

pmdarima Pmdarima (originally pyramid-arima, for the anagram of 'py' + 'arima') is a statistical library designed to fill the void in Python's time se

alkaline-ml 1.3k Jan 06, 2023
Greykite: A flexible, intuitive and fast forecasting library

The Greykite library provides flexible, intuitive and fast forecasts through its flagship algorithm, Silverkite.

LinkedIn 1.7k Jan 04, 2023
Automatically build ARIMA, SARIMAX, VAR, FB Prophet and XGBoost Models on Time Series data sets with a Single Line of Code. Now updated with Dask to handle millions of rows.

Auto_TS: Auto_TimeSeries Automatically build multiple Time Series models using a Single Line of Code. Now updated with Dask. Auto_timeseries is a comp

AutoViz and Auto_ViML 519 Jan 03, 2023
This repo implements a Topological SLAM: Deep Visual Odometry with Long Term Place Recognition (Loop Closure Detection)

This repo implements a topological SLAM system. Deep Visual Odometry (DF-VO) and Visual Place Recognition are combined to form the topological SLAM system.

Best of Australian Centre for Robotic Vision (ACRV) 32 Jun 23, 2022
Python bindings for MPI

MPI for Python Overview Welcome to MPI for Python. This package provides Python bindings for the Message Passing Interface (MPI) standard. It is imple

MPI for Python 604 Dec 29, 2022
learn python in 100 days, a simple step could be follow from beginner to master of every aspect of python programming and project also include side project which you can use as demo project for your personal portfolio

learn python in 100 days, a simple step could be follow from beginner to master of every aspect of python programming and project also include side project which you can use as demo project for your

BDFD 6 Nov 05, 2022
Toolkit for building machine learning models that generalize to unseen domains and are robust to privacy and other attacks.

Toolkit for Building Robust ML models that generalize to unseen domains (RobustDG) Divyat Mahajan, Shruti Tople, Amit Sharma Privacy & Causal Learning

Microsoft 149 Jan 06, 2023
Machine-care - A simple python script to take care of simple maintenance tasks

Machine care An simple python script to take care of simple maintenance tasks fo

2 Jul 10, 2022
A library of sklearn compatible categorical variable encoders

Categorical Encoding Methods A set of scikit-learn-style transformers for encoding categorical variables into numeric by means of different techniques

2.1k Jan 07, 2023
Neighbourhood Retrieval (Nearest Neighbours) with Distance Correlation.

Neighbourhood Retrieval with Distance Correlation Assign Pseudo class labels to datapoints in the latent space. NNDC is a slim wrapper around FAISS. N

The Learning Machines 1 Jan 16, 2022
Evaluate on three different ML model for feature selection using Breast cancer data.

Anomaly-detection-Feature-Selection Evaluate on three different ML model for feature selection using Breast cancer data. ML models: SVM, KNN and MLP.

Tarek idrees 1 Mar 17, 2022
Machine Learning from Scratch

Machine Learning from Scratch Author: Shengxuan Wang From: Oregon State University Content: Building Machine Learning model from Scratch, without usin

ShawnWang 0 Jul 05, 2022
A GitHub action that suggests type annotations for Python using machine learning.

Typilus: Suggest Python Type Annotations A GitHub action that suggests type annotations for Python using machine learning. This action makes suggestio

40 Sep 18, 2022
Projeto: Machine Learning: Linguagens de Programacao 2004-2001

Projeto: Machine Learning: Linguagens de Programacao 2004-2001 Projeto de Data Science e Machine Learning de análise de linguagens de programação de 2

Victor Hugo Negrisoli 0 Jun 29, 2021
Machine learning algorithms implementation

Machine learning algorithms implementation This repository consisits of implementation of various machine learning algorithms. The algorithms implemen

Karun Dawadi 1 Jan 03, 2022