The python SDK for Eto, the AI focused data platform for teams bringing AI models to production

Overview

Eto Labs Python SDK

This is the python SDK for Eto, the AI focused data platform for teams bringing AI models to production. The python SDK makes it easy to integrate Eto's features into your AI training and analysis workflow.

Installation

The Eto python SDK is available on PyPI and can be installed via Pip:

pip install etosdk

Eto SDK is compatible with Python 3.7+

Setup

Before using the SDK for the first time, you must configure it with your Eto API url and the API token.

import eto
eto.configure(url='<eto-api-url>', token='<api-token>')

The above configuration function creates a configuration file under $XDG_CONFIG_HOME/eto/eto.conf, which is usually ~/.config/eto/eto.conf.

Ingesting data

To create an ingestion job to convert raw data in Coco format and create a new dataset:

import eto
job = eto.ingest_coco('<dataset_name>',
                      {'image_dir': '<path/to/images>',
                       'annotations': '<path/to/annotations>',
                       'extras': {'key': 'value'}})

The ingestion job will run asynchronously server-side and convert the data to Rikai (parquet) format. Once complete, you should be able to see it in the data registry:

import eto

eto.list_datasets() # list all datasets

eto.get_dataset('<dataset_name>') # get information about a single dataset

Analysis

Accessing a particular dataset is easy via Pandas:

import eto
import pandas as pd

df = pd.read_eto('<dataset_name>') # Eto SDK adds a pandas extension

Training

To train a pytorch model, you can use the Dataset/DataLoader classes in Rikai:

import eto
from rikai.torch.vision import Dataset

dataset = Dataset('<dataset_name>') # Eto SDK adds an extension to Rikai to resolve dataset references 

for next_record in dataset:
    # training loop
    pass

A plain pytorch dataloader is also available from rikai.torch.data.DataLoader.

Local Spark configuration

For now, the Eto SDK relies on PySpark locally to read some of the custom Rikai types like annotations. While PySpark should be automatically installed as a transitive dependency, you may find that you need to change the Spark configurations to suit your local setup.

Your $SPARK_HOME/conf/spark-defaults.conf file should look something like the following:

spark.sql.extensions               ai.eto.rikai.sql.spark.RikaiSparkSessionExtensions
spark.jars.packages                ai.eto:rikai_2.12:0.0.13,org.apache.hadoop:hadoop-aws:3.2.0

# AWS
spark.executor.extraJavaOptions -Dcom.amazonaws.services.s3.enableV4=true -Dio.netty.tryReflectionSetAccessible=true
spark.driver.extraJavaOptions -Dcom.amazonaws.services.s3.enableV4=true -Dio.netty.tryReflectionSetAccessible=true
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version 2
spark.hadoop.com.amazonaws.services.s3.enableV4 true
fs.AbstractFileSystem.s3a.impl org.apache.hadoop.fs.s3a.S3A
fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem
fs.s3a.aws.credentials.provider com.amazonaws.auth.InstanceProfileCredentialsProvider,com.amazonaws.auth.DefaultAWSCredentialsProviderChain
You might also like...
BeeDrive: Open Source Privacy File Transfering System for Teams and Individual Developers
BeeDrive: Open Source Privacy File Transfering System for Teams and Individual Developers

BeeDrive For privacy and convenience purposes, more and more people try to keep data on their own hardwires instead of third-party cloud services such

Automatic login to Microsoft Teams conferences

Automatic login to Microsoft Teams conferences

To dynamically change the split direction in I3/Sway so as to split new windows automatically based on the width and height of the focused window
To dynamically change the split direction in I3/Sway so as to split new windows automatically based on the width and height of the focused window

To dynamically change the split direction in I3/Sway so as to split new windows automatically based on the width and height of the focused window Insp

This repository will be a draft of a package about the latest total marine fish production in Indonesia. Data will be collected from PIPP (Pusat Informasi Pelabuhan Perikanan).

indomarinefish This package will give us information about the latest total marine fish production in Indonesia. The Name of the fish is written in In

A wrapper for aqquiring Choice Coin directly through a Python Terminal. Leverages the TinyMan Python-SDK.

CHOICE_TinyMan_Wrapper A wrapper that allows users to acquire Choice Coin directly through their Terminal using ALGO and various Algorand Standard Ass

AWS SDK for Python

Boto3 - The AWS SDK for Python Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python, which allows Python developers to wri

Python SDK for Facebook's Graph API

Facebook Python SDK This client library is designed to support the Facebook Graph API and the official Facebook JavaScript SDK, which is the canonical

Box SDK for Python

Box Python SDK Installing Getting Started Authorization Server-to-Server Auth with JWT Traditional 3-legged OAuth2 Other Auth Options Usage Documentat

The Official Dropbox API V2 SDK for Python
The Official Dropbox API V2 SDK for Python

The offical Dropbox SDK for Python. Documentation can be found on Read The Docs. Installation Create an app via the Developer Console. Install via pip

Releases(v0.2)
  • v0.2(Dec 23, 2021)

    1. Access the Eto dataset registry API
    2. Submit Coco ingestion jobs
    3. Read into pandas dataframes and pytorch datasets/dataloaders

    What's Changed

    • Changhiskhan/sdk fixes by @changhiskhan in https://github.com/eto-ai/etosdk/pull/3
    • Changhiskhan/normalize uri by @changhiskhan in https://github.com/eto-ai/etosdk/pull/4
    • user specifies account name and SDK will formulate the url automatically by @changhiskhan in https://github.com/eto-ai/etosdk/pull/5
    • Sample notebook by @changhiskhan in https://github.com/eto-ai/etosdk/pull/6
    • minor notebook fix to wrap up M2 by @changhiskhan in https://github.com/eto-ai/etosdk/pull/7

    Full Changelog: https://github.com/eto-ai/etosdk/commits/v0.2

    Source code(tar.gz)
    Source code(zip)
Isobot is originally made by notsniped. This is a remix of iso.bot by archisha.

iso6.9-1.2beta iso.bot is originally made by notsniped#0002. This is a remix of iso.bot by αrchιshα#5518. iso6.9 is a Discord bot written in Python an

Kamilla Youver 3 Jan 11, 2022
DeKrypt 24 Sep 21, 2022
A Python API to retrieve and read MLB GameDay data

mlbgame mlbgame is a Python API to retrieve and read MLB GameDay data. mlbgame works with real time data, getting information as games are being playe

Zach Panzarino 493 Dec 13, 2022
a discord bot for searching your movies, and bot return movie url for you :)

IMDb Discord Bot how to run this bot. the first step you must create prefixes.json file the second step you must create a virtualenv if you use window

Mehdi Radfar 6 Dec 20, 2022
基于nonebot2开发的群管机器人qbot,支持上传并运行python代码以及一些基础管理功能

nonebot2-Eleina 基于nonebot2开发的群管机器人qbot,支持上传并运行python代码以及一些基础管理功能 Readme 环境:python3.7.3+,go-cqhttp 安装及配置:参见(https://v2.nonebot.dev/guide/installation.h

1 Dec 06, 2022
A telegram bot which can show you the status of telegram bot

BotStatus-Ts-Bot An open source telegram Bot Status bot For demo you can check here The status is updated in every 1 hour About Bot This is a Bot stat

Ts_Bots 8 Nov 17, 2022
Osmopy - osmo python client library

osmopy Version 0.0.2 Tools for Osmosis wallet management and offline transaction

5 May 22, 2022
Unofficial Python wrapper for official Hacker News API

haxor Unofficial Python wrapper for official Hacker News API. Installation pip install haxor Usage Import and initialization: from hackernews import H

147 Sep 18, 2022
A Telegram bot to extracting text from images. All languages supported.

OCR Bot A Telegram bot to extracting text from images. All languages supported. Deploy to Heroku Local Deploying Clone the repo git clone https://gith

6 Oct 21, 2022
Tglogging - A python package to send your app logs to a telegram chat in realtime

Telegram Logger A simple python package to send your app logs to a telegram chat

SUBIN 60 Dec 27, 2022
Official python API for Phish.AI public and private API to detect zero-day phishing websites

phish-ai-api Summary Official python API for Phish.AI public and private API to detect zero-day phishing websites How it Works (TLDR) Essentially we h

Phish.AI 168 May 17, 2022
Project to list all resources in an AWS account with tags.

AWS-ListAll Project to list all resources in an AWS account with tags. This script works on any system Get started: Install python3 and pip3 along wit

Connor Shubham Verlekar 3 Jan 30, 2022
A collective list of free APIs for use in software and web development.

Public APIs A collective list of free APIs for use in software and web development. A public API for this project can be found here! For information o

222.5k Jan 02, 2023
❄️ Don't waste your money paying for new tokens, once you have used your tokens, clean them up and resell them!

TokenCleaner Don't waste your money paying for new tokens, once you have used your tokens, clean them up and resell them! If you have a very large qua

0xVichy 59 Nov 14, 2022
The fastest nuker on discord, Proxy support and more

About okuru nuker is a nuker for discord written in python, It uses methods such as threading and requests to ban faster and perform at higher speeds.

63 Dec 31, 2022
🛒 Bot de lista de compras compartilhada para o Telegram

Lista de Compras Lista de compras de Cuducos e Flávia. Comandos do bot Comando Descrição /add item Adiciona item à lista de compras /remove item

Eduardo Cuducos 4 Jan 15, 2022
Jalali version of python calendar :date:

jcalendar jcalendar is Jalali implementation of Python's calendar module Status Install pip install jcalendar Documents This module almost follows Py

Iman Kermani 7 Aug 09, 2022
a simple floating window for watch cryptocurrency price

floating-monitor with cryptocurrency 浮動視窗虛擬貨幣價格監控 a floating monitor window to show price of cryptocurrency. use binance api to get price 半透明的浮動視窗讓你方便

Lin_Yi_Shen 1 Oct 22, 2021
This asynchronous telegram bot sells books.

Selling_Books_Bot Description Say, you have a bunch of items you need no more and you want to sell it all out. That's where you're going to have to us

Roman 1 Oct 24, 2021
Set up recurring buys in Gemini

Overview Set up recurring buys in Gemini. Given some keys (Create API Keys), allows you to configure a recurring buy using the reduced API maker and t

Ahmad Abuomar 3 Jan 06, 2022