The python SDK for Eto, the AI focused data platform for teams bringing AI models to production

Overview

Eto Labs Python SDK

This is the python SDK for Eto, the AI focused data platform for teams bringing AI models to production. The python SDK makes it easy to integrate Eto's features into your AI training and analysis workflow.

Installation

The Eto python SDK is available on PyPI and can be installed via Pip:

pip install etosdk

Eto SDK is compatible with Python 3.7+

Setup

Before using the SDK for the first time, you must configure it with your Eto API url and the API token.

import eto
eto.configure(url='<eto-api-url>', token='<api-token>')

The above configuration function creates a configuration file under $XDG_CONFIG_HOME/eto/eto.conf, which is usually ~/.config/eto/eto.conf.

Ingesting data

To create an ingestion job to convert raw data in Coco format and create a new dataset:

import eto
job = eto.ingest_coco('<dataset_name>',
                      {'image_dir': '<path/to/images>',
                       'annotations': '<path/to/annotations>',
                       'extras': {'key': 'value'}})

The ingestion job will run asynchronously server-side and convert the data to Rikai (parquet) format. Once complete, you should be able to see it in the data registry:

import eto

eto.list_datasets() # list all datasets

eto.get_dataset('<dataset_name>') # get information about a single dataset

Analysis

Accessing a particular dataset is easy via Pandas:

import eto
import pandas as pd

df = pd.read_eto('<dataset_name>') # Eto SDK adds a pandas extension

Training

To train a pytorch model, you can use the Dataset/DataLoader classes in Rikai:

import eto
from rikai.torch.vision import Dataset

dataset = Dataset('<dataset_name>') # Eto SDK adds an extension to Rikai to resolve dataset references 

for next_record in dataset:
    # training loop
    pass

A plain pytorch dataloader is also available from rikai.torch.data.DataLoader.

Local Spark configuration

For now, the Eto SDK relies on PySpark locally to read some of the custom Rikai types like annotations. While PySpark should be automatically installed as a transitive dependency, you may find that you need to change the Spark configurations to suit your local setup.

Your $SPARK_HOME/conf/spark-defaults.conf file should look something like the following:

spark.sql.extensions               ai.eto.rikai.sql.spark.RikaiSparkSessionExtensions
spark.jars.packages                ai.eto:rikai_2.12:0.0.13,org.apache.hadoop:hadoop-aws:3.2.0

# AWS
spark.executor.extraJavaOptions -Dcom.amazonaws.services.s3.enableV4=true -Dio.netty.tryReflectionSetAccessible=true
spark.driver.extraJavaOptions -Dcom.amazonaws.services.s3.enableV4=true -Dio.netty.tryReflectionSetAccessible=true
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version 2
spark.hadoop.com.amazonaws.services.s3.enableV4 true
fs.AbstractFileSystem.s3a.impl org.apache.hadoop.fs.s3a.S3A
fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem
fs.s3a.aws.credentials.provider com.amazonaws.auth.InstanceProfileCredentialsProvider,com.amazonaws.auth.DefaultAWSCredentialsProviderChain
You might also like...
BeeDrive: Open Source Privacy File Transfering System for Teams and Individual Developers
BeeDrive: Open Source Privacy File Transfering System for Teams and Individual Developers

BeeDrive For privacy and convenience purposes, more and more people try to keep data on their own hardwires instead of third-party cloud services such

Automatic login to Microsoft Teams conferences

Automatic login to Microsoft Teams conferences

To dynamically change the split direction in I3/Sway so as to split new windows automatically based on the width and height of the focused window
To dynamically change the split direction in I3/Sway so as to split new windows automatically based on the width and height of the focused window

To dynamically change the split direction in I3/Sway so as to split new windows automatically based on the width and height of the focused window Insp

This repository will be a draft of a package about the latest total marine fish production in Indonesia. Data will be collected from PIPP (Pusat Informasi Pelabuhan Perikanan).

indomarinefish This package will give us information about the latest total marine fish production in Indonesia. The Name of the fish is written in In

A wrapper for aqquiring Choice Coin directly through a Python Terminal. Leverages the TinyMan Python-SDK.

CHOICE_TinyMan_Wrapper A wrapper that allows users to acquire Choice Coin directly through their Terminal using ALGO and various Algorand Standard Ass

AWS SDK for Python

Boto3 - The AWS SDK for Python Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python, which allows Python developers to wri

Python SDK for Facebook's Graph API

Facebook Python SDK This client library is designed to support the Facebook Graph API and the official Facebook JavaScript SDK, which is the canonical

Box SDK for Python

Box Python SDK Installing Getting Started Authorization Server-to-Server Auth with JWT Traditional 3-legged OAuth2 Other Auth Options Usage Documentat

The Official Dropbox API V2 SDK for Python
The Official Dropbox API V2 SDK for Python

The offical Dropbox SDK for Python. Documentation can be found on Read The Docs. Installation Create an app via the Developer Console. Install via pip

Releases(v0.2)
  • v0.2(Dec 23, 2021)

    1. Access the Eto dataset registry API
    2. Submit Coco ingestion jobs
    3. Read into pandas dataframes and pytorch datasets/dataloaders

    What's Changed

    • Changhiskhan/sdk fixes by @changhiskhan in https://github.com/eto-ai/etosdk/pull/3
    • Changhiskhan/normalize uri by @changhiskhan in https://github.com/eto-ai/etosdk/pull/4
    • user specifies account name and SDK will formulate the url automatically by @changhiskhan in https://github.com/eto-ai/etosdk/pull/5
    • Sample notebook by @changhiskhan in https://github.com/eto-ai/etosdk/pull/6
    • minor notebook fix to wrap up M2 by @changhiskhan in https://github.com/eto-ai/etosdk/pull/7

    Full Changelog: https://github.com/eto-ai/etosdk/commits/v0.2

    Source code(tar.gz)
    Source code(zip)
Criando Lambda Functions para Ingerir Dados de APIs com AWS CDK

LIVE001 - AWS Lambda para Ingerir Dados de APIs Fazer o deploy de uma função lambda com infraestrutura como código Lambda vai numa API externa e extra

Andre Sionek 12 Nov 20, 2022
Using GNU Radio and HackRF One to Receive, Analyze and Send ASK/OOK signals

play_with_ask NIS-8016 Lab A code: Recv.grc/py: Receive signals and match with ASK button using HackRF and GNU radio. I use AM demod block(can also in

Chen Anxue 1 Jul 04, 2022
Telegram bot for stream music or video on telegram

KYURA MUSIC Telegram bot for stream music or video on telegram, powered by PyTgCalls and Pyrogram Help Need Help me to translate this repo, click the

0 Dec 08, 2022
Instagram-follower-bot - An Instagram follower bot written in Python

Instagram Follower Bot An Instagram follower bot written in Python. The bot follows the follower of which account you want. e.g. (You want to follow @

Aytaç Kaşoğlu 1 Dec 31, 2021
Github integration with Telegram

The Telegram bot myGit is your GiHub assistant. In your conversations with your team, you can simply insert the information about the projects you are working at.

Alexandru Buzescu 2 Jan 06, 2022
Picot - A discord bot made to fetch images from Pexels and unsplash API and provide raw images directly in channels

Picot A discord bot made to fetch images from Pexels and unsplash API and provid

Ayush Chandwani 5 Jan 12, 2022
REPO USERBOT YANG DIBUAT DARI BERBAGAI REPO USERBOT GITHUB.

Lord Userbot Userbot Yang Digunakan Untuk Bersenang-Senang Di Telegram Repo Lord Userbot Repo Yang Dibuat Alvin Dari Berbagai Repo Userbot Github CARA

Alvin 70 Jan 02, 2023
微信支付接口V3版python库

wechatpayv3 介绍 微信支付接口V3版python库。 适用对象 wechatpayv3支持微信支付直连商户,接口说明详见 官网。 特性 平台证书自动更新,无需开发者关注平台证书有效性,无需手动下载更新; 支持本地缓存平台证书,初始化时指定平台证书保存目录即可。 适配进度 微信支付V3版A

chen gang 258 Jan 06, 2023
Automatically pick a winner who Retweeted, Commented, and Followed your Twitter account!

AutomaticTwitterGiveaways automates selecting winners for "Retweet, Comment, Follow" type Twitter giveaways.

1 Jan 13, 2022
A python script that can send notifications to your phone via SMS text

Discord SMS Notification A python script that help you send text message to your phone one of your desire discord channel have a new message. The proj

2 Apr 25, 2022
An powerfull telegram group management anime themed bot.

ErzaScarlet Erza Scarlet is the female deuteragonist of the anime/manga series Fairy Tail. She is an S-class Mage from the Guild Fairy Tail. Like most

ꜱōʜᴇʀᴜ ᴋāɴ (AKA) ꜱᴏʜᴀɪʟ ᴋʜᴀɴ 2 May 19, 2022
Requests based multi-threaded script for increasing followers on Spotify

Proxyless Spotify Follow Bot Requests based multi-threaded script for increasing followers on Spotify. Click here to report bugs. Usage Download ZIP h

397 Jan 03, 2023
☄️ High performance, easy to use and feature-rich Solana SDK for Python.

Solathon is an high performance, easy to use and feature-rich Solana SDK for Python. Easy for beginners, powerful for real world applications.

Bolt 28 Oct 10, 2022
Demo to explain how to use AWS Chalice to connect to twitter API and change profile picture at scheduled times.

chalice-twitter-demo Demo to explain how to use AWS Chalice to connect to twitter API and change profile picture at scheduled times. Video Demo Click

Ahmed Mohamed 4 Dec 13, 2021
Python Business Transactions Library - ContractsPY

Python Business Transactions Library - ContractsPY Declare and define business transactions in Python. Use the contracts library to validate business

Arzu Huseynov 7 Jun 21, 2022
Script que realiza a identificação de todos os logins e senhas dos wifis conectados em uma máquina e envia os dados para um e-mail especificado.

getWIFIConnection Script que realiza a identificação de todos os logins e senhas dos wifis conectados em uma máquina e envia os dados para um e-mail e

Vinícius Azevedo 3 Nov 27, 2022
Twitter FakeNFT With Python

This project is a server that fetches your Twitter profile picture and applies the hexagonal transparency mask displayed on the profiles of users who have an NFT profile picture.

Mathis HAMMEL 29 Apr 23, 2022
This is a simple bot for running Python code through Discord

Python Code Runner Discord Bot This is a simple bot for running Python code through Discord. It was originally developed for the Beginner.Codes Discor

beginner.py 1 Feb 14, 2022
Telegram bot for making Heroku app.json by @AbirHasan2005

Heroku-app.json A Telegram bot for making Heroku app.json by @AbirHasan2005. Demo Bot Host Bot Deploy to Heroku Click Below Button to Deploy to Heroku

Abir Hasan 46 Nov 13, 2022
Calendars for various securities exchanges.

IMPORTANT NOTE This package is currently unmaintained as the sponsor, quantopian, is going through corporate changes. As such there is a fork of this

Quantopian, Inc. 545 Jan 07, 2023