The python SDK for Eto, the AI focused data platform for teams bringing AI models to production

Overview

Eto Labs Python SDK

This is the python SDK for Eto, the AI focused data platform for teams bringing AI models to production. The python SDK makes it easy to integrate Eto's features into your AI training and analysis workflow.

Installation

The Eto python SDK is available on PyPI and can be installed via Pip:

pip install etosdk

Eto SDK is compatible with Python 3.7+

Setup

Before using the SDK for the first time, you must configure it with your Eto API url and the API token.

import eto
eto.configure(url='<eto-api-url>', token='<api-token>')

The above configuration function creates a configuration file under $XDG_CONFIG_HOME/eto/eto.conf, which is usually ~/.config/eto/eto.conf.

Ingesting data

To create an ingestion job to convert raw data in Coco format and create a new dataset:

import eto
job = eto.ingest_coco('<dataset_name>',
                      {'image_dir': '<path/to/images>',
                       'annotations': '<path/to/annotations>',
                       'extras': {'key': 'value'}})

The ingestion job will run asynchronously server-side and convert the data to Rikai (parquet) format. Once complete, you should be able to see it in the data registry:

import eto

eto.list_datasets() # list all datasets

eto.get_dataset('<dataset_name>') # get information about a single dataset

Analysis

Accessing a particular dataset is easy via Pandas:

import eto
import pandas as pd

df = pd.read_eto('<dataset_name>') # Eto SDK adds a pandas extension

Training

To train a pytorch model, you can use the Dataset/DataLoader classes in Rikai:

import eto
from rikai.torch.vision import Dataset

dataset = Dataset('<dataset_name>') # Eto SDK adds an extension to Rikai to resolve dataset references 

for next_record in dataset:
    # training loop
    pass

A plain pytorch dataloader is also available from rikai.torch.data.DataLoader.

Local Spark configuration

For now, the Eto SDK relies on PySpark locally to read some of the custom Rikai types like annotations. While PySpark should be automatically installed as a transitive dependency, you may find that you need to change the Spark configurations to suit your local setup.

Your $SPARK_HOME/conf/spark-defaults.conf file should look something like the following:

spark.sql.extensions               ai.eto.rikai.sql.spark.RikaiSparkSessionExtensions
spark.jars.packages                ai.eto:rikai_2.12:0.0.13,org.apache.hadoop:hadoop-aws:3.2.0

# AWS
spark.executor.extraJavaOptions -Dcom.amazonaws.services.s3.enableV4=true -Dio.netty.tryReflectionSetAccessible=true
spark.driver.extraJavaOptions -Dcom.amazonaws.services.s3.enableV4=true -Dio.netty.tryReflectionSetAccessible=true
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version 2
spark.hadoop.com.amazonaws.services.s3.enableV4 true
fs.AbstractFileSystem.s3a.impl org.apache.hadoop.fs.s3a.S3A
fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem
fs.s3a.aws.credentials.provider com.amazonaws.auth.InstanceProfileCredentialsProvider,com.amazonaws.auth.DefaultAWSCredentialsProviderChain
You might also like...
BeeDrive: Open Source Privacy File Transfering System for Teams and Individual Developers
BeeDrive: Open Source Privacy File Transfering System for Teams and Individual Developers

BeeDrive For privacy and convenience purposes, more and more people try to keep data on their own hardwires instead of third-party cloud services such

Automatic login to Microsoft Teams conferences

Automatic login to Microsoft Teams conferences

To dynamically change the split direction in I3/Sway so as to split new windows automatically based on the width and height of the focused window
To dynamically change the split direction in I3/Sway so as to split new windows automatically based on the width and height of the focused window

To dynamically change the split direction in I3/Sway so as to split new windows automatically based on the width and height of the focused window Insp

This repository will be a draft of a package about the latest total marine fish production in Indonesia. Data will be collected from PIPP (Pusat Informasi Pelabuhan Perikanan).

indomarinefish This package will give us information about the latest total marine fish production in Indonesia. The Name of the fish is written in In

A wrapper for aqquiring Choice Coin directly through a Python Terminal. Leverages the TinyMan Python-SDK.

CHOICE_TinyMan_Wrapper A wrapper that allows users to acquire Choice Coin directly through their Terminal using ALGO and various Algorand Standard Ass

AWS SDK for Python

Boto3 - The AWS SDK for Python Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python, which allows Python developers to wri

Python SDK for Facebook's Graph API

Facebook Python SDK This client library is designed to support the Facebook Graph API and the official Facebook JavaScript SDK, which is the canonical

Box SDK for Python

Box Python SDK Installing Getting Started Authorization Server-to-Server Auth with JWT Traditional 3-legged OAuth2 Other Auth Options Usage Documentat

The Official Dropbox API V2 SDK for Python
The Official Dropbox API V2 SDK for Python

The offical Dropbox SDK for Python. Documentation can be found on Read The Docs. Installation Create an app via the Developer Console. Install via pip

Releases(v0.2)
  • v0.2(Dec 23, 2021)

    1. Access the Eto dataset registry API
    2. Submit Coco ingestion jobs
    3. Read into pandas dataframes and pytorch datasets/dataloaders

    What's Changed

    • Changhiskhan/sdk fixes by @changhiskhan in https://github.com/eto-ai/etosdk/pull/3
    • Changhiskhan/normalize uri by @changhiskhan in https://github.com/eto-ai/etosdk/pull/4
    • user specifies account name and SDK will formulate the url automatically by @changhiskhan in https://github.com/eto-ai/etosdk/pull/5
    • Sample notebook by @changhiskhan in https://github.com/eto-ai/etosdk/pull/6
    • minor notebook fix to wrap up M2 by @changhiskhan in https://github.com/eto-ai/etosdk/pull/7

    Full Changelog: https://github.com/eto-ai/etosdk/commits/v0.2

    Source code(tar.gz)
    Source code(zip)
Tools untuk krek akun igeh :v

Instalasi pkg update && pkg upgrade -y pkg install python git -y rm -rf InstaCrack git clone https://github.com/AngCyber/InstaCrack pip install reques

Moch Aang Ardiansyah-XD 12 Apr 05, 2022
The source code of the bot that displays erotic images on Discord

説明 このコードはDiscord.pyとNeko APIを使ったNsfw画像表示ボットのソースコードです。 成人向けコンテンツを含むボットなので、不快になる方はこのボットの作成中止をおすすめします。 使い方 まず、install.batを起動してください。 そのあとに、config.json を開き

はなくそ 1 Dec 28, 2021
WeChat SDK for Python

___ __ _______ ________ ___ ___ ________ _________ ________ ___ ___ |\ \ |\ \|\ ___ \ |\ ____\|\ \|\ \|\ __ \|\___

wechatpy 3.3k Dec 26, 2022
A powerfull Telegram Leech Bot

owner of this repo :- Abijthkutty contact me :- Abijth Telegram Torrent and Direct links Leecher Dont Abuse The Repo ... this is intented to run in Sm

αвιנтн 9 Jun 11, 2022
Simple Craigslist wrapper

python-craigslist A simple Craigslist wrapper. License: MIT-Zero. Disclaimer I don't work for or have any affiliation with Craigslist. This module was

Julio M. Alegria 370 Dec 22, 2022
An API wrapper for discord; maintained and improved from discord.py

Fusion.py Documentation What is Fusion.py you might ask; Fusion.py is a Discord.py fork that has most of the good features from most of the big Discor

Senarc Studios 5 Apr 19, 2022
pokemon-colorscripts compatible for mac

Pokemon colorscripts some scripts to print out images of pokemons to terminal. Inspired by DT's colorscripts compilation Description Prints out colore

43 Jan 06, 2023
Build better AWS infrastructure

Sceptre About Sceptre is a tool to drive AWS CloudFormation. It automates the mundane, repetitive and error-prone tasks, enabling you to concentrate o

sceptre 1.4k Jan 04, 2023
Microsoft Azure Storage Library for Python

Microsoft Azure Storage Library for Python

Microsoft Azure 329 Dec 16, 2022
An Open-Source Discord bot created to provide basic functionality which should be in every discord guild. We use this same bot with additional configurations for our guilds.

A Discord bot completely written to be taken from the source and built according to your own custom needs. This bot supports some core features and is

Tesseract Coding 14 Jan 11, 2022
IMDb + Auto + Unlimited Filter BoT

Telegram Movie Bot Features Auto Filter Manuel Filter IMDB Admin Commands Broadcast Index IMDB search Inline Search Random pics ids and User info Stat

Team AlinaX 1 Dec 03, 2021
A Discord bot that enables using breakout rooms on a server

Discord Breakout Room Bot This bot enables you to use breakout rooms on your Discord server! Note This bot was thrown together within a few hours, so

Till Müller 2 Nov 23, 2021
Public release of Telepathy, an OSINT toolkit for investigating Telegram groups. Enhanced features and improvements will be added over time.

Telepathy Welcome to Telepathy, an OSINT toolkit for scraping Telegram data to help investigate shady goings on. Currently, the tool is limited to scr

Jordan Wildon 484 Jan 01, 2023
Unarchive Bot for Telegram

Telegram UnArchiver Bot UnArchiveBot: 🇬🇧 Bot that allows you to extract supported archive formats in telegram. 🇹🇷 Desteklenen arşiv biçimleri tele

Hüzünlü Artemis [HuzunluArtemis] 25 May 07, 2022
A google search telegram bot.

Google-Search-Bot A google search telegram bot. Made with Python3 (C) @FayasNoushad Copyright permission under MIT License License - https://github.c

Fayas Noushad 37 Nov 24, 2022
Webservice that notifies users on Slack when a change in GitLab concern them.

Gitlab Slack Notifier Webservice that notifies users on Slack when a change in GitLab concern them. Setup Slack Create a Slack app, go to "OAuth & Per

Heuritech 2 Nov 04, 2021
Bot facebook

botfb Bot facebook Login via cookies cara install $pkg update && pkg upgrade $pkg install git python $git clone https://github.com/Ainx-BOT/botfb $cd

Fahmi Dev 12 Dec 18, 2022
Bot Realm of the Mad God Exalt (ROTMG). (Auto_nexus, Auto_HP, Auto_Loot)

Bot_ROTMG Bot Realm of the Mad God Exalt (ROTMG). (Auto_nexus, Auto_HP, Auto_Loot) *Este projeto visa apenas o aprendizado, quem faz mal uso é o único

Guilherme Silva Uchoa 1 Oct 30, 2021
High-Resolution Differential Z-Belt Mod for V0 (with optional Kirigami support)

V0-DBM This is a high-resolution differential pulley system belt mod for the Z-axis on Voron 0 with optional Kirigami Bed support. NOTE: Alpha version

Simon Küppers 11 Jan 07, 2023
PyMusic Player is a music player written in python3.

PyMusic Player is a music player written in python3. It harvests r

PythonSerious 2 Jan 30, 2022