Simple helper library to convert a collection of numpy data to tfrecord, and build a tensorflow dataset from the tfrecord.

Last update: Jan 16, 2022

Related tags

Overview

numpy2tfrecord

Simple helper library to convert a collection of numpy data to tfrecord, and build a tensorflow dataset from the tfrecord.

Installation

$ git clone [email protected]:yonetaniryo/numpy2tfrecord.git
$ cd numpy2tfrecord
$ pip install .

Test

$ pytest -v

How to use

Convert a collection of numpy data to tfrecord

import numpy as np
from numpy2tfrecord import Numpy2Tfrecord

converter = Numpy2Tfrecord()
x = np.arange(100).reshape(10, 10).astype(np.float32)  # float array
y = np.arange(100).reshape(10, 10).astype(np.int64)  # int array
a = 5  # int
b = 0.3  # float
entry = {"x": x, "y": y, "a": a, "b": b}
converter.add(entry)  # add data entry
...

converter.export_to_tfrecord("test.tfrecord")  # export to tfrecord

Build a tensorflow dataset from tfrecord

dataset = build_dataset_from_tfrecord("test.tfrecord")  # load tfrecord and build tf.data.Dataset

You might also like...

MMdnn is a set of tools to help users inter-operate among different deep learning frameworks. E.g. model conversion and visualization. Convert models between Caffe, Keras, MXNet, Tensorflow, CNTK, PyTorch Onnx and CoreML.

MMdnn MMdnn is a comprehensive and cross-framework tool to convert, visualize and diagnose deep learning (DL) models. The "MM" stands for model manage

5.7k Jan 9, 2023

A tutorial showing how to train, convert, and run TensorFlow Lite object detection models on Android devices, the Raspberry Pi, and more!

1.3k Jan 2, 2023

tf2onnx - Convert TensorFlow, Keras and Tflite models to ONNX.

tf2onnx converts TensorFlow (tf-1.x or tf-2.x), tf.keras and tflite models to ONNX via command line or python api.

1.8k Jan 8, 2023

TensorFlow Ranking is a library for Learning-to-Rank (LTR) techniques on the TensorFlow platform

2.6k Jan 4, 2023

Minimal deep learning library written from scratch in Python, using NumPy/CuPy.

SmallPebble Project status: experimental, unstable. SmallPebble is a minimal/toy automatic differentiation/deep learning library written from scratch

92 Dec 30, 2022

Technical Indicators implemented in Python only using Numpy-Pandas as Magic - Very Very Fast! Very tiny! Stock Market Financial Technical Analysis Python library . Quant Trading automation or cryptocoin exchange

MyTT Technical Indicators implemented in Python only using Numpy-Pandas as Magic - Very Very Fast! to Stock Market Financial Technical Analysis Python

34 Dec 27, 2022

Calculates JMA (Japan Meteorological Agency) seismic intensity (shindo) scale from acceleration data recorded in NumPy array

shindo.py Calculates JMA (Japan Meteorological Agency) seismic intensity (shindo) scale from acceleration data stored in NumPy array Introduction Japa

3 Sep 23, 2022

Build tensorflow keras model pipelines in a single line of code. Created by Ram Seshadri. Collaborators welcome. Permission granted upon request.

deep_autoviml Build keras pipelines and models in a single line of code! Table of Contents Motivation How it works Technology Install Usage API Image

102 Dec 17, 2022

A hue shift helper for OBS

obs-hue-shift A hue shift helper for OBS This is a repo based on the really nice script Hegemege made. The original script can be found https://gist.g

1 Jan 10, 2022

Comments

Add some more choices for adding samples
add -> add_sample

add_batch: takes a dictionary whose elements each have the batch axis

add_list_of_samples: takes a list of dictionaries as input

add_list_of_batches

enhancement
opened by yonetaniryo 0

Releases(0.0.2)

0.0.2(Jan 26, 2022)

Source code(tar.gz)
Source code(zip)

Owner

Ryo Yonetani

GitHub Repository

A benchmark dataset for emulating atmospheric radiative transfer in weather and climate models with machine learning (NeurIPS 2021 Datasets and Benchmarks Track)

ClimART - A Benchmark Dataset for Emulating Atmospheric Radiative Transfer in Weather and Climate Models Official PyTorch Implementation Using deep le

21 Dec 31, 2022

PyTorch implementation of paper "IBRNet: Learning Multi-View Image-Based Rendering", CVPR 2021.

IBRNet: Learning Multi-View Image-Based Rendering PyTorch implementation of paper "IBRNet: Learning Multi-View Image-Based Rendering", CVPR 2021. IBRN

371 Jan 03, 2023

To Design and Implement Logistic Regression to Classify Between Benign and Malignant Cancer Types

To Design and Implement Logistic Regression to Classify Between Benign and Malignant Cancer Types, from a Database Taken From Dr. Wolberg reports his Clinic Cases.

1 Jul 31, 2022

Fast and customizable reconnaissance workflow tool based on simple YAML based DSL.

Fast and customizable reconnaissance workflow tool based on simple YAML based DSL, with support of notifications and distributed workload of that work

3 Mar 11, 2022

A generalist algorithm for cell and nucleus segmentation.

Cellpose | A generalist algorithm for cell and nucleus segmentation. Cellpose was written by Carsen Stringer and Marius Pachitariu. To learn about Cel

733 Dec 29, 2022

L-Verse: Bidirectional Generation Between Image and Text

Far beyond learning long-range interactions of natural language, transformers are becoming the de-facto standard for many vision tasks with their power and scalabilty

102 Dec 21, 2022

Weakly Supervised Posture Mining with Reverse Cross-entropy for Fine-grained Classification

Fine-grainedImageClassification Weakly Supervised Posture Mining with Reverse Cross-entropy for Fine-grained Classification We trained model here: lin

14 Oct 21, 2022

Aws-machine-learning-university-accelerated-tab - Machine Learning University: Accelerated Tabular Data Class

Machine Learning University: Accelerated Tabular Data Class This repository contains slides, notebooks, and datasets for the Machine Learning Universi

916 Dec 23, 2022

Dual Attention Network for Scene Segmentation (CVPR2019)

Dual Attention Network for Scene Segmentation(CVPR2019) Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang,and Hanqing Lu Introduction W

2.2k Dec 28, 2022

Sign-to-Speech for Sign Language Understanding: A case study of Nigerian Sign Language

Sign-to-Speech for Sign Language Understanding: A case study of Nigerian Sign Language This repository contains the code, model, and deployment config

16 Oct 23, 2022

ThunderGBM: Fast GBDTs and Random Forests on GPUs

Documentations | Installation | Parameters | Python (scikit-learn) interface What's new? ThunderGBM won 2019 Best Paper Award from IEEE Transactions o

647 Jan 04, 2023

yolov5目标检测模型的知识蒸馏（基于响应的蒸馏）

代码地址： https://github.com/Sharpiless/yolov5-knowledge-distillation 教师模型： python train.py --weights weights/yolov5m.pt \ --cfg models/yolov5m.ya

52 Dec 04, 2022

Pytorch Implementation of Auto-Compressing Subset Pruning for Semantic Image Segmentation

Pytorch Implementation of Auto-Compressing Subset Pruning for Semantic Image Segmentation Introduction ACoSP is an online pruning algorithm that compr

8 Dec 07, 2022

The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate.

The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate. Website • Key Features • How To Use • Docs •

21.1k Dec 29, 2022

The code for our paper Semi-Supervised Learning with Multi-Head Co-Training

Semi-Supervised Learning with Multi-Head Co-Training (PyTorch) Abstract Co-training, extended from self-training, is one of the frameworks for semi-su

6 Dec 04, 2022

Large scale embeddings on a single machine.

Marius Marius is a system under active development for training embeddings for large-scale graphs on a single machine. Training on large scale graphs

107 Jan 03, 2023

This repository contains FEDOT - an open-source framework for automated modeling and machine learning (AutoML)

package tests docs license stats support This repository contains FEDOT - an open-source framework for automated modeling and machine learning (AutoML

482 Dec 26, 2022

Official PyTorch implementation of DD3D: Is Pseudo-Lidar needed for Monocular 3D Object detection? (ICCV 2021), Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon.

DD3D: "Is Pseudo-Lidar needed for Monocular 3D Object detection?" Install // Datasets // Experiments // Models // License // Reference Full video Offi

364 Dec 27, 2022

Automated image registration. Registrationimation was too much of a mouthful.

alignimation Automated image registration. Registrationimation was too much of a mouthful. This repo contains the code used for my blog post Alignimat

9 Oct 13, 2022

This tutorial repository is to introduce the functionality of KGTK to first-time users

Welcome to the KGTK notebook tutorial The goal of this tutorial repository is to introduce the functionality of KGTK to first-time users. The Knowledg

58 Dec 21, 2022

Simple helper library to convert a collection of numpy data to tfrecord, and build a tensorflow dataset from the tfrecord.

Related tags

Overview

numpy2tfrecord

Installation

Test

How to use

Convert a collection of numpy data to tfrecord

Build a tensorflow dataset from tfrecord

You might also like...

MMdnn is a set of tools to help users inter-operate among different deep learning frameworks. E.g. model conversion and visualization. Convert models between Caffe, Keras, MXNet, Tensorflow, CNTK, PyTorch Onnx and CoreML.

A tutorial showing how to train, convert, and run TensorFlow Lite object detection models on Android devices, the Raspberry Pi, and more!

tf2onnx - Convert TensorFlow, Keras and Tflite models to ONNX.

TensorFlow Ranking is a library for Learning-to-Rank (LTR) techniques on the TensorFlow platform

Minimal deep learning library written from scratch in Python, using NumPy/CuPy.

Technical Indicators implemented in Python only using Numpy-Pandas as Magic - Very Very Fast! Very tiny! Stock Market Financial Technical Analysis Python library . Quant Trading automation or cryptocoin exchange

Calculates JMA (Japan Meteorological Agency) seismic intensity (shindo) scale from acceleration data recorded in NumPy array

Build tensorflow keras model pipelines in a single line of code. Created by Ram Seshadri. Collaborators welcome. Permission granted upon request.

A hue shift helper for OBS

Comments

Add some more choices for adding samples

Releases(0.0.2)

0.0.2(Jan 26, 2022)

Owner

Ryo Yonetani

A benchmark dataset for emulating atmospheric radiative transfer in weather and climate models with machine learning (NeurIPS 2021 Datasets and Benchmarks Track)

PyTorch implementation of paper "IBRNet: Learning Multi-View Image-Based Rendering", CVPR 2021.

To Design and Implement Logistic Regression to Classify Between Benign and Malignant Cancer Types

Fast and customizable reconnaissance workflow tool based on simple YAML based DSL.

A generalist algorithm for cell and nucleus segmentation.

L-Verse: Bidirectional Generation Between Image and Text

Weakly Supervised Posture Mining with Reverse Cross-entropy for Fine-grained Classification

Aws-machine-learning-university-accelerated-tab - Machine Learning University: Accelerated Tabular Data Class

Dual Attention Network for Scene Segmentation (CVPR2019)

Sign-to-Speech for Sign Language Understanding: A case study of Nigerian Sign Language

ThunderGBM: Fast GBDTs and Random Forests on GPUs

yolov5目标检测模型的知识蒸馏（基于响应的蒸馏）

Pytorch Implementation of Auto-Compressing Subset Pruning for Semantic Image Segmentation

The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate.

The code for our paper Semi-Supervised Learning with Multi-Head Co-Training

Large scale embeddings on a single machine.

This repository contains FEDOT - an open-source framework for automated modeling and machine learning (AutoML)

Official PyTorch implementation of DD3D: Is Pseudo-Lidar needed for Monocular 3D Object detection? (ICCV 2021), Dennis Park*, Rares Ambrus*, Vitor Guizilini, Jie Li, and Adrien Gaidon.

Automated image registration. Registrationimation was too much of a mouthful.

This tutorial repository is to introduce the functionality of KGTK to first-time users

Official PyTorch implementation of DD3D: Is Pseudo-Lidar needed for Monocular 3D Object detection? (ICCV 2021), Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon.