A simple python tool for explore your object detection dataset

Overview

Pyodi

Python Object Detection Insights

Documentation: https://gradiant.github.io/pyodi

Introduction

A simple tool for explore your object detection dataset. The goal of this library is to provide simple and intuitive visualizations from your dataset and automatically find the best parameters for generating a specific grid of anchors that can fit you data characteristics

Component Description
paint annotations paints COCO format annotations and predictions
ground-truth explore your dataset ground truth characteristics
evaluation evaluates AP and AR for between predictions and ground truth
train-config generation automatically generate anchors for your data
train-config evaluation evaluate the fitness between you data and your anchors
coco merge automatically merge COCO annotation files
coco split automatically split COCO annotation files in train and val subsets
crops split creates a new dataset by splitting images into crops and adapting the annotations file
crops merge translate COCO ground truth or COCO predictions crops split into original image coordinates

Installation

pip install pyodi

Usage

Pyodi includes different applications that can help you to extract the most from your dataset. You can download our TINY_COCO_ANIMAL dataset from the releases page in order to test the example commands. A classic flow could follow the following steps:

1. Annotation visualization

With pyodi paint_annotations you can easily visualize in a beautiful format your object detection dataset.

pyodi paint-annotations \
  $TINY_COCO_ANIMAL/annotations/train.json \
  $TINY_COCO_ANIMAL/sample_images \
  $TINY_COCO_ANIMAL/painted_images

COCO image with painted annotations

2. Ground truth exploration

It is very recommended to intensively explore your dataset before starting training. The analysis of your images and annotations will allow you to optimize aspects as the optimum image input size for your network or the shape distribution of the bounding boxes. You can use the ground_truth app for this task:

pyodi ground-truth $TINY_COCO_ANIMAL/annotations/train.json

Image shape distribution

Bbox distribution

Bbox center distribution

3. Train config generation

The design of anchors is critical for the performance of one-stage detectors. Pyodi can help you to automatically design a set of anchors that fit your data distribution.

pyodi train-config generation \
  $TINY_COCO_ANIMAL/annotations/train.json \
  --input-size [1280,720] \
  --n-ratios 3 --n-scales 3

Anchor clustering plot

4. Train config evaluation

Pyodi evaluation app has been designed with the aim of providing a simple tool to understand how well are your anchors matching your dataset. It automatically runs by default after executing train-config generation but it can also be run independently with:

pyodi train-config evaluation \
  $TINY_COCO_ANIMAL/annotations/train.json \
  $TINY_COCO_ANIMAL/resources/anchor_config.py \
  --input-size [1280,720]

Anchor overlap plot

Contributing

We appreciate all contributions to improve Pyodi. Please refer to Contributing guide for more info.

Comments
  • the results are not displayed or saved,

    the results are not displayed or saved,

    When I run pyodi ground-truth ../instances_train2017.json according to the tutorial, the results are not displayed or saved, and the code does not report errors. The following are the results:

    2021-08-27 21:57:18.223 | INFO     | pyodi.core.utils:coco_ground_truth_to_df:38 - Loading Ground Truth File
    2021-08-27 21:57:18.302 | INFO     | pyodi.core.utils:coco_ground_truth_to_df:48 - Converting COCO Ground Truth to pd.DataFrame
    2021-08-27 21:57:18.418 | INFO     | pyodi.plots.common:plot_scatter_with_histograms:61 - Plotting Scatter with Histograms
    2021-08-27 21:57:18.735 | INFO     | pyodi.plots.common:plot_scatter_with_histograms:61 - Plotting Scatter with Histograms
    
    
    bug 
    opened by cyh-123 5
  • add windows support by updating dependency

    add windows support by updating dependency

    @mmeendez8 @igonro @cgiraldo thanks a lot for this awesome package!

    pycocotools-fix package currently doesnt work on windows: https://github.com/junjuew/cocoapi/pull/4

    official pycocotools package is constantly updated by the facebook research team: https://pypi.org/project/pycocotools/

    by accepting this pr, pyodi package will be supported on windows.

    opened by fcakyon 5
  • Question about coco merge

    Question about coco merge

    I am attempting to merge 2 coco format file with pyodi and i cannot get it to work at all.

    The command I am running: pyodi coco merge result.json studio_test_val.json output.json

    The error I am getting: Traceback (most recent call last):

      File "/home/christ/miniconda3/envs/label-studio/bin/pyodi", line 5, in <module>
        from pyodi.cli import app
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pyodi/cli.py", line 5, in <module>
        from pyodi.apps.evaluation import evaluation
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pyodi/apps/evaluation.py", line 39, in <module>
        from pycocotools.cocoeval import COCOeval
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pycocotools/cocoeval.py", line 7, in <module>
        from . import mask as maskUtils
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pycocotools/mask.py", line 3, in <module>
        import pycocotools._mask as _mask
      File "pycocotools/_mask.pyx", line 1, in init pycocotools._mask
    ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
    

    Please help me with this issue. Thank you.

    opened by ChrisTurnbull04 2
  • Update kaleido and plotly versions

    Update kaleido and plotly versions

    Fixes kaleido permission err:

    PermissionError: [Errno 13] Permission denied: '/home/mmendez/anaconda3/envs/pyodi/lib/python3.7/site-packages/kaleido-0.1.0-py3.7-linux-x86_64.egg/kaleido/executable/kaleido'

    bug 
    opened by mmeendez8 2
  • Fix precommit issue between black and flake

    Fix precommit issue between black and flake

    Code to reproduce:

        image_0[
            int(image_0_bbox_top) : int(image_0_bbox_top + image_0_bbox_height),
            int(image_0_bbox_left) : int(image_0_bbox_left + image_0_bbox_width),
        ] = 0
    

    Output:

    flake8...................................................................Failed
    - hook id: flake8
    - exit code: 1
    
    tests/apps/test_paint_annotations.py:137:30: E203 whitespace before ':'
    

    Black is adding extra whitespaces that do not match with flake8 config

    bug CI/CD 
    opened by mmeendez8 2
  • [apps.coco_merge] `image_id` not being properly handled

    [apps.coco_merge] `image_id` not being properly handled

    The way image_id is currently updated only works when all the jsons contain image_ids filling all the range [0, N).

    If any of the jsons start it's image_id at other number or some image_id is missing from the range, the results are incorrect.

    bug 
    opened by daavoo 2
  • Add versioneer

    Add versioneer

    Use https://github.com/warner/python-versioneer for version control.

    Follow setup: https://jacobtomlinson.dev/posts/2020/versioning-and-formatting-your-python-code/

    enhancement CI/CD 
    opened by daavoo 2
  • [apps.paint_annotations] Extend coloring options

    [apps.paint_annotations] Extend coloring options

    Currently, when using pyodi paint-annotations each of the bounding boxes is painted with a random color:

    https://github.com/pyodi/pyodi/blob/master/pyodi/apps/paint_annotations.py#L83

    It would be a desirable feature to make this behaviour optional and allow the user to specify different coloring options. For example:

    • Constant color. Use same color for all annotations.
    • Color by key (property inside each annotation dictionary; i.e. category_id). Use same color for annotations sharing the same key.

    Example content of annotation:

    {
        "image_id": 0,
        "bbox": [704.2313842773438, 441.23284912109375, 15.76861572265625, 16.6729736328125],
        "score": 0.7878580093383789, 
        "category_id": 1
    }
    
    enhancement good first issue 
    opened by daavoo 2
  • Change histograms colors

    Change histograms colors

    Right now the histograms have the same color, so the appearance of the legend is a little bit weird.

    It would be nice if both histograms have different colors. The colors should be also different from the default plotly color sequence (More info: https://plotly.com/python/discrete-color/#color-sequences-in-plotly-express)

    enhancement 
    opened by igonro 2
  • Updated numpy version.

    Updated numpy version.

    Input:

    pip install pyodi
    pyodi coco merge 1.json 2.json output.json
    

    Output:

    ValueError: numpy.ndarray size changed, may indicate binary incompatibility
    

    When you run this code, the error will be fixed.

    pip install numpy==1.23.5
    
    opened by kadirnar 1
  • Question about Train Config Generation App.

    Question about Train Config Generation App.

    I want to get the anchor setting of my dataset, the example you showed is as follows: pyodi train-config generation \ $TINY_COCO_ANIMAL/annotations/train.json \ --input-size [1280,720] \ --n-ratios 3 --n-scales 3

    However, for Faster R-CNN, the size of input image is not fixed, in this case, how should I set the ' --input-size'?

    opened by missyoudaisy 0
  • Question: how precisely we can design anchors with pyodi visualisations

    Question: how precisely we can design anchors with pyodi visualisations

    The readme states Pyodi can help you to automatically design a set of anchors that fit your data distribution can you provide an example or reference that describes this process, and how the visualisations relate? Many thanks

    opened by robmarkcole 2
  • About clustering

    About clustering

    Excuse me, I want to know how clustering is done. I don't see the call of core/clustering.py in the place where the plot is drawn? Did I understand it wrong?

    question 
    opened by cyh-123 1
  • [apps.ground_truth] Add Image & Bounding Box combined plots.

    [apps.ground_truth] Add Image & Bounding Box combined plots.

    The current plots analyze the images and bounding boxes separately. There is some important insights missing that need to combine both sources of information. For example:

    • Total Area occupied by bounding boxes in each image.
    • Number of bounding boxes per image.
    • Mean distance between bounding box centroids per image.
    enhancement 
    opened by daavoo 0
Releases(v0.0.8)
  • v0.0.8(Dec 19, 2022)

    What's Changed

    • Feature/issue 174 update setup packages version by @jorgenusan in https://github.com/Gradiant/pyodi/pull/176
    • Bump to 0.0.8 by @jorgenusan in https://github.com/Gradiant/pyodi/pull/181

    New Contributors

    • @jorgenusan made their first contribution in https://github.com/Gradiant/pyodi/pull/176

    Full Changelog: https://github.com/Gradiant/pyodi/compare/v0.0.7...v0.0.8

    Source code(tar.gz)
    Source code(zip)
  • v0.0.7(Dec 16, 2021)

    What's Changed

    • Remove logger.catch decorator by @mmeendez8 in https://github.com/Gradiant/pyodi/pull/166
    • Feature: make JSON indentation configurable when merging COCO files by @philenius in https://github.com/Gradiant/pyodi/pull/171

    New Contributors

    • @philenius made their first contribution in https://github.com/Gradiant/pyodi/pull/171

    Full Changelog: https://github.com/Gradiant/pyodi/compare/v0.0.6...v0.0.7

    Source code(tar.gz)
    Source code(zip)
  • v0.0.6(Oct 29, 2021)

    What's Changed

    • Update kaleido and plotly versions by @mmeendez8 in https://github.com/Gradiant/pyodi/pull/163
    • Fix broken release link in README by @mmeendez8 in https://github.com/Gradiant/pyodi/pull/168
    • Fix macOS install by @daavoo in https://github.com/Gradiant/pyodi/pull/170

    Full Changelog: https://github.com/Gradiant/pyodi/compare/v0.0.5...v0.0.6

    Source code(tar.gz)
    Source code(zip)
  • v0.0.5(Aug 26, 2021)

    • Readme add pypi badge (#154)
    • Bugfix crops split fix minor bugs in crops split (#156)
    • Add windows to ci (#159)
    • update pycocotools dependency (#157)
    • [core nms] remove ensemble boxes dependency
    Source code(tar.gz)
    Source code(zip)
  • v0.0.4(Jul 5, 2021)

  • v0.0.3(Jul 2, 2021)

    • Call train-config evaluation from train-config generation app #147
    • Refactor load_ground_truth #149
    • [apps.paint_annotations] Add argument to paint only first n images #151
    Source code(tar.gz)
    Source code(zip)
Owner
GRADIANT - Centro Tecnolóxico de Telecomunicacións de Galicia
GRADIANT - Centro Tecnolóxico de Telecomunicacións de Galicia
Keir&'s Visualizing Data on Life Expectancy

Keir's Visualizing Data on Life Expectancy Below is information on life expectancy in the United States from 1900-2017. You will also find information

9 Jun 06, 2022
Boltzmann visualization - Visualize the Boltzmann distribution for simple quantum models of molecular motion

Boltzmann visualization - Visualize the Boltzmann distribution for simple quantum models of molecular motion

1 Jan 22, 2022
Visualization ideas for data science

Nuance I use Nuance to curate varied visualization thoughts during my data scientist career. It is not yet a package but a list of small ideas. Welcom

Li Jiangchun 16 Nov 03, 2022
Visualizations for machine learning datasets

Introduction The facets project contains two visualizations for understanding and analyzing machine learning datasets: Facets Overview and Facets Dive

PAIR code 7.1k Jan 07, 2023
2D maze path solver visualizer implemented with python

2D maze path solver visualizer implemented with python

SS 14 Dec 21, 2022
These data visualizations were created for my introductory computer science course using Python

Homework 2: Matplotlib and Data Visualization Overview These data visualizations were created for my introductory computer science course using Python

Sophia Huang 12 Oct 20, 2022
Python toolkit for defining+simulating+visualizing+analyzing attractors, dynamical systems, iterated function systems, roulette curves, and more

Attractors A small module that provides functions and classes for very efficient simulation and rendering of iterated function systems; dynamical syst

1 Aug 04, 2021
Parallel t-SNE implementation with Python and Torch wrappers.

Multicore t-SNE This is a multicore modification of Barnes-Hut t-SNE by L. Van der Maaten with python and Torch CFFI-based wrappers. This code also wo

Dmitry Ulyanov 1.7k Jan 09, 2023
EPViz is a tool to aid researchers in developing, validating, and reporting their predictive modeling outputs.

EPViz (EEG Prediction Visualizer) EPViz is a tool to aid researchers in developing, validating, and reporting their predictive modeling outputs. A lig

Jeff 2 Oct 19, 2022
Displaying plot of death rates from past years in Poland. Data source from these years is in readme

Average-Death-Rate Displaying plot of death rates from past years in Poland The goal collect the data from a CSV file count the ADR (Average Death Rat

Oliwier Szymański 0 Sep 12, 2021
DALLE-tools provided useful dataset utilities to improve you workflow with WebDatasets.

DALLE tools DALLE-tools is a github repository with useful tools to categorize, annotate or check the sanity of your datasets. Installation Just clone

11 Dec 25, 2022
View part of your screen in grayscale or simulated color vision deficiency.

monolens View part of your screen in grayscale or filtered to simulate color vision deficiency. Watch the demo on YouTube. Install with pip install mo

Hans Dembinski 31 Oct 11, 2022
Complex heatmaps are efficient to visualize associations between different sources of data sets and reveal potential patterns.

Make Complex Heatmaps Complex heatmaps are efficient to visualize associations between different sources of data sets and reveal potential patterns. H

Zuguang Gu 973 Jan 09, 2023
The open-source tool for building high-quality datasets and computer vision models

The open-source tool for building high-quality datasets and computer vision models. Website • Docs • Try it Now • Tutorials • Examples • Blog • Commun

Voxel51 2.4k Jan 07, 2023
Open-questions - Open questions for Bellingcat technical contributors

Open questions for Bellingcat technical contributors These are difficult, long-term projects that would contribute to open source investigations at Be

Bellingcat 234 Dec 31, 2022
Data Visualizer Web-Application

Viz-It Data Visualizer Web-Application If I ask you where most of the data wrangler looses their time ? It is Data Overview and EDA. Presenting "Viz-I

Sagnik Roy 17 Nov 20, 2022
Implementation of SOMs (Self-Organizing Maps) with neighborhood-based map topologies.

py-self-organizing-maps Simple implementation of self-organizing maps (SOMs) A SOM is an unsupervised method for learning a mapping from a discrete ne

Jonas Grebe 6 Nov 22, 2022
The Metabolomics Integrator (MINT) is a post-processing tool for liquid chromatography-mass spectrometry (LCMS) based metabolomics.

MINT (Metabolomics Integrator) The Metabolomics Integrator (MINT) is a post-processing tool for liquid chromatography-mass spectrometry (LCMS) based m

Sören Wacker 0 May 04, 2022
Handout for the tutorial "Creating publication-quality figures with matplotlib"

Handout for the tutorial "Creating publication-quality figures with matplotlib"

JB Mouret 1.9k Jan 02, 2023
Generate a 3D Skyline in STL format and a OpenSCAD file from Gitlab contributions

Your Gitlab's contributions in a 3D Skyline gitlab-skyline is a Python command to generate a skyline figure from Gitlab contributions as Github did at

Félix Gómez 70 Dec 22, 2022