A simple python tool for explore your object detection dataset

Overview

Pyodi

Python Object Detection Insights

Documentation: https://gradiant.github.io/pyodi

Introduction

A simple tool for explore your object detection dataset. The goal of this library is to provide simple and intuitive visualizations from your dataset and automatically find the best parameters for generating a specific grid of anchors that can fit you data characteristics

Component Description
paint annotations paints COCO format annotations and predictions
ground-truth explore your dataset ground truth characteristics
evaluation evaluates AP and AR for between predictions and ground truth
train-config generation automatically generate anchors for your data
train-config evaluation evaluate the fitness between you data and your anchors
coco merge automatically merge COCO annotation files
coco split automatically split COCO annotation files in train and val subsets
crops split creates a new dataset by splitting images into crops and adapting the annotations file
crops merge translate COCO ground truth or COCO predictions crops split into original image coordinates

Installation

pip install pyodi

Usage

Pyodi includes different applications that can help you to extract the most from your dataset. You can download our TINY_COCO_ANIMAL dataset from the releases page in order to test the example commands. A classic flow could follow the following steps:

1. Annotation visualization

With pyodi paint_annotations you can easily visualize in a beautiful format your object detection dataset.

pyodi paint-annotations \
  $TINY_COCO_ANIMAL/annotations/train.json \
  $TINY_COCO_ANIMAL/sample_images \
  $TINY_COCO_ANIMAL/painted_images

COCO image with painted annotations

2. Ground truth exploration

It is very recommended to intensively explore your dataset before starting training. The analysis of your images and annotations will allow you to optimize aspects as the optimum image input size for your network or the shape distribution of the bounding boxes. You can use the ground_truth app for this task:

pyodi ground-truth $TINY_COCO_ANIMAL/annotations/train.json

Image shape distribution

Bbox distribution

Bbox center distribution

3. Train config generation

The design of anchors is critical for the performance of one-stage detectors. Pyodi can help you to automatically design a set of anchors that fit your data distribution.

pyodi train-config generation \
  $TINY_COCO_ANIMAL/annotations/train.json \
  --input-size [1280,720] \
  --n-ratios 3 --n-scales 3

Anchor clustering plot

4. Train config evaluation

Pyodi evaluation app has been designed with the aim of providing a simple tool to understand how well are your anchors matching your dataset. It automatically runs by default after executing train-config generation but it can also be run independently with:

pyodi train-config evaluation \
  $TINY_COCO_ANIMAL/annotations/train.json \
  $TINY_COCO_ANIMAL/resources/anchor_config.py \
  --input-size [1280,720]

Anchor overlap plot

Contributing

We appreciate all contributions to improve Pyodi. Please refer to Contributing guide for more info.

Comments
  • the results are not displayed or saved,

    the results are not displayed or saved,

    When I run pyodi ground-truth ../instances_train2017.json according to the tutorial, the results are not displayed or saved, and the code does not report errors. The following are the results:

    2021-08-27 21:57:18.223 | INFO     | pyodi.core.utils:coco_ground_truth_to_df:38 - Loading Ground Truth File
    2021-08-27 21:57:18.302 | INFO     | pyodi.core.utils:coco_ground_truth_to_df:48 - Converting COCO Ground Truth to pd.DataFrame
    2021-08-27 21:57:18.418 | INFO     | pyodi.plots.common:plot_scatter_with_histograms:61 - Plotting Scatter with Histograms
    2021-08-27 21:57:18.735 | INFO     | pyodi.plots.common:plot_scatter_with_histograms:61 - Plotting Scatter with Histograms
    
    
    bug 
    opened by cyh-123 5
  • add windows support by updating dependency

    add windows support by updating dependency

    @mmeendez8 @igonro @cgiraldo thanks a lot for this awesome package!

    pycocotools-fix package currently doesnt work on windows: https://github.com/junjuew/cocoapi/pull/4

    official pycocotools package is constantly updated by the facebook research team: https://pypi.org/project/pycocotools/

    by accepting this pr, pyodi package will be supported on windows.

    opened by fcakyon 5
  • Question about coco merge

    Question about coco merge

    I am attempting to merge 2 coco format file with pyodi and i cannot get it to work at all.

    The command I am running: pyodi coco merge result.json studio_test_val.json output.json

    The error I am getting: Traceback (most recent call last):

      File "/home/christ/miniconda3/envs/label-studio/bin/pyodi", line 5, in <module>
        from pyodi.cli import app
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pyodi/cli.py", line 5, in <module>
        from pyodi.apps.evaluation import evaluation
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pyodi/apps/evaluation.py", line 39, in <module>
        from pycocotools.cocoeval import COCOeval
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pycocotools/cocoeval.py", line 7, in <module>
        from . import mask as maskUtils
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pycocotools/mask.py", line 3, in <module>
        import pycocotools._mask as _mask
      File "pycocotools/_mask.pyx", line 1, in init pycocotools._mask
    ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
    

    Please help me with this issue. Thank you.

    opened by ChrisTurnbull04 2
  • Update kaleido and plotly versions

    Update kaleido and plotly versions

    Fixes kaleido permission err:

    PermissionError: [Errno 13] Permission denied: '/home/mmendez/anaconda3/envs/pyodi/lib/python3.7/site-packages/kaleido-0.1.0-py3.7-linux-x86_64.egg/kaleido/executable/kaleido'

    bug 
    opened by mmeendez8 2
  • Fix precommit issue between black and flake

    Fix precommit issue between black and flake

    Code to reproduce:

        image_0[
            int(image_0_bbox_top) : int(image_0_bbox_top + image_0_bbox_height),
            int(image_0_bbox_left) : int(image_0_bbox_left + image_0_bbox_width),
        ] = 0
    

    Output:

    flake8...................................................................Failed
    - hook id: flake8
    - exit code: 1
    
    tests/apps/test_paint_annotations.py:137:30: E203 whitespace before ':'
    

    Black is adding extra whitespaces that do not match with flake8 config

    bug CI/CD 
    opened by mmeendez8 2
  • [apps.coco_merge] `image_id` not being properly handled

    [apps.coco_merge] `image_id` not being properly handled

    The way image_id is currently updated only works when all the jsons contain image_ids filling all the range [0, N).

    If any of the jsons start it's image_id at other number or some image_id is missing from the range, the results are incorrect.

    bug 
    opened by daavoo 2
  • Add versioneer

    Add versioneer

    Use https://github.com/warner/python-versioneer for version control.

    Follow setup: https://jacobtomlinson.dev/posts/2020/versioning-and-formatting-your-python-code/

    enhancement CI/CD 
    opened by daavoo 2
  • [apps.paint_annotations] Extend coloring options

    [apps.paint_annotations] Extend coloring options

    Currently, when using pyodi paint-annotations each of the bounding boxes is painted with a random color:

    https://github.com/pyodi/pyodi/blob/master/pyodi/apps/paint_annotations.py#L83

    It would be a desirable feature to make this behaviour optional and allow the user to specify different coloring options. For example:

    • Constant color. Use same color for all annotations.
    • Color by key (property inside each annotation dictionary; i.e. category_id). Use same color for annotations sharing the same key.

    Example content of annotation:

    {
        "image_id": 0,
        "bbox": [704.2313842773438, 441.23284912109375, 15.76861572265625, 16.6729736328125],
        "score": 0.7878580093383789, 
        "category_id": 1
    }
    
    enhancement good first issue 
    opened by daavoo 2
  • Change histograms colors

    Change histograms colors

    Right now the histograms have the same color, so the appearance of the legend is a little bit weird.

    It would be nice if both histograms have different colors. The colors should be also different from the default plotly color sequence (More info: https://plotly.com/python/discrete-color/#color-sequences-in-plotly-express)

    enhancement 
    opened by igonro 2
  • Updated numpy version.

    Updated numpy version.

    Input:

    pip install pyodi
    pyodi coco merge 1.json 2.json output.json
    

    Output:

    ValueError: numpy.ndarray size changed, may indicate binary incompatibility
    

    When you run this code, the error will be fixed.

    pip install numpy==1.23.5
    
    opened by kadirnar 1
  • Question about Train Config Generation App.

    Question about Train Config Generation App.

    I want to get the anchor setting of my dataset, the example you showed is as follows: pyodi train-config generation \ $TINY_COCO_ANIMAL/annotations/train.json \ --input-size [1280,720] \ --n-ratios 3 --n-scales 3

    However, for Faster R-CNN, the size of input image is not fixed, in this case, how should I set the ' --input-size'?

    opened by missyoudaisy 0
  • Question: how precisely we can design anchors with pyodi visualisations

    Question: how precisely we can design anchors with pyodi visualisations

    The readme states Pyodi can help you to automatically design a set of anchors that fit your data distribution can you provide an example or reference that describes this process, and how the visualisations relate? Many thanks

    opened by robmarkcole 2
  • About clustering

    About clustering

    Excuse me, I want to know how clustering is done. I don't see the call of core/clustering.py in the place where the plot is drawn? Did I understand it wrong?

    question 
    opened by cyh-123 1
  • [apps.ground_truth] Add Image & Bounding Box combined plots.

    [apps.ground_truth] Add Image & Bounding Box combined plots.

    The current plots analyze the images and bounding boxes separately. There is some important insights missing that need to combine both sources of information. For example:

    • Total Area occupied by bounding boxes in each image.
    • Number of bounding boxes per image.
    • Mean distance between bounding box centroids per image.
    enhancement 
    opened by daavoo 0
Releases(v0.0.8)
  • v0.0.8(Dec 19, 2022)

    What's Changed

    • Feature/issue 174 update setup packages version by @jorgenusan in https://github.com/Gradiant/pyodi/pull/176
    • Bump to 0.0.8 by @jorgenusan in https://github.com/Gradiant/pyodi/pull/181

    New Contributors

    • @jorgenusan made their first contribution in https://github.com/Gradiant/pyodi/pull/176

    Full Changelog: https://github.com/Gradiant/pyodi/compare/v0.0.7...v0.0.8

    Source code(tar.gz)
    Source code(zip)
  • v0.0.7(Dec 16, 2021)

    What's Changed

    • Remove logger.catch decorator by @mmeendez8 in https://github.com/Gradiant/pyodi/pull/166
    • Feature: make JSON indentation configurable when merging COCO files by @philenius in https://github.com/Gradiant/pyodi/pull/171

    New Contributors

    • @philenius made their first contribution in https://github.com/Gradiant/pyodi/pull/171

    Full Changelog: https://github.com/Gradiant/pyodi/compare/v0.0.6...v0.0.7

    Source code(tar.gz)
    Source code(zip)
  • v0.0.6(Oct 29, 2021)

    What's Changed

    • Update kaleido and plotly versions by @mmeendez8 in https://github.com/Gradiant/pyodi/pull/163
    • Fix broken release link in README by @mmeendez8 in https://github.com/Gradiant/pyodi/pull/168
    • Fix macOS install by @daavoo in https://github.com/Gradiant/pyodi/pull/170

    Full Changelog: https://github.com/Gradiant/pyodi/compare/v0.0.5...v0.0.6

    Source code(tar.gz)
    Source code(zip)
  • v0.0.5(Aug 26, 2021)

    • Readme add pypi badge (#154)
    • Bugfix crops split fix minor bugs in crops split (#156)
    • Add windows to ci (#159)
    • update pycocotools dependency (#157)
    • [core nms] remove ensemble boxes dependency
    Source code(tar.gz)
    Source code(zip)
  • v0.0.4(Jul 5, 2021)

  • v0.0.3(Jul 2, 2021)

    • Call train-config evaluation from train-config generation app #147
    • Refactor load_ground_truth #149
    • [apps.paint_annotations] Add argument to paint only first n images #151
    Source code(tar.gz)
    Source code(zip)
Owner
GRADIANT - Centro Tecnolóxico de Telecomunicacións de Galicia
GRADIANT - Centro Tecnolóxico de Telecomunicacións de Galicia
100 Days of Code The Complete Python Pro Bootcamp for 2022

100-Day-With-Python 100 Days of Code - The Complete Python Pro Bootcamp for 2022. In this course, I spend with python language over 100 days, and I up

Rajdip Das 8 Jun 22, 2022
A high-level plotting API for pandas, dask, xarray, and networkx built on HoloViews

hvPlot A high-level plotting API for the PyData ecosystem built on HoloViews. Build Status Coverage Latest dev release Latest release Docs What is it?

HoloViz 694 Jan 04, 2023
A custom qq-plot for two sample data comparision

QQ-Plot 2 Sample Just a gist to include the custom code to draw a qq-plot in python when dealing with a "two sample problem". This means when u try to

1 Dec 20, 2021
Productivity Tools for Plotly + Pandas

Cufflinks This library binds the power of plotly with the flexibility of pandas for easy plotting. This library is available on https://github.com/san

Jorge Santos 2.7k Dec 30, 2022
This plugin plots the time you spent on a tag as a histogram.

This plugin plots the time you spent on a tag as a histogram.

Tom Dörr 7 Sep 09, 2022
A streamlit component for bi-directional communication with bokeh plots.

Streamlit Bokeh Events A streamlit component for bi-directional communication with bokeh plots. Its just a workaround till streamlit team releases sup

Ashish Shukla 123 Dec 25, 2022
A GUI for Pandas DataFrames

PandasGUI A GUI for analyzing Pandas DataFrames. Demo Installation Install latest release from PyPi: pip install pandasgui Install directly from Githu

Adam 2.8k Jan 03, 2023
CONTRIBUTIONS ONLY: Voluptuous, despite the name, is a Python data validation library.

CONTRIBUTIONS ONLY What does this mean? I do not have time to fix issues myself. The only way fixes or new features will be added is by people submitt

Alec Thomas 1.8k Dec 31, 2022
A simple script that displays pixel-based animation on GitHub Activity

GitHub Activity Animator This project contains a simple Javascript snippet that produces an animation on your GitHub activity tracker. The project als

16 Nov 15, 2021
https://there.oughta.be/a/macro-keyboard

inkkeys Details and instructions can be found on https://there.oughta.be/a/macro-keyboard In contrast to most of my other projects, I decided to put t

Sebastian Staacks 209 Dec 21, 2022
Collection of scripts for making high quality beautiful math-related posters.

Poster Collection of scripts for making high quality beautiful math-related posters. The poster can have as large printing size as 3x2 square feet wit

Nattawut Phetmak 3 Jun 09, 2022
A toolkit to generate MR sequence diagrams

mrsd: a toolkit to generate MR sequence diagrams mrsd is a Python toolkit to generate MR sequence diagrams, as shown below for the basic FLASH sequenc

Julien Lamy 3 Dec 25, 2021
plotly scatterplots which show molecule images on hover!

molplotly Plotly scatterplots which show molecule images on hovering over the datapoints! Required packages: pandas rdkit jupyter_dash ➡️ See example.

150 Dec 28, 2022
Python package to visualize and cluster partial dependence.

partial_dependence A python library for plotting partial dependence patterns of machine learning classifiers. The technique is a black box approach to

NYU Visualization Lab 25 Nov 14, 2022
Gallery of applications built using bqplot and widget libraries like ipywidgets, ipydatagrid etc.

bqplot Gallery This is a gallery of bqplot examples. View the gallery at https://bqplot.github.io/bqplot-gallery. Contributing new examples Clone this

8 Aug 23, 2022
2021 grafana arbitrary file read

2021_grafana_arbitrary_file_read base on pocsuite3 try 40 default plugins of grafana alertlist annolist barchart cloudwatch dashlist elasticsearch gra

ATpiu 5 Nov 09, 2022
D-Analyst : High Performance Visualization Tool

D-Analyst : High Performance Visualization Tool D-Analyst is a high performance data visualization built with python and based on OpenGL. It allows to

4 Apr 14, 2022
🧇 Make Waffle Charts in Python.

PyWaffle PyWaffle is an open source, MIT-licensed Python package for plotting waffle charts. It provides a Figure constructor class Waffle, which coul

Guangyang Li 528 Jan 02, 2023
view cool stats related to your discord account.

DiscoStats cool statistics generated using your discord data. How? DiscoStats is not a service that breaks the Discord Terms of Service or Community G

ibrahim hisham 5 Jun 02, 2022
The official colors of the FAU as matplotlib/seaborn colormaps

FAU - Colors The official colors of Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) as matplotlib / seaborn colormaps. We support the old colo

Machine Learning and Data Analytics Lab FAU 9 Sep 05, 2022