A simple python tool for explore your object detection dataset

Overview

Pyodi

Python Object Detection Insights

Documentation: https://gradiant.github.io/pyodi

Introduction

A simple tool for explore your object detection dataset. The goal of this library is to provide simple and intuitive visualizations from your dataset and automatically find the best parameters for generating a specific grid of anchors that can fit you data characteristics

Component Description
paint annotations paints COCO format annotations and predictions
ground-truth explore your dataset ground truth characteristics
evaluation evaluates AP and AR for between predictions and ground truth
train-config generation automatically generate anchors for your data
train-config evaluation evaluate the fitness between you data and your anchors
coco merge automatically merge COCO annotation files
coco split automatically split COCO annotation files in train and val subsets
crops split creates a new dataset by splitting images into crops and adapting the annotations file
crops merge translate COCO ground truth or COCO predictions crops split into original image coordinates

Installation

pip install pyodi

Usage

Pyodi includes different applications that can help you to extract the most from your dataset. You can download our TINY_COCO_ANIMAL dataset from the releases page in order to test the example commands. A classic flow could follow the following steps:

1. Annotation visualization

With pyodi paint_annotations you can easily visualize in a beautiful format your object detection dataset.

pyodi paint-annotations \
  $TINY_COCO_ANIMAL/annotations/train.json \
  $TINY_COCO_ANIMAL/sample_images \
  $TINY_COCO_ANIMAL/painted_images

COCO image with painted annotations

2. Ground truth exploration

It is very recommended to intensively explore your dataset before starting training. The analysis of your images and annotations will allow you to optimize aspects as the optimum image input size for your network or the shape distribution of the bounding boxes. You can use the ground_truth app for this task:

pyodi ground-truth $TINY_COCO_ANIMAL/annotations/train.json

Image shape distribution

Bbox distribution

Bbox center distribution

3. Train config generation

The design of anchors is critical for the performance of one-stage detectors. Pyodi can help you to automatically design a set of anchors that fit your data distribution.

pyodi train-config generation \
  $TINY_COCO_ANIMAL/annotations/train.json \
  --input-size [1280,720] \
  --n-ratios 3 --n-scales 3

Anchor clustering plot

4. Train config evaluation

Pyodi evaluation app has been designed with the aim of providing a simple tool to understand how well are your anchors matching your dataset. It automatically runs by default after executing train-config generation but it can also be run independently with:

pyodi train-config evaluation \
  $TINY_COCO_ANIMAL/annotations/train.json \
  $TINY_COCO_ANIMAL/resources/anchor_config.py \
  --input-size [1280,720]

Anchor overlap plot

Contributing

We appreciate all contributions to improve Pyodi. Please refer to Contributing guide for more info.

Comments
  • the results are not displayed or saved,

    the results are not displayed or saved,

    When I run pyodi ground-truth ../instances_train2017.json according to the tutorial, the results are not displayed or saved, and the code does not report errors. The following are the results:

    2021-08-27 21:57:18.223 | INFO     | pyodi.core.utils:coco_ground_truth_to_df:38 - Loading Ground Truth File
    2021-08-27 21:57:18.302 | INFO     | pyodi.core.utils:coco_ground_truth_to_df:48 - Converting COCO Ground Truth to pd.DataFrame
    2021-08-27 21:57:18.418 | INFO     | pyodi.plots.common:plot_scatter_with_histograms:61 - Plotting Scatter with Histograms
    2021-08-27 21:57:18.735 | INFO     | pyodi.plots.common:plot_scatter_with_histograms:61 - Plotting Scatter with Histograms
    
    
    bug 
    opened by cyh-123 5
  • add windows support by updating dependency

    add windows support by updating dependency

    @mmeendez8 @igonro @cgiraldo thanks a lot for this awesome package!

    pycocotools-fix package currently doesnt work on windows: https://github.com/junjuew/cocoapi/pull/4

    official pycocotools package is constantly updated by the facebook research team: https://pypi.org/project/pycocotools/

    by accepting this pr, pyodi package will be supported on windows.

    opened by fcakyon 5
  • Question about coco merge

    Question about coco merge

    I am attempting to merge 2 coco format file with pyodi and i cannot get it to work at all.

    The command I am running: pyodi coco merge result.json studio_test_val.json output.json

    The error I am getting: Traceback (most recent call last):

      File "/home/christ/miniconda3/envs/label-studio/bin/pyodi", line 5, in <module>
        from pyodi.cli import app
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pyodi/cli.py", line 5, in <module>
        from pyodi.apps.evaluation import evaluation
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pyodi/apps/evaluation.py", line 39, in <module>
        from pycocotools.cocoeval import COCOeval
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pycocotools/cocoeval.py", line 7, in <module>
        from . import mask as maskUtils
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pycocotools/mask.py", line 3, in <module>
        import pycocotools._mask as _mask
      File "pycocotools/_mask.pyx", line 1, in init pycocotools._mask
    ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
    

    Please help me with this issue. Thank you.

    opened by ChrisTurnbull04 2
  • Update kaleido and plotly versions

    Update kaleido and plotly versions

    Fixes kaleido permission err:

    PermissionError: [Errno 13] Permission denied: '/home/mmendez/anaconda3/envs/pyodi/lib/python3.7/site-packages/kaleido-0.1.0-py3.7-linux-x86_64.egg/kaleido/executable/kaleido'

    bug 
    opened by mmeendez8 2
  • Fix precommit issue between black and flake

    Fix precommit issue between black and flake

    Code to reproduce:

        image_0[
            int(image_0_bbox_top) : int(image_0_bbox_top + image_0_bbox_height),
            int(image_0_bbox_left) : int(image_0_bbox_left + image_0_bbox_width),
        ] = 0
    

    Output:

    flake8...................................................................Failed
    - hook id: flake8
    - exit code: 1
    
    tests/apps/test_paint_annotations.py:137:30: E203 whitespace before ':'
    

    Black is adding extra whitespaces that do not match with flake8 config

    bug CI/CD 
    opened by mmeendez8 2
  • [apps.coco_merge] `image_id` not being properly handled

    [apps.coco_merge] `image_id` not being properly handled

    The way image_id is currently updated only works when all the jsons contain image_ids filling all the range [0, N).

    If any of the jsons start it's image_id at other number or some image_id is missing from the range, the results are incorrect.

    bug 
    opened by daavoo 2
  • Add versioneer

    Add versioneer

    Use https://github.com/warner/python-versioneer for version control.

    Follow setup: https://jacobtomlinson.dev/posts/2020/versioning-and-formatting-your-python-code/

    enhancement CI/CD 
    opened by daavoo 2
  • [apps.paint_annotations] Extend coloring options

    [apps.paint_annotations] Extend coloring options

    Currently, when using pyodi paint-annotations each of the bounding boxes is painted with a random color:

    https://github.com/pyodi/pyodi/blob/master/pyodi/apps/paint_annotations.py#L83

    It would be a desirable feature to make this behaviour optional and allow the user to specify different coloring options. For example:

    • Constant color. Use same color for all annotations.
    • Color by key (property inside each annotation dictionary; i.e. category_id). Use same color for annotations sharing the same key.

    Example content of annotation:

    {
        "image_id": 0,
        "bbox": [704.2313842773438, 441.23284912109375, 15.76861572265625, 16.6729736328125],
        "score": 0.7878580093383789, 
        "category_id": 1
    }
    
    enhancement good first issue 
    opened by daavoo 2
  • Change histograms colors

    Change histograms colors

    Right now the histograms have the same color, so the appearance of the legend is a little bit weird.

    It would be nice if both histograms have different colors. The colors should be also different from the default plotly color sequence (More info: https://plotly.com/python/discrete-color/#color-sequences-in-plotly-express)

    enhancement 
    opened by igonro 2
  • Updated numpy version.

    Updated numpy version.

    Input:

    pip install pyodi
    pyodi coco merge 1.json 2.json output.json
    

    Output:

    ValueError: numpy.ndarray size changed, may indicate binary incompatibility
    

    When you run this code, the error will be fixed.

    pip install numpy==1.23.5
    
    opened by kadirnar 1
  • Question about Train Config Generation App.

    Question about Train Config Generation App.

    I want to get the anchor setting of my dataset, the example you showed is as follows: pyodi train-config generation \ $TINY_COCO_ANIMAL/annotations/train.json \ --input-size [1280,720] \ --n-ratios 3 --n-scales 3

    However, for Faster R-CNN, the size of input image is not fixed, in this case, how should I set the ' --input-size'?

    opened by missyoudaisy 0
  • Question: how precisely we can design anchors with pyodi visualisations

    Question: how precisely we can design anchors with pyodi visualisations

    The readme states Pyodi can help you to automatically design a set of anchors that fit your data distribution can you provide an example or reference that describes this process, and how the visualisations relate? Many thanks

    opened by robmarkcole 2
  • About clustering

    About clustering

    Excuse me, I want to know how clustering is done. I don't see the call of core/clustering.py in the place where the plot is drawn? Did I understand it wrong?

    question 
    opened by cyh-123 1
  • [apps.ground_truth] Add Image & Bounding Box combined plots.

    [apps.ground_truth] Add Image & Bounding Box combined plots.

    The current plots analyze the images and bounding boxes separately. There is some important insights missing that need to combine both sources of information. For example:

    • Total Area occupied by bounding boxes in each image.
    • Number of bounding boxes per image.
    • Mean distance between bounding box centroids per image.
    enhancement 
    opened by daavoo 0
Releases(v0.0.8)
  • v0.0.8(Dec 19, 2022)

    What's Changed

    • Feature/issue 174 update setup packages version by @jorgenusan in https://github.com/Gradiant/pyodi/pull/176
    • Bump to 0.0.8 by @jorgenusan in https://github.com/Gradiant/pyodi/pull/181

    New Contributors

    • @jorgenusan made their first contribution in https://github.com/Gradiant/pyodi/pull/176

    Full Changelog: https://github.com/Gradiant/pyodi/compare/v0.0.7...v0.0.8

    Source code(tar.gz)
    Source code(zip)
  • v0.0.7(Dec 16, 2021)

    What's Changed

    • Remove logger.catch decorator by @mmeendez8 in https://github.com/Gradiant/pyodi/pull/166
    • Feature: make JSON indentation configurable when merging COCO files by @philenius in https://github.com/Gradiant/pyodi/pull/171

    New Contributors

    • @philenius made their first contribution in https://github.com/Gradiant/pyodi/pull/171

    Full Changelog: https://github.com/Gradiant/pyodi/compare/v0.0.6...v0.0.7

    Source code(tar.gz)
    Source code(zip)
  • v0.0.6(Oct 29, 2021)

    What's Changed

    • Update kaleido and plotly versions by @mmeendez8 in https://github.com/Gradiant/pyodi/pull/163
    • Fix broken release link in README by @mmeendez8 in https://github.com/Gradiant/pyodi/pull/168
    • Fix macOS install by @daavoo in https://github.com/Gradiant/pyodi/pull/170

    Full Changelog: https://github.com/Gradiant/pyodi/compare/v0.0.5...v0.0.6

    Source code(tar.gz)
    Source code(zip)
  • v0.0.5(Aug 26, 2021)

    • Readme add pypi badge (#154)
    • Bugfix crops split fix minor bugs in crops split (#156)
    • Add windows to ci (#159)
    • update pycocotools dependency (#157)
    • [core nms] remove ensemble boxes dependency
    Source code(tar.gz)
    Source code(zip)
  • v0.0.4(Jul 5, 2021)

  • v0.0.3(Jul 2, 2021)

    • Call train-config evaluation from train-config generation app #147
    • Refactor load_ground_truth #149
    • [apps.paint_annotations] Add argument to paint only first n images #151
    Source code(tar.gz)
    Source code(zip)
Owner
GRADIANT - Centro Tecnolóxico de Telecomunicacións de Galicia
GRADIANT - Centro Tecnolóxico de Telecomunicacións de Galicia
Multi-class confusion matrix library in Python

Table of contents Overview Installation Usage Document Try PyCM in Your Browser Issues & Bug Reports Todo Outputs Dependencies Contribution References

Sepand Haghighi 1.3k Dec 31, 2022
Joyplots in Python with matplotlib & pandas :chart_with_upwards_trend:

JoyPy JoyPy is a one-function Python package based on matplotlib + pandas with a single purpose: drawing joyplots (a.k.a. ridgeline plots). The code f

Leonardo Taccari 462 Jan 02, 2023
This is a super simple visualization toolbox (script) for transformer attention visualization ✌

Trans_attention_vis This is a super simple visualization toolbox (script) for transformer attention visualization ✌ 1. How to prepare your attention m

Mingyu Wang 3 Jul 09, 2022
A shimmer pre-load component for Plotly Dash

dash-loading-shimmer A shimmer pre-load component for Plotly Dash Installation Get it with pip: pip install dash-loading-extras Or maybe you prefer Pi

Lucas Durand 4 Oct 12, 2022
A GUI for Pandas DataFrames

About Demo Installation Usage Features More Info About PandasGUI is a GUI for viewing, plotting and analyzing Pandas DataFrames. Demo Installation Ins

Adam Rose 2.8k Dec 24, 2022
Minimal Ethereum fee data viewer for the terminal, contained in a single python script.

Minimal Ethereum fee data viewer for the terminal, contained in a single python script. Connects to your node and displays some metrics in real-time.

48 Dec 05, 2022
An interactive GUI for WhiteboxTools in a Jupyter-based environment

whiteboxgui An interactive GUI for WhiteboxTools in a Jupyter-based environment GitHub repo: https://github.com/giswqs/whiteboxgui Documentation: http

Qiusheng Wu 105 Dec 15, 2022
`charts.css.py` brings `charts.css` to Python. Online documentation and samples is available at the link below.

charts.css.py charts.css.py provides a python API to convert your 2-dimension data lists into html snippet, which will be rendered into charts by CSS,

Ray Luo 3 Sep 23, 2021
GDSHelpers is an open-source package for automatized pattern generation for nano-structuring.

GDSHelpers GDSHelpers in an open-source package for automatized pattern generation for nano-structuring. It allows exporting the pattern in the GDSII-

Helge Gehring 76 Dec 16, 2022
Python histogram library - histograms as updateable, fully semantic objects with visualization tools. [P]ython [HYST]ograms.

physt P(i/y)thon h(i/y)stograms. Inspired (and based on) numpy.histogram, but designed for humans(TM) on steroids(TM). The goal is to unify different

Jan Pipek 120 Dec 08, 2022
The Timescale NFT Starter Kit is a step-by-step guide to get up and running with collecting, storing, analyzing and visualizing NFT data from OpenSea, using PostgreSQL and TimescaleDB.

Timescale NFT Starter Kit The Timescale NFT Starter Kit is a step-by-step guide to get up and running with collecting, storing, analyzing and visualiz

Timescale 102 Dec 24, 2022
Streamlit-template - A streamlit app template based on streamlit-option-menu

streamlit-template A streamlit app template for geospatial applications based on

Qiusheng Wu 41 Dec 10, 2022
Render Jupyter notebook in the terminal

jut - JUpyter notebook Terminal viewer. The command line tool view the IPython/Jupyter notebook in the terminal. Install pip install jut Usage $jut --

Kracekumar 169 Dec 27, 2022
GitHub Stats Visualizations : Transparent

GitHub Stats Visualizations : Transparent Generate visualizations of GitHub user and repository statistics using GitHub Actions. ⚠️ Disclaimer The pro

YuanYap 7 Apr 05, 2022
Sky attention heatmap of submissions to astrometry.net

astroheat Installation Requires Python 3.6+, Tested with Python 3.9.5 Install library dependencies pip install -r requirements.txt The program require

4 Jun 20, 2022
又一个云探针

ServerStatus-Murasame 感谢ServerStatus-Hotaru,又一个云探针诞生了(大雾 本项目在ServerStatus-Hotaru的基础上使用fastapi重构了服务端,部分修改了客户端与前端 项目还在非常原始的阶段,可能存在严重的问题 演示站:https://stat

6 Oct 19, 2021
Kglab - an abstraction layer in Python for building knowledge graphs

Graph Data Science: an abstraction layer in Python for building knowledge graphs, integrated with popular graph libraries – atop Pandas, RDFlib, pySHACL, RAPIDS, NetworkX, iGraph, PyVis, pslpython, p

derwen.ai 466 Jan 09, 2023
A tool to plot and execute Rossmos's Formula, that helps to catch serial criminals using mathematics

Rossmo Plotter A tool to plot and execute Rossmos's Formula using python, that helps to catch serial criminals using mathematics Author: Amlan Saha Ku

Amlan Saha Kundu 3 Aug 29, 2022
Simple and fast histogramming in Python accelerated with OpenMP.

pygram11 Simple and fast histogramming in Python accelerated with OpenMP with help from pybind11. pygram11 provides functions for very fast histogram

Doug Davis 28 Dec 14, 2022
Graphing communities on Twitch.tv in a visually intuitive way

VisualizingTwitchCommunities This project maps communities of streamers on Twitch.tv based on shared viewership. The data is collected from the Twitch

Kiran Gershenfeld 312 Jan 07, 2023