Javascript image annotation tool based on image segmentation.

Overview

JS Segment Annotator

Javascript image annotation tool based on image segmentation.

  • Label image regions with mouse.
  • Written in vanilla Javascript, with require.js dependency (packaged).
  • Pure client-side implementation of image segmentation.

A browser must support HTML canvas to use this tool.

There is an online demo.

Importing data

Prepare a JSON file that looks like the following. The required fields are labels and imageURLs. The annotationURLs are for existing data and can be omitted. Place the JSON file inside the data/ directory.

{
  "labels": [
    "background",
    "skin",
    "hair",
    "dress",
    "glasses",
    "jacket",
    "skirt"
  ],
  "imageURLs": [
    "data/images/1.jpg",
    "data/images/2.jpg"
  ],
  "annotationURLs": [
    "data/annotations/1.png",
    "data/annotations/2.png"
  ]
}

Then edit main.js to point to this JSON file. Open a Web browser and visit index.html.

Known issues

Browser incompatibility

A segmentation result can greatly differ due to the difference in Javascript implementation across Web browsers. The difference stems from numerical precision of floating point numbers, and there is no easy way to produce the exact same result across browsers.

Python tips

Annotation PNG

The annotation PNG file contains label map encoded in RGB value. Do the following to encode an index map.

import numpy as np
from PIL import Image

# Decode
encoded = np.array(Image.open('data/annotations/1.png'))
annotation = np.bitwise_or(np.bitwise_or(
    encoded[:, :, 0].astype(np.uint32),
    encoded[:, :, 1].astype(np.uint32) << 8),
    encoded[:, :, 2].astype(np.uint32) << 16)

print(np.unique(annotation))

# Encode
Image.fromarray(np.stack([
    np.bitwise_and(annotation, 255),
    np.bitwise_and(annotation >> 8, 255),
    np.bitwise_and(annotation >> 16, 255),
    ], axis=2).astype(np.uint8)).save('encoded.png')

JSON

Use JSON module.

import json

with open('data/example.json', 'r') as f:
    dataset = json.load(f)

Using dataURL

Do the following to convert between dataURL and NumPy format.

from PIL import Image
import base64
import io

# Encode
with io.BytesIO() as buffer:
    encoded.save(buffer, format='png')
    data_url = b'data:image/png;base64,' + base64.b64encode(buffer.getvalue())

# Decode
binary = base64.b64decode(data_url.replace(b'data:image/png;base64,', b''))
encoded = Image.open(io.BytesIO(binary))

Matlab tips

Annotation PNG

The annotation PNG file contains label map encoded in RGB value. Do the following to encode an index map.

% Decode

X = imread('data/annotations/0.png');
annotation = X(:, :, 1);
annotation = bitor(annotation, bitshift(X(:, :, 2), 8));
annotation = bitor(annotation, bitshift(X(:, :, 3), 16));

% Encode

X = cat(3, bitand(annotation, 255), ...
           bitand(bitshift(annotation, -8), 255), ...
           bitand(bitshift(annotation, -16)), 255));
imwrite(uint8(X), 'data/annotations/0.png');

JSON

Use the matlab-json package.

Using dataURL

Get the byte encoding tools.

Do the following to convert between dataURL and Matlab format.

% Decode

dataURL = 'data:image/png;base64,...';
png_data = base64decode(strrep(dataURL, 'data:image/png;base64,', ''));
annotation = imdecode(png_data, 'png');

% Encode

png_data = imencode(annotation, 'png');
dataURL = ['data:image/png;base64,', base64encode(png_data)];

Citation

We appreciate if you cite the following article in an academic paper. The tool was originally developed for this work.

@article{tangseng2017looking,
Author        = {Pongsate Tangseng and Zhipeng Wu and Kota Yamaguchi},
Title         = {Looking at Outfit to Parse Clothing},
Eprint        = {1703.01386v1},
ArchivePrefix = {arXiv},
PrimaryClass  = {cs.CV},
Year          = {2017},
Month         = {Mar},
Url           = {http://arxiv.org/abs/1703.01386v1}
}
Comments
  • Tool crashes on loading Pre-Annotated Images

    Tool crashes on loading Pre-Annotated Images

    Hi @kyamagu ,

    Thank you for sharing this project.

    I'm trying to use custom annotation images and I've created the JSON file accordingly. The tool loads up the relevant classes color-coded over the image. But when I try to correct some segmentation the tool stops working and the following error is printed in the console.

    Browser Console Dump localhost-1505715475195.txt

    SampleJSON untitled )

    Sample Browser Screenshot untitled

    opened by anandcu3 15
  • cant compile paperdoll

    cant compile paperdoll

    Hi, sorry to write it here, i was not able to find a repo for paper doll. I can not compile the paper doll. it seems I have some problem with gco, if I comment it out everything else will work, then when I type in

    result = feature_calculator.apply(config, input_sample) I will get Invalid MEX-file '/BS/wild-search-gaze/work/paperdoll-v1.0/paperdoll-v1.0/lib/pose/+pose/private/shiftdt.mexa64': /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /BS/wild-search-gaze/work/paperdoll-v1.0/paperdoll-v1.0/lib/pose/+pose/private/shiftdt.mexa64)

    Error in detect_fast>passmsg (line 128) [score0(:,:,k),Ix0(:,:,k),Iy0(:,:,k)] = shiftdt(child.score(:,:,k), child.w(1,k), child.w(2,k), child.w(3,k), child.w(4,k),child.startx(k),child.starty(k),Nx,Ny,child.step);

    Error in detect_fast (line 43) [msg,parts(k).Ix,parts(k).Iy,parts(k).Ik] = passmsg(parts(k),parts(par));

    Error in pose.estimate>process (line 38) box = detect_fast(im, model, detector_threshold);

    Error in pose.estimate (line 21) boxes{i} = process(model, samples{i}, scale, nms_threshold);

    Error in pose_calculator.apply (line 24) samples(i).(config.output) = pose.estimate(config.model, ...

    Error in feature_calculator.apply (line 25) sample = calculators{j}(config{j}, sample, varargin{:}, 'Encode', false);

    result =

     []
    

    can you help me with it please?

    opened by hosnasattar 11
  • Erase functionality

    Erase functionality

    I want to add an Erase button which erases any selection previously done. I used an button which when clicked calls a function very similar to the function responsible for Brush, which is:

    for (var y = -1 * brushSize; y <= brushSize; y++) {
        for (var x = -1 * brushSize; x <= brushSize; x++) {
          if (x * x + y * y > brushLimit) continue;
          var offset =
            4 *
            ((pos[1] + y) * this.layers.visualization.canvas.width + (pos[0] + x));
          offsets.push(offset);
          labels.push(label);
        }
      }
      annotator._updateAnnotation(offsets, labels);
    

    with one difference:

     labels.push(label);
    

    The last line inside the for loop I've hardcoded 0 being pushed into labels.

     labels.push(0);
    

    0 represents the background white color so on clicking the Erase button and selecting any area within a selection, it looks like stuff is being erased since the color of the label gets replaced by the background. But in truth this isn't actually erasing anything, just continues to add label with white background. So when I export it, the erased area expectedly still shows up.

    Any pointers on how to delete previously selected area?

    opened by Abdul-Ghani 10
  • How to code RGB and index information (8b)

    How to code RGB and index information (8b)

    Hi @kyamagu

    thank you for such a nice tool to annotate pictures. I'd like to ask if it is possible to do following:

    I already customized your tool and add 3 classes (Dress, Trousers, Jacket) as can be seen on this picture: 33 On the next picture below is annotated image from Pascal VOC 2012 for segmentation, containing RGB information + index for object and also the same for background. 11

    However, output from JS segmenter looks like this: 22

    If it is possible, could you please tell us on this example how to convert image from 32b to 8b and how to code the demanded information into image as it is in attached annotated VOC Pascal above. E.g. for background --> index 0 and RGB{0 0 0} for jacket --> index 3 and RGB{192 128 0}.

    Thank you in advance, much appreciated!

    opened by marticsvk 3
  • can't export annotation image with more than 2 images

    can't export annotation image with more than 2 images

    when I put more than 2 images in the imageURLs field in the json file, I can only view the first two images when I visit index.html. I can view all the other images by clicking "Next" though.

    The problem is: I can annotate the first two images and export the annotation pictures. But as for the other images, I am able to annotate the image, but not able to export it: There is no response when I click the "export" button.

    How can I solve this issue?

    p.s. The browser I use is Firefox(v42.0 Mac). index.html appeared to be blank with Chrome(v47.0 Mac) and Safari(v9.0).

    Thanks, john

    opened by johnzw 3
  • Some code using js-segment-annotator

    Some code using js-segment-annotator

    I am working on a code that uses this project, thought you might be interested.

    Repo: https://github.com/AKSHAYUBHAT/Vision Demo: http://vision.akshaybhat.com/ 3

    opened by ghost 3
  • Anyway way to debug when finding boundaries goes wrong?

    Anyway way to debug when finding boundaries goes wrong?

    I love your work here and greatly appreciate you sharing it. I'm not at all overly familiar with some of the concepts but wanted to use this in a proof of concept.

    I was using most of the default values and simply porting the code over to not be AMD modules. I have everything working except for the most important part - I've introduced a bug in the segmentation logic. What I end up with is almost perfect squares -

    image

    I've been combing over settings and options and trying to see what is different but nothing stands out. I know this isn't specific to your work on this library, but wanted to see if you had any tips for debugging this part of the code or if you had any tips for what could be going wrong?

    question 
    opened by plwalters 2
  • Annotated image are mostly zero pixel value

    Annotated image are mostly zero pixel value

    @kyamagu i used this tool for annotating my images for caffe-segnet,i first saw that all images are darker(you can hardly notice edges of annotated objects) but i thought on pixel level it might be different,but after checking in matlab,the whole image is almolt zero pixel.i would be happy if you could point out what i am doing wrong.thx

    opened by eliethesaiyan 2
  • Add a polygon-tool.

    Add a polygon-tool.

    • polygon tool
      • left click: span line between points
      • right click: erase drawn lines
    • keyboard shortcuts:
      • ctrl-key: switch between SLIC and polygon-tool
    • new buttons
    opened by NicolaiHarich 2
  • Image export

    Image export

    I cannot export the labeled image. I guess it was the incompatibility of the browser. Could you please suggest the right browser to use? Thanks in advance.

    SL

    question 
    opened by lsymuyu 1
  • Possible to update the annotation image after editing?

    Possible to update the annotation image after editing?

    I wanted to know if it is possible to have a save option which will update the annotation image in the data/annotations/ folder.

    If this is possible, where should I add this functionality?

    wontfix 
    opened by ghost 1
  • Polygon is difficult to close

    Polygon is difficult to close

    I find a bit difficult to close and complete a polygon. I checked the code but I don't understand where the relevant part is. How is the polygon closure checked?

    opened by FSet89 0
  • Brush tool used in the lateral border paints pixels on the other side of the image

    Brush tool used in the lateral border paints pixels on the other side of the image

    When using brush tool in the lateral border of one image, the area of the brush tool that stand out paints the pixels on the oposite border of the image.

    opened by polalbacar 0
  • The annotation masks disappear when clicked on Next button

    The annotation masks disappear when clicked on Next button

    I am able to export the annotated masks but as soon as I click on the "next" so that I can annotate the other images, the masks previously made are gone. We have to annotate the images again.

    It seems as if as soon as we hit "next", the app is removing the overlaid mask that has just been made and I am left with nothing. I have just cloned the library and started working.

    question 
    opened by nikhil-occipitaltech 5
  • Better annotation display

    Better annotation display

    Hi, Thanks for this awesome tool. May I propose an improvement in order to speed up the annotation process. I saw this in other tools and it was really efficient : the possibility to display raw images only in already annoted zones. I put below two examples of vegetation / soil annotation.

    Only vegetation toto

    Only soil: tata

    I checked your code but Im not so familiar with images overlay with javascript and I failed to display annotations with alpha = 1. Could you give me an advice ?

    enhancement question 
    opened by jerem-lab 3
  • The Upload image size is limited

    The Upload image size is limited

    HI, I just meet a problem when trying to upload images which are more than 50kb,these images showing in Firefox all are grey with nothing to see. Maybe the image compression is ok, beside that, any solutions ? Thank you for advance..

    wontfix 
    opened by wzx918 2
Releases(v2.0.1)
Owner
Kota Yamaguchi
Research scientist
Kota Yamaguchi
Deep Illuminator is a data augmentation tool designed for image relighting.

Deep Illuminator Deep Illuminator is a data augmentation tool designed for image relighting. It can be used to easily and efficiently genera

George Chogovadze 52 Nov 29, 2022
Create a QR-code Generator app using only Python.

QR-code_Generator Create a QR-code Generator app using only Python. This apps generated a QR code for a single link. Libraryes used in this app -- py

Soham P Phasalkar 1 Oct 17, 2021
hashmask reverse lookup

ImageHashMasks Lookup Hashmask NFT index from a picture Setup pip install pillow click imagehash Usage $ python imagehashmasks.py

17 Nov 29, 2021
Remove Background from Image With Python

Install Library pypi $ pip3 install xremovebg

krypton 4 May 26, 2022
Make GIFs from time-stacked xarray.DataArrays (time, [optional band], y, x), dead-simple.

GeoGIF Make GIFs from time-stacked xarray.DataArrays (time, [optional band], y, x), dead-simple. from geogif import gif, dgif gif(data_array) dgif(das

Gabe Joseph 47 Dec 22, 2022
3D Reconstruction Software

Meshroom is a free, open-source 3D Reconstruction Software based on the AliceVision Photogrammetric Computer Vision framework. Learn more details abou

AliceVision 8.7k Jan 02, 2023
将位图转为彩色矢量 svg 图片

一个将位图描摹为彩色矢量 svg 图片的程序,是一个命令行工具,使用 Python 脚本实现,运行环境 Python3.8+。 ✨ 效果 以一个字帖图片为例,这是 png 格式的位图(370KB): 这是颜

Haujet Zhao 104 Dec 30, 2022
Short piece of code to create a rainbow gif of gradual contours from two shapefiles

rainbow-elevation-gif Short piece of code to create a rainbow gif of gradual con

Jess Roberts 6 Jan 17, 2022
A minimal, standalone viewer for 3D animations stored as stop-motion sequences of individual .obj mesh files.

ObjSequenceViewer V0.5 A minimal, standalone viewer for 3D animations stored as stop-motion sequences of individual .obj mesh files. Installation: pip

csmailis 2 Aug 04, 2022
Water marker for images.

watermarker linux users: To fix this error,please add truetype font path File "watermark.py", line 58, in module font = ImageFont.truetype("Dro

13 Oct 27, 2022
Draw a torus passing through three given points.

PyTorusThreePoints Draw a torus passing through three given points. Usage import numpy as np import pyvista as pv from torus_three_points.main import

2 Nov 19, 2021
A 3D structural engineering finite element library for Python.

An easy to use elastic 3D structural engineering finite element analysis library for Python.

Craig 220 Dec 27, 2022
starfish is a Python library for processing images of image-based spatial transcriptomics.

starfish: scalable pipelines for image-based transcriptomics starfish is a Python library for processing images of image-based spatial transcriptomics

199 Dec 08, 2022
Python Script to generate posters out of the images in directory.

Poster-Maker Python Script to generate posters out of the images in directory. This version is very basic ligthweight code to combine organize images

1 Feb 02, 2022
Hacking github graph with a easy python script

Hacking-Github-Graph Hacking github graph with a easy python script Requirements git latest version installed. A text editor (eg: vs code, sublime tex

SENPAI LEGEND 1 Nov 01, 2021
Parking management project which generates barcode parking ticket with user-friendly Tkinter program GUI

Parking-management-system Parking management project which generates barcode parking ticket with user-friendly Tkinter program GUI How to run Download

1 Jul 03, 2022
Graphical tool to make photo collage posters

PhotoCollage Graphical tool to make photo collage posters PhotoCollage allows you to create photo collage posters. It assembles the input photographs

Adrien Vergé 350 Jan 02, 2023
Polyfoto - Create image mosaics.

Polyfoto Create image mosaics. Showcase "Before and After Science" by Brian Eno "Scott 3" by Scott Walker Installation Clone this repository to your l

Shane Drabing 149 Dec 25, 2022
A Gtk based Image Selector with Preview

gtk-image-selector This is an attempt to restore Gtk Image Chooser "lost functionality": displaying an image preview when selecting images... This is

Spiros Georgaras 2 Sep 28, 2022
Create a static HTML/CSS image gallery from a bunch of images.

gallerize Create a static HTML/CSS image gallery from a bunch of images.

Jochen Kupperschmidt 19 Aug 21, 2022