Javascript image annotation tool based on image segmentation.

Overview

JS Segment Annotator

Javascript image annotation tool based on image segmentation.

  • Label image regions with mouse.
  • Written in vanilla Javascript, with require.js dependency (packaged).
  • Pure client-side implementation of image segmentation.

A browser must support HTML canvas to use this tool.

There is an online demo.

Importing data

Prepare a JSON file that looks like the following. The required fields are labels and imageURLs. The annotationURLs are for existing data and can be omitted. Place the JSON file inside the data/ directory.

{
  "labels": [
    "background",
    "skin",
    "hair",
    "dress",
    "glasses",
    "jacket",
    "skirt"
  ],
  "imageURLs": [
    "data/images/1.jpg",
    "data/images/2.jpg"
  ],
  "annotationURLs": [
    "data/annotations/1.png",
    "data/annotations/2.png"
  ]
}

Then edit main.js to point to this JSON file. Open a Web browser and visit index.html.

Known issues

Browser incompatibility

A segmentation result can greatly differ due to the difference in Javascript implementation across Web browsers. The difference stems from numerical precision of floating point numbers, and there is no easy way to produce the exact same result across browsers.

Python tips

Annotation PNG

The annotation PNG file contains label map encoded in RGB value. Do the following to encode an index map.

import numpy as np
from PIL import Image

# Decode
encoded = np.array(Image.open('data/annotations/1.png'))
annotation = np.bitwise_or(np.bitwise_or(
    encoded[:, :, 0].astype(np.uint32),
    encoded[:, :, 1].astype(np.uint32) << 8),
    encoded[:, :, 2].astype(np.uint32) << 16)

print(np.unique(annotation))

# Encode
Image.fromarray(np.stack([
    np.bitwise_and(annotation, 255),
    np.bitwise_and(annotation >> 8, 255),
    np.bitwise_and(annotation >> 16, 255),
    ], axis=2).astype(np.uint8)).save('encoded.png')

JSON

Use JSON module.

import json

with open('data/example.json', 'r') as f:
    dataset = json.load(f)

Using dataURL

Do the following to convert between dataURL and NumPy format.

from PIL import Image
import base64
import io

# Encode
with io.BytesIO() as buffer:
    encoded.save(buffer, format='png')
    data_url = b'data:image/png;base64,' + base64.b64encode(buffer.getvalue())

# Decode
binary = base64.b64decode(data_url.replace(b'data:image/png;base64,', b''))
encoded = Image.open(io.BytesIO(binary))

Matlab tips

Annotation PNG

The annotation PNG file contains label map encoded in RGB value. Do the following to encode an index map.

% Decode

X = imread('data/annotations/0.png');
annotation = X(:, :, 1);
annotation = bitor(annotation, bitshift(X(:, :, 2), 8));
annotation = bitor(annotation, bitshift(X(:, :, 3), 16));

% Encode

X = cat(3, bitand(annotation, 255), ...
           bitand(bitshift(annotation, -8), 255), ...
           bitand(bitshift(annotation, -16)), 255));
imwrite(uint8(X), 'data/annotations/0.png');

JSON

Use the matlab-json package.

Using dataURL

Get the byte encoding tools.

Do the following to convert between dataURL and Matlab format.

% Decode

dataURL = 'data:image/png;base64,...';
png_data = base64decode(strrep(dataURL, 'data:image/png;base64,', ''));
annotation = imdecode(png_data, 'png');

% Encode

png_data = imencode(annotation, 'png');
dataURL = ['data:image/png;base64,', base64encode(png_data)];

Citation

We appreciate if you cite the following article in an academic paper. The tool was originally developed for this work.

@article{tangseng2017looking,
Author        = {Pongsate Tangseng and Zhipeng Wu and Kota Yamaguchi},
Title         = {Looking at Outfit to Parse Clothing},
Eprint        = {1703.01386v1},
ArchivePrefix = {arXiv},
PrimaryClass  = {cs.CV},
Year          = {2017},
Month         = {Mar},
Url           = {http://arxiv.org/abs/1703.01386v1}
}
Comments
  • Tool crashes on loading Pre-Annotated Images

    Tool crashes on loading Pre-Annotated Images

    Hi @kyamagu ,

    Thank you for sharing this project.

    I'm trying to use custom annotation images and I've created the JSON file accordingly. The tool loads up the relevant classes color-coded over the image. But when I try to correct some segmentation the tool stops working and the following error is printed in the console.

    Browser Console Dump localhost-1505715475195.txt

    SampleJSON untitled )

    Sample Browser Screenshot untitled

    opened by anandcu3 15
  • cant compile paperdoll

    cant compile paperdoll

    Hi, sorry to write it here, i was not able to find a repo for paper doll. I can not compile the paper doll. it seems I have some problem with gco, if I comment it out everything else will work, then when I type in

    result = feature_calculator.apply(config, input_sample) I will get Invalid MEX-file '/BS/wild-search-gaze/work/paperdoll-v1.0/paperdoll-v1.0/lib/pose/+pose/private/shiftdt.mexa64': /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /BS/wild-search-gaze/work/paperdoll-v1.0/paperdoll-v1.0/lib/pose/+pose/private/shiftdt.mexa64)

    Error in detect_fast>passmsg (line 128) [score0(:,:,k),Ix0(:,:,k),Iy0(:,:,k)] = shiftdt(child.score(:,:,k), child.w(1,k), child.w(2,k), child.w(3,k), child.w(4,k),child.startx(k),child.starty(k),Nx,Ny,child.step);

    Error in detect_fast (line 43) [msg,parts(k).Ix,parts(k).Iy,parts(k).Ik] = passmsg(parts(k),parts(par));

    Error in pose.estimate>process (line 38) box = detect_fast(im, model, detector_threshold);

    Error in pose.estimate (line 21) boxes{i} = process(model, samples{i}, scale, nms_threshold);

    Error in pose_calculator.apply (line 24) samples(i).(config.output) = pose.estimate(config.model, ...

    Error in feature_calculator.apply (line 25) sample = calculators{j}(config{j}, sample, varargin{:}, 'Encode', false);

    result =

     []
    

    can you help me with it please?

    opened by hosnasattar 11
  • Erase functionality

    Erase functionality

    I want to add an Erase button which erases any selection previously done. I used an button which when clicked calls a function very similar to the function responsible for Brush, which is:

    for (var y = -1 * brushSize; y <= brushSize; y++) {
        for (var x = -1 * brushSize; x <= brushSize; x++) {
          if (x * x + y * y > brushLimit) continue;
          var offset =
            4 *
            ((pos[1] + y) * this.layers.visualization.canvas.width + (pos[0] + x));
          offsets.push(offset);
          labels.push(label);
        }
      }
      annotator._updateAnnotation(offsets, labels);
    

    with one difference:

     labels.push(label);
    

    The last line inside the for loop I've hardcoded 0 being pushed into labels.

     labels.push(0);
    

    0 represents the background white color so on clicking the Erase button and selecting any area within a selection, it looks like stuff is being erased since the color of the label gets replaced by the background. But in truth this isn't actually erasing anything, just continues to add label with white background. So when I export it, the erased area expectedly still shows up.

    Any pointers on how to delete previously selected area?

    opened by Abdul-Ghani 10
  • How to code RGB and index information (8b)

    How to code RGB and index information (8b)

    Hi @kyamagu

    thank you for such a nice tool to annotate pictures. I'd like to ask if it is possible to do following:

    I already customized your tool and add 3 classes (Dress, Trousers, Jacket) as can be seen on this picture: 33 On the next picture below is annotated image from Pascal VOC 2012 for segmentation, containing RGB information + index for object and also the same for background. 11

    However, output from JS segmenter looks like this: 22

    If it is possible, could you please tell us on this example how to convert image from 32b to 8b and how to code the demanded information into image as it is in attached annotated VOC Pascal above. E.g. for background --> index 0 and RGB{0 0 0} for jacket --> index 3 and RGB{192 128 0}.

    Thank you in advance, much appreciated!

    opened by marticsvk 3
  • can't export annotation image with more than 2 images

    can't export annotation image with more than 2 images

    when I put more than 2 images in the imageURLs field in the json file, I can only view the first two images when I visit index.html. I can view all the other images by clicking "Next" though.

    The problem is: I can annotate the first two images and export the annotation pictures. But as for the other images, I am able to annotate the image, but not able to export it: There is no response when I click the "export" button.

    How can I solve this issue?

    p.s. The browser I use is Firefox(v42.0 Mac). index.html appeared to be blank with Chrome(v47.0 Mac) and Safari(v9.0).

    Thanks, john

    opened by johnzw 3
  • Some code using js-segment-annotator

    Some code using js-segment-annotator

    I am working on a code that uses this project, thought you might be interested.

    Repo: https://github.com/AKSHAYUBHAT/Vision Demo: http://vision.akshaybhat.com/ 3

    opened by ghost 3
  • Anyway way to debug when finding boundaries goes wrong?

    Anyway way to debug when finding boundaries goes wrong?

    I love your work here and greatly appreciate you sharing it. I'm not at all overly familiar with some of the concepts but wanted to use this in a proof of concept.

    I was using most of the default values and simply porting the code over to not be AMD modules. I have everything working except for the most important part - I've introduced a bug in the segmentation logic. What I end up with is almost perfect squares -

    image

    I've been combing over settings and options and trying to see what is different but nothing stands out. I know this isn't specific to your work on this library, but wanted to see if you had any tips for debugging this part of the code or if you had any tips for what could be going wrong?

    question 
    opened by plwalters 2
  • Annotated image are mostly zero pixel value

    Annotated image are mostly zero pixel value

    @kyamagu i used this tool for annotating my images for caffe-segnet,i first saw that all images are darker(you can hardly notice edges of annotated objects) but i thought on pixel level it might be different,but after checking in matlab,the whole image is almolt zero pixel.i would be happy if you could point out what i am doing wrong.thx

    opened by eliethesaiyan 2
  • Add a polygon-tool.

    Add a polygon-tool.

    • polygon tool
      • left click: span line between points
      • right click: erase drawn lines
    • keyboard shortcuts:
      • ctrl-key: switch between SLIC and polygon-tool
    • new buttons
    opened by NicolaiHarich 2
  • Image export

    Image export

    I cannot export the labeled image. I guess it was the incompatibility of the browser. Could you please suggest the right browser to use? Thanks in advance.

    SL

    question 
    opened by lsymuyu 1
  • Possible to update the annotation image after editing?

    Possible to update the annotation image after editing?

    I wanted to know if it is possible to have a save option which will update the annotation image in the data/annotations/ folder.

    If this is possible, where should I add this functionality?

    wontfix 
    opened by ghost 1
  • Polygon is difficult to close

    Polygon is difficult to close

    I find a bit difficult to close and complete a polygon. I checked the code but I don't understand where the relevant part is. How is the polygon closure checked?

    opened by FSet89 0
  • Brush tool used in the lateral border paints pixels on the other side of the image

    Brush tool used in the lateral border paints pixels on the other side of the image

    When using brush tool in the lateral border of one image, the area of the brush tool that stand out paints the pixels on the oposite border of the image.

    opened by polalbacar 0
  • The annotation masks disappear when clicked on Next button

    The annotation masks disappear when clicked on Next button

    I am able to export the annotated masks but as soon as I click on the "next" so that I can annotate the other images, the masks previously made are gone. We have to annotate the images again.

    It seems as if as soon as we hit "next", the app is removing the overlaid mask that has just been made and I am left with nothing. I have just cloned the library and started working.

    question 
    opened by nikhil-occipitaltech 5
  • Better annotation display

    Better annotation display

    Hi, Thanks for this awesome tool. May I propose an improvement in order to speed up the annotation process. I saw this in other tools and it was really efficient : the possibility to display raw images only in already annoted zones. I put below two examples of vegetation / soil annotation.

    Only vegetation toto

    Only soil: tata

    I checked your code but Im not so familiar with images overlay with javascript and I failed to display annotations with alpha = 1. Could you give me an advice ?

    enhancement question 
    opened by jerem-lab 3
  • The Upload image size is limited

    The Upload image size is limited

    HI, I just meet a problem when trying to upload images which are more than 50kb,these images showing in Firefox all are grey with nothing to see. Maybe the image compression is ok, beside that, any solutions ? Thank you for advance..

    wontfix 
    opened by wzx918 2
Releases(v2.0.1)
Owner
Kota Yamaguchi
Research scientist
Kota Yamaguchi
Create a random fluent image based on multiple colors.

FluentGenerator Create a random fluent image based on multiple colors. Navigation Example Install Update Usage In Python console FluentGenerator Fluen

1 Feb 02, 2022
A collection of python scripts which help you programatically create PNGs or GIFs

A collection of python scripts which help you programatically create PNGs or GIFs and their Metadata in bulk with custom rarity rates, upload them to OpenSea & list them for sale.

Tom 30 Dec 24, 2022
A Icon Maker GUI Made - Convert your image into icon ( .ico format ).

Icon-Maker-GUI A Icon Maker GUI Made Using Python 3.9.0 . It will take any image and convert it to ICO file, for web site favicon or Windows applicati

Insanecodes 12 Dec 15, 2021
【萝莉图片算法】高损图像压缩算法!?

【萝莉图片算法】高损图像压缩算法!? 我又发明出新算法了! 这次我发明的是新型高损图像压缩算法——萝莉图片算法!为什么是萝莉图片,这是因为它是使动用法,让图片变小所以是萝莉图片,大家一定要学好语文哦! 压缩效果 太神奇了!压缩率竟然高达99.97%! 与常见压缩算法对比 在图片最终大小为1KB的情况

黄巍 49 Oct 17, 2022
Blender addon to generate better building models from satellite imagery.

Blender addon to generate better building models from satellite imagery.

Ivan Ereshchenko 24 Apr 14, 2022
A script to generate a profile picture and a banner that show the same image on Discord.

Discord profile picture & banner generator A script to generate a profile picture and a banner that show the same image on Discord. Installation / Upd

Victor B. 9 Nov 27, 2022
GIMP script to export bitmap as GRAPHICS 4 file (aka SCREEN 5)

gimpfu-msx-gr4.py GIMP script to export bitmap as GRAPHICS 4 file (aka SCREEN 5). GRAPHICS 4 specs are: 256x212 (or 256x192); 16 color palette (from 5

Pedro de Medeiros 4 Oct 17, 2022
A GUI-based (PyQt5) tool used to design 2D linkage mechanism.

Pyslvs-UI A GUI-based (PyQt5) tool used to design 2D linkage mechanism. Planar Linkages Simulation Python-Solvespace: Kernel from Solvespace with Cyth

Yuan Chang 141 Dec 13, 2022
Manipulate EXIF and IFD metadata.

Tyf Copyright Distribution Support this project Buy Ѧ and: Send Ѧ to AUahWfkfr5J4tYakugRbfow7RWVTK35GPW Vote arky on Ark blockchain and earn Ѧ weekly

16 Jan 21, 2022
A simple image to text converter with GUI!

TEXTEMAGE! Textemage is a quick tool that extracts text from images, it is a Python based GUI program(also available in executable version). This is a

Akascape 5 Oct 26, 2022
An python script to convert images to upscaled versions made out of one-colour emojis.

ABOUT This is an python script to convert png, jpg and gif(output isnt animated :( ) images to scaled versions made out of one-colour emojis. Please n

0 Oct 19, 2022
A functional and efficient python implementation of the 3D version of Maxwell's equations

py-maxwell-fdfd Solving Maxwell's equations via A python implementation of the 3D curl-curl E-field equations. This code contains additional work to e

Nathan Zhao 12 Dec 11, 2022
View images in the terminal using ansi escape codes and python

terminal-photo-viewer view images in the terminal using ansi escape codes and python !! Only tested on Ubuntu 20.04.3 LTS with python version 3.8.10 D

1 Nov 30, 2021
CropImage is a simple toolkit for image cropping, detecting and cropping main body from pictures.

CropImage is a simple toolkit for image cropping, detecting and cropping main body from pictures. Support face and saliency detection.

Haofan Wang 15 Dec 22, 2022
Napari 3D Ortho Viewer - an ortho viewer for napari for 3D images

napari-3d-ortho-viewer Napari 3D Ortho Viewer - an ortho viewer for napari for 3D images This napari plugin was generated with Cookiecutter using @nap

niklas netter 5 Nov 28, 2022
Create a QR-code Generator app using only Python.

QR-code_Generator Create a QR-code Generator app using only Python. This apps generated a QR code for a single link. Libraryes used in this app -- py

Soham P Phasalkar 1 Oct 17, 2021
This Github Action automatically creates a GIF from a given web page to display on your project README

This Github Action automatically creates a GIF from a given web page to display on your project README

Pablo Lecolinet 28 Dec 15, 2022
Convert bitmap images to seeds for Tiny-83 NFT project.

What is this? This tool allows you to convert any 14p high and 22p wide Bitmap (.bmp) to the seed needed for the Tiny-83 NFT project. Project Twitter:

shib_maximalist 1 Oct 31, 2021
Raven is a tool written in Python3 allowing you to generate an unique image with some text.

🐦 Raven is a tool written in Python3 allowing you to generate an unique image with some text. It does it by searching the text on Google, do

Billy 39 Dec 20, 2022
Python scripts for semi-automated morphometric analysis of atolls from Landsat satellite Imagery.

AtollGeoMorph Python scripts for semi-automated morphometric analysis of atolls from Landsat satellite Imagery. The python scripts included allow user

1 Dec 16, 2022