A simple flask application to collect annotations for the Turing Change Point Dataset, a benchmark dataset for change point detection algorithms

Overview

AnnotateChange

Welcome to the repository of the "AnnotateChange" application. This application was created to collect annotations of time series data in order to construct the Turing Change Point Dataset (TCPD). The TCPD is a dataset of real-world time series used to evaluate change point detection algorithms. For the change point detection benchmark that was created using this dataset, see the Turing Change Point Detection Benchmark repository.

Any work that uses this repository should cite our paper: Van den Burg & Williams - An Evaluation of Change Point Detection Algorithms (2020). You can use the following BibTeX entry:

@article{vandenburg2020evaluation,
        title={An Evaluation of Change Point Detection Algorithms},
        author={{Van den Burg}, G. J. J. and Williams, C. K. I.},
        journal={arXiv preprint arXiv:2003.06222},
        year={2020}
}

Here's a screenshot of what the application looks like during the annotation process:

screenshot of 
AnnotateChange

Some of the features of AnnotateChange include:

  • Admin panel to add/remove datasets, add/remove annotation tasks, add/remove users, and inspect incoming annotations.

  • Basic user management: authentication, email confirmation, forgotten password, automatic log out after inactivity, etc. Users are only allowed to register using an email address from an approved domain.

  • Task assignment of time series to user is done on the fly, ensuring no user ever annotates the same dataset twice, and prioritising datasets that are close to a desired number of annotations.

  • Interactive graph of a time series that supports pan and zoom, support for multidimensional time series.

  • Mandatory "demo" to onboard the user to change point annotation.

  • Backup of annotations to the admin via email.

  • Time series datasets are verified upon upload acccording to a strict schema.

Getting Started

Below are instructions for setting up the application for local development and for running the application with Docker.

Basic

AnnotateChange can be launched quickly for local development as follows:

  1. Clone the repo

    $ git clone https://github.com/alan-turing-institute/AnnotateChange
    $ cd AnnotateChange
    
  2. Set up a virtual environment and install dependencies (requires Python 3.7+)

    $ sudo apt-get install -y python3-venv # assuming Ubuntu
    $ pip install wheel
    $ python3 -m venv ./venv
    $ source ./venv/bin/activate
    $ pip install -r requirements.txt
    
  3. Create local development environment file

    $ cp .env.example .env.development
    $ sed -i 's/DB_TYPE=mysql/DB_TYPE=sqlite3/g' .env.development
    

    With DB_TYPE=sqlite3, we don't have to deal with MySQL locally.

  4. Initialize the database (this will be a local app.db file).

    $ ./flask.sh db upgrade
    
  5. Create the admin user account

    $ ./flask.sh admin add --auto-confirm-email
    

    The --auto-confirm-email flag automatically marks the email address of the admin user as confirmed. This is mostly useful in development environments when you don't have a mail address set up yet.

  6. Run the application

    $ ./flask.sh run
    

    This should tell you where its running, probably localhost:5000. You should be able to log in with the admin account you've just created.

  7. As admin, upload ALL demo datasets (included in demo_data) through: Admin Panel -> Add dataset. You should then be able to follow the introduction to the app (available from the landing page).

  8. After completing the instruction, you then will be able to access the user interface ("Home") to annotate your own time series.

Docker

To use AnnotateChange locally using Docker, follow the steps below. For a full-fledged installation on a server, see the deployment instructions.

  1. Install docker and docker-compose.

  2. Clone this repository and switch to it:

    $ git clone https://github.com/alan-turing-institute/AnnotateChange
    $ cd AnnotateChange
    
  3. Build the docker image:

    $ docker build -t gjjvdburg/annotatechange .
    
  4. Create the directory for persistent MySQL database storage:

    $ mkdir -p persist/{instance,mysql}
    $ sudo chown :1024 persist/instance
    $ chmod 775 persist/instance
    $ chmod g+s persist/instance
    
  5. Copy the environment variables file:

    $ cp .env.example .env
    

    Some environment variables can be adjusted if needed. For example, when moving to production, you'll need to change the FLASK_ENV variable accordingly. Please also make sure to set a proper SECRET_KEY and AC_MYSQL_PASSWORD (= MYSQL_PASSWORD). You'll also need to configure a mail account so the application can send out emails for registration etc. This is what the variables prefixed with MAIL_ are for. The ADMIN_EMAIL is likely your own email, it is used when the app encounters an error and to send backups of the annotation records. You can limit the email domains users can use with the USER_EMAIL_DOMAINS variable. See the config.py file for more info on the configuration options.

  6. Create a local docker network for communiation between the AnnotateChange app and the MySQL server:

    $ docker network create web
    
  7. Launch the services with docker-compose

    $ docker-compose up
    

    You may need to wait 2 minutes here before the database is initialized. If all goes well, you should be able to point your browser to localhost:7831 and see the landing page of the application. Stop the service before continuing to the next step (by pressing Ctrl+C).

  8. Once you have the app running, you'll want to create an admin account so you can upload datasets, manage tasks and users, and download annotation results. This can be done using the following command:

    $ docker-compose run --entrypoint 'flask admin add --auto-confirm-email' annotatechange
    
  9. As admin, upload ALL demo datasets (included in demo_data) through: Admin Panel -> Add dataset. You should then be able to follow the introduction to the app (available from the landing page).

  10. After completing the instruction, you then will be able to access the user interface ("Home") to annotate your own time series.

Notes

This codebase is provided "as is". If you find any problems, please raise an issue on GitHub.

The code is licensed under the MIT License.

This code was written by Gertjan van den Burg with helpful comments provided by Chris Williams.

Some implementation details

Below are some thoughts that may help make sense of the codebase.

  • AnnotateChange is a web application build on the Flask framework. See this excellent tutorial for an introduction to Flask. The flask.sh shell script loads the appropriate environment variables and runs the application.

  • The application handles user management and is centered around the idea of a "task" which links a particular user to a particular time series to annotate.

  • An admin role is available, and the admin user can manually assign and delete tasks as well as add/delete users, datasets, etc. The admin user is created using the cli (see the Getting Started documentation above).

  • All datasets must adhere to a specific dataset schema (see utils/dataset_schema.json). See the files in [demo_data] for examples, as well as those in TCPD.

  • Annotations are stored in the database using 0-based indexing. Tasks are assigned on the fly when a user requests a time series to annotate (see utils/tasks.py).

  • Users can only begin annotating when they have successfully passed the introduction.

  • Configuration of the app is done through environment variables, see the .env.example file for an example.

  • Docker is used for deployment (see the deployment documentation in docs), and Traefik is used for SSL, etc.

  • The time series graph is plotted using d3.js.

Owner
The Alan Turing Institute
The UK's national institute for data science and artificial intelligence.
The Alan Turing Institute
epub2sphinx is a tool to convert epub files to ReST for Sphinx

epub2sphinx epub2sphinx is a tool to convert epub files to ReST for Sphinx. It uses Pandoc for converting HTML data inside epub files into ReST. It cr

Nihaal 8 Dec 15, 2022
script to calculate total GPA out of 4, based on input gpa.csv

gpa_calculator script to calculate total GPA out of 4 based on input gpa.csv to use, create a total.csv file containing only one integer showing the t

Mohamad Bastin 1 Feb 07, 2022
Lightweight, configurable Sphinx theme. Now the Sphinx default!

What is Alabaster? Alabaster is a visually (c)lean, responsive, configurable theme for the Sphinx documentation system. It is Python 2+3 compatible. I

Jeff Forcier 670 Dec 19, 2022
Generate YARA rules for OOXML documents using ZIP local header metadata.

apooxml Generate YARA rules for OOXML documents using ZIP local header metadata. To learn more about this tool and the methodology behind it, check ou

MANDIANT 34 Jan 26, 2022
CoderByte | Practice, Tutorials & Interview Preparation Solutions|

CoderByte | Practice, Tutorials & Interview Preparation Solutions This repository consists of solutions to CoderByte practice, tutorials, and intervie

Eda AYDIN 6 Aug 09, 2022
An introduction to hikari, complete with different examples for different command handlers.

An intro to hikari This repo provides some simple examples to get you started with hikari. Contained in this repo are bots designed with both the hika

Ethan Henderson 18 Nov 29, 2022
A simple XLSX/CSV reader - to dictionary converter

sheet2dict A simple XLSX/CSV reader - to dictionary converter Installing To install the package from pip, first run: python3 -m pip install --no-cache

Tomas Pytel 216 Nov 25, 2022
A website for courses of Major Computer Science, NKU

A website for courses of Major Computer Science, NKU

Sakura 0 Oct 06, 2022
A comprehensive and FREE Online Python Development tutorial going step-by-step into the world of Python.

FREE Reverse Engineering Self-Study Course HERE Fundamental Python The book and code repo for the FREE Fundamental Python book by Kevin Thomas. FREE B

Kevin Thomas 7 Mar 19, 2022
Watch a Sphinx directory and rebuild the documentation when a change is detected. Also includes a livereload enabled web server.

sphinx-autobuild Rebuild Sphinx documentation on changes, with live-reload in the browser. Installation sphinx-autobuild is available on PyPI. It can

Executable Books 440 Jan 06, 2023
Crystal Smp plugin for show scoreboards

MCDR-CrystalScoreboards Crystal plugin for show scoreboards | Only 1.12 Usage !!s : Plugin help message !!s hide : Hide scoreboard !!s show : Show Sco

CristhianCd 3 Oct 12, 2021
Modified fork of CPython's ast module that parses `# type:` comments

Typed AST typed_ast is a Python 3 package that provides a Python 2.7 and Python 3 parser similar to the standard ast library. Unlike ast up to Python

Python 217 Dec 06, 2022
Portfolio project for Code Institute Full Stack software development course.

Comic Sales tracker This project is the third milestone project for the Code Institute Diploma in Full Stack Software Development. You can see the fin

1 Jan 10, 2022
EasyMultiClipboard - Python script written to handle more than 1 string in clipboard

EasyMultiClipboard - Python script written to handle more than 1 string in clipboard

WVlab 1 Jun 18, 2022
The mitosheet package, trymito.io, and other public Mito code.

Mito Monorepo Mito is a spreadsheet that lives inside your JupyterLab notebooks. It allows you to edit Pandas dataframes like an Excel file, and gener

Mito 1.4k Dec 31, 2022
Dynamic Resume Generator

Dynamic Resume Generator

Quinten Lisowe 15 May 19, 2022
Essential Document Generator

Essential Document Generator Dead Simple Document Generation Whether it's testing database performance or a new web interface, we've all needed a dead

Shane C Mason 59 Nov 11, 2022
level2-data-annotation_cv-level2-cv-15 created by GitHub Classroom

[AI Tech 3기 Level2 P Stage] 글자 검출 대회 팀원 소개 김규리_T3016 박정현_T3094 석진혁_T3109 손정균_T3111 이현진_T3174 임종현_T3182 Overview OCR (Optimal Character Recognition) 기술

6 Jun 10, 2022
A curated list of awesome mathematics resources

A curated list of awesome mathematics resources

Cyrille Rossant 6.7k Jan 05, 2023
Source Code for 'Practical Python Projects' (video) by Sunil Gupta

Apress Source Code This repository accompanies %Practical Python Projects by Sunil Gupta (Apress, 2021). Download the files as a zip using the green b

Apress 2 Jun 01, 2022