Crowd-Kit: Computational Quality Control for Crowdsourcing
Crowd-Kit is a powerful Python library that implements commonly-used aggregation methods for crowdsourced annotation and offers the relevant metrics and datasets. We strive to implement functionality that simplifies working with crowdsourced data.
Currently, Crowd-Kit contains:
- implementations of commonly-used aggregation methods for categorical, pairwise, textual, and segmentation responses
- metrics of uncertainty, consistency, and agreement with aggregate
- loaders for popular crowdsourced datasets
The library is currently in a heavy development state, and interfaces are subject to change.
Installing
Installing Crowd-Kit is as easy as pip install crowd-kit
Getting Started
This example shows how to use Crowd-Kit for categorical aggregation using the classical Dawid-Skene algorithm.
First, let us do all the necessary imports.
from crowdkit.aggregation import DawidSkene
from crowdkit.datasets import load_dataset
import pandas as pd
Then, you need to read your annotations into Pandas DataFrame with columns task, performer, label. Alternatively, you can download an example dataset.
df = pd.read_csv('results.csv') # should contain columns: task, performer, label
# df, ground_truth = load_dataset('relevance-2') # or download an example dataset
Then you can aggregate the performer responses as easily as in scikit-learn:
aggregated_labels = DawidSkene(n_iter=100).fit_predict(df)
Implemented Aggregation Methods
Below is the list of currently implemented methods, including the already available (
Categorical Responses
| Method | Status |
|---|---|
| Majority Vote | |
| Dawid-Skene | |
| Gold Majority Vote | |
| M-MSR | |
| Wawa | |
| Zero-Based Skill | |
| GLAD | |
| BCC | |
Textual Responses
| Method | Status |
|---|---|
| RASA | |
| HRRASA | |
| ROVER | |
Image Segmentation
| Method | Status |
|---|---|
| Segmentation MV | |
| Segmentation RASA | |
| Segmentation EM | |
Pairwise Comparisons
| Method | Status |
|---|---|
| Bradley-Terry | |
| Noisy Bradley-Terry | |
Citation
- Ustalov D., Pavlichenko N., Losev V., Giliazev I., and Tulin E. A General-Purpose Crowdsourcing Computational Quality Control Toolkit for Python. The Ninth AAAI Conference on Human Computation and Crowdsourcing: Works-in-Progress and Demonstration Track. HCOMP 2021. 2021. arXiv: 2109.08584 [cs.HC].
@inproceedings{HCOMP2021/CrowdKit,
author = {Ustalov, Dmitry and Pavlichenko, Nikita and Losev, Vladimir and Giliazev, Iulian and Tulin, Evgeny},
title = {{A General-Purpose Crowdsourcing Computational Quality Control Toolkit for Python}},
year = {2021},
booktitle = {The Ninth AAAI Conference on Human Computation and Crowdsourcing: Works-in-Progress and Demonstration Track},
series = {HCOMP~2021},
eprint = {2109.08584},
eprinttype = {arxiv},
eprintclass = {cs.HC},
url = {https://www.humancomputation.com/assets/wips_demos/HCOMP_2021_paper_85.pdf},
language = {english},
}
Questions and Bug Reports
- For reporting bugs please use the Toloka/bugreport page.
- Join our English-speaking slack community for both tech and abstract questions.
License
© YANDEX LLC, 2020-2021. Licensed under the Apache License, Version 2.0. See LICENSE file for more details.
4.5k Dec 31, 2022
427 Dec 27, 2022
2 Jan 14, 2022
27 Nov 23, 2022
8.7k Dec 31, 2022
5 Nov 25, 2022
111 Nov 15, 2022
1.1k Jan 01, 2023
433 Nov 24, 2022
64 Dec 29, 2022
12 Dec 19, 2022
192 Jan 03, 2023
342 Dec 16, 2022
103 Dec 23, 2022
593 Dec 08, 2022
4 Feb 18, 2022
47 Dec 23, 2022
4 Aug 23, 2022
2 Jul 09, 2022
278 Jan 03, 2023