The evaluator covering all of the metrics required by tasks within the DUE Benchmark.

Last update: Jan 21, 2022

Related tags

Overview

DUE Evaluator

The repository contains the evaluator covering all of the metrics required by tasks within the DUE Benchmark, i.e., set-based F1 (for KIE), ANLS (used in document VQA), accuracy (including variant used in WTQ), as well as group-based ANLS we proposed for KIE problems with structured output.

Usage

The deval command will be available after the package installation. Every time, it is required to provide input and output files (both in the DU-Schema format) using -o and -r parameters.

Other settings are task-specific and limited to metric (-m) and optional case-insensitiveness (-i). Recommended values of these are:

Dataset	Metric	Case insensitive
DocVQA, InfographicsVQA	ANLS	Yes
Kleister Charity, DeepForm	F1	Yes
PapersWithCode	GROUP-ANLS	Yes
WikiTableQuestions	WTQ	No (handled by metric itself)
TabFact	F1 (obtained value will be equal to Accuracy)	No

Owner

DUE Benchmark

The benchmark consisting of both available and reformulated datasets to measure the end-to-end capabilities of systems in real-world scenarios.

GitHub Repository

An interactive TLS-capable intercepting HTTP proxy for penetration testers and software developers.

mitmproxy mitmproxy is an interactive, SSL/TLS-capable intercepting proxy with a console interface for HTTP/1, HTTP/2, and WebSockets. mitmdump is the

29.7k Jan 02, 2023

Repository for JIDA SNP Browser Web Application: Local Deployment

JIDA JIDA is a web application that retrieves SNP information for a genomic region of interest in Homo sapiens and calculates specific summary statist

3 Mar 03, 2022

A pure Python script to easily get a reverse shell

easy-shell A pure Python script to easily get a reverse shell. How it works? After sending a request, it generates a payload with different commands a

48 Dec 12, 2022

Python 3 wrapper of Microsoft UIAutomation. Support UIAutomation for MFC, WindowsForm, WPF, Modern UI(Metro UI), Qt, IE, Firefox, Chrome ...

1.6k Dec 29, 2022

A grab-bag of nifty pytest plugins

A goody-bag of nifty plugins for pytest OS Build Coverage Plugin Description Supported OS pytest-server-fixtures Extensible server-running framework w

492 Jan 03, 2023

This is a pytest plugin, that enables you to test your code that relies on a running MongoDB database

This is a pytest plugin, that enables you to test your code that relies on a running MongoDB database. It allows you to specify fixtures for MongoDB process and client.

19 Oct 21, 2022

User-interest mock backend server implemnted using flask restful, and SQLAlchemy ORM confiugred with sqlite

Flask_Restful_SQLAlchemy_server User-interest mock backend server implemnted using flask restful, and SQLAlchemy ORM confiugred with sqlite. Backend b

1 Nov 17, 2022

Auto Click by pyautogui and excel operations.

2 Dec 21, 2021

Wraps any WSGI application and makes it easy to send test requests to that application, without starting up an HTTP server.

WebTest This wraps any WSGI application and makes it easy to send test requests to that application, without starting up an HTTP server. This provides

325 Dec 30, 2022

A pytest plugin to skip `@pytest.mark.slow` tests by default.

pytest-skip-slow A pytest plugin to skip @pytest.mark.slow tests by default. Include the slow tests with --slow. Installation $ pip install pytest-ski

19 Jan 04, 2023

Um scraper feito em python que gera arquivos de excel baseados nas tier lists do site LoLalytics.

LoLalytics-scraper Um scraper feito em python que gera arquivos de excel baseados nas tier lists do site LoLalytics. Começando por um único script com

1 Feb 19, 2022

frwk_51pwn is an open-sourced remote vulnerability testing and proof-of-concept development framework

frwk_51pwn Legal Disclaimer Usage of frwk_51pwn for attacking targets without prior mutual consent is illegal. frwk_51pwn is for security testing purp

4 Apr 24, 2022

This is a web test framework based on python+selenium

Basic thoughts for this framework There should have a BasePage.py to be the parent page and all the page object should inherit this class BasePage.py

2 Mar 09, 2022

This is a Python script for Github Bot which uses Selenium to Automate things.

github-follow-unfollow-bot This is a Python script for Github Bot which uses Selenium to Automate things. Pre-requisites :- Python A Github Account Re

10 Jul 01, 2022

Redis fixtures and fixture factories for Pytest.

Redis fixtures and fixture factories for Pytest.This is a pytest plugin, that enables you to test your code that relies on a running Redis database. It allows you to specify additional fixtures for R

86 Dec 23, 2022

Coverage plugin for pytest.

Overview docs tests package This plugin produces coverage reports. Compared to just using coverage run this plugin does some extras: Subprocess suppor

1.4k Dec 29, 2022

Python version of the Playwright testing and automation library.

🎭 Playwright for Python Docs | API Playwright is a Python library to automate Chromium, Firefox and WebKit browsers with a single API. Playwright del

7.8k Jan 02, 2023

Data App Performance Tests

Data App Performance Tests My hypothesis is that The different architectures of

6 Dec 14, 2022

Make Selenium work on Github Actions

Make Selenium work on Github Actions Scraping with BeautifulSoup on GitHub Actions is easy-peasy. But what about Selenium?? After you jump through som

33 Dec 27, 2022

A Python Selenium library inspired by the Testing Library

Selenium Testing Library Slenium Testing Library (STL) is a Python library for Selenium inspired by Testing-Library. Dependencies Python 3.6, 3.7, 3.8

12 Dec 26, 2022