Testbed of AI Systems Quality Management

Related tags

Deep Learningpythondockerflaskairflowvuejspostgresqljupyter-notebookqualitiesmanaging-ai
Overview

qunomon

Description

A testbed for testing and managing AI system qualities.

Demo

Sorry. Not deployment public server at alpha version.

Requirement

Installation prerequisites

Support os is Windows10 Pro and macOS.

  • Windows10 Pro 1909 later
  • macOS v10.15 later

Installation

Usage

1.launch

Execute the following command as root of this repository.

docker-compose up

2.access web browser

http://127.0.0.1:8888/

Development for windows

Installation

1.PackageManager

  • Launch powershell with administrator permission.

  • powershell

    Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

2.Python

  • powershell 
    cinst python --version=3.6.8 -y

Setup python virtual environment for Backend

1.go to the source you checked out and create a virtual environment

  • launch command prompt
cd {checkout_dir}\src\backend
python -m venv venv

2.virtual environment activate

.\venv\Scripts\activate

3.install python package

pip install -r requirements_dev.txt

Setup python virtual environment for IP

1.go to the source you checked out and create a virtual environment

  • launch command prompt
cd {checkout_dir}\src\integration-provider
python -m venv venv

2.virtual environment activate

.\venv\Scripts\activate

3.install python package

pip install -r constraints.txt

launch by without container

1.execute bat file

start_up.bat

2.checking web browser

http://127.0.0.1:8080/

3.checking Backend

  • powershell 
    curl http://127.0.0.1:5000/qai-testbed/api/0.0.1/health-check

4.checking IP

  • powershell 
    curl http://127.0.0.1:6000/qai-ip/api/0.0.1/health-check

Contribution

Bug reports and pull requests are welcome on GitHub at aistairc/qunomon.

Disclaimer

qunomon is an OSS and alpha version. so qunomon may cause damage to your system and data. You agree to use it at your own risk.

License

Apache License Version 2.0

Author

AIST

You might also like...
Seeing Dynamic Scene in the Dark: High-Quality Video Dataset with Mechatronic Alignment (ICCV2021)
Seeing Dynamic Scene in the Dark: High-Quality Video Dataset with Mechatronic Alignment (ICCV2021)

Seeing Dynamic Scene in the Dark: High-Quality Video Dataset with Mechatronic Alignment This is a pytorch project for the paper Seeing Dynamic Scene i

DV Lab 21 Nov 28, 2022
Official PyTorch implementation of the paper
Official PyTorch implementation of the paper "Recycling Discriminator: Towards Opinion-Unaware Image Quality Assessment Using Wasserstein GAN", accepted to ACM MM 2021 BNI Track.

RecycleD Official PyTorch implementation of the paper "Recycling Discriminator: Towards Opinion-Unaware Image Quality Assessment Using Wasserstein GAN

Yunan Zhu 23 Nov 5, 2022
[ICCV 2021] Group-aware Contrastive Regression for Action Quality Assessment
[ICCV 2021] Group-aware Contrastive Regression for Action Quality Assessment

CoRe Created by Xumin Yu*, Yongming Rao*, Wenliang Zhao, Jiwen Lu, Jie Zhou This is the PyTorch implementation for ICCV paper Group-aware Contrastive

Xumin Yu 31 Dec 24, 2022
NUANCED is a user-centric conversational recommendation dataset that contains 5.1k annotated dialogues and 26k high-quality user turns.
NUANCED is a user-centric conversational recommendation dataset that contains 5.1k annotated dialogues and 26k high-quality user turns.

NUANCED: Natural Utterance Annotation for Nuanced Conversation with Estimated Distributions Overview NUANCED is a user-centric conversational recommen

Facebook Research 18 Dec 28, 2021
Cross Quality LFW: A database for Analyzing Cross-Resolution Image Face Recognition in Unconstrained Environments

Cross-Quality Labeled Faces in the Wild (XQLFW) Here, we release the database, evaluation protocol and code for the following paper: Cross Quality LFW

Martin Knoche 10 Dec 12, 2022
A data annotation pipeline to generate high-quality, large-scale speech datasets with machine pre-labeling and fully manual auditing.
A data annotation pipeline to generate high-quality, large-scale speech datasets with machine pre-labeling and fully manual auditing.

About This repository provides data and code for the paper: Scalable Data Annotation Pipeline for High-Quality Large Speech Datasets Development (subm

Appen Repos 86 Dec 7, 2022
PyTorch Implementation of PortaSpeech: Portable and High-Quality Generative Text-to-Speech
PyTorch Implementation of PortaSpeech: Portable and High-Quality Generative Text-to-Speech

PortaSpeech - PyTorch Implementation PyTorch Implementation of PortaSpeech: Portable and High-Quality Generative Text-to-Speech. Model Size Module Nor

Keon Lee 279 Jan 4, 2023
Code for the paper SphereRPN: Learning Spheres for High-Quality Region Proposals on 3D Point Clouds Object Detection, ICIP 2021.

SphereRPN Code for the paper SphereRPN: Learning Spheres for High-Quality Region Proposals on 3D Point Clouds Object Detection, ICIP 2021. Authors: Th

Thang Vu 15 Dec 2, 2022
Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Zhengzhong Tu 5 Sep 16, 2022
Releases(0.1.15)

  • 0.1.15(Jun 25, 2021)

  • 0.1.14(Jun 8, 2021)

    Added

    #1462 SHAP AITの実装とテスト alyz_regression_shap_0.1 #1485 SHAP AIT plots_scatterの出力figにタイトル（カラムを対象）追加

    Fixed

    #1492 Dependabot alerts対応（urllib3） #1476 Dependabot alerts対応（TensorFlow2.4系から変更） #1465 AITパラメータ見直し（eval_adversarial_example_acc_test_tf2.3_0.1）

    Source code(tar.gz)
    Source code(zip)

  • 0.1.13(May 26, 2021)

    Added

    #1434 クライテリア範囲外でTDを作成できないようにする(バックエンド) #1435 クライテリア範囲外でTDを作成できないようにする(フロントエンド) #1436 パラメータ範囲外でTDを作成できないようにする(フロントエンド) #1437 パラメータ範囲外でTDを作成できないようにする(バックエンド) #1438 インベントリチェック　警告ポップアップを表示する（フロントエンド） #1440 インベントリチェック　ファイルフォーマットチェック(一般)

    Fixed

    #1423 AITパラメータ見直し（eval_dnn_coverage_tf1.13_0.1） #1425 AITパラメータ見直し（eval_mnist_acc_tf2.3_0.1） #1454 DependencyAlert解消(5/12)

    Source code(tar.gz)
    Source code(zip)

  • 0.1.12(May 12, 2021)

    Added

    #1416 インベントリチェック　ファイル存在チェック #1421 インベントリチェック　TD実行時ハッシュチェック

    Fixed

    #1362 #1213の変更部分をテストコードに反映させる #1403 AIT発生エラー見直し AIT-SDK入れ替え #1432 GET TestRunnerでエンコードエラーログが出力される #1442 docker起動でインベントリ登録ができない #1447 DependencyAlert解消(5/10) #1452 pipインストールモジュールのバージョンを固定化する

    Source code(tar.gz)
    Source code(zip)

  • 0.1.11(Apr 28, 2021)

    Added

    #1370 AITの更新 (AITのパラメータ上限下限を表示する) #1374 ait-installerの更新 (AITのパラメータ上限下限を表示する)

    Fixed

    #1402 AIT発生エラー見直し AIT-SDK修正 #1404 AIT発生エラー見直し IP修正 #1405 AIT発生エラー見直し バックエンド修正 #1406 AIT発生エラー見直し フロントエンド修正 #1416 AIT発生エラー見直し AIT-SDK修正（出力先フォルダがない場合に対応）

    Source code(tar.gz)
    Source code(zip)

  • 0.1.10(Apr 14, 2021)

    Added

    #1349 [TF3]MLComponent一覧画面でMLComponentを削除できるようにしたい

    Fixed

    #1385 DependencyAlert解消(3/26) #1391 DependencyAlert解消(4/2) #1361 #1335 により変更された部分をWEB API仕様書に反映をさせる

    Source code(tar.gz)
    Source code(zip)

  • 0.1.9(Mar 26, 2021)

    Added

    #1335 AITのパラメータ上限下限を表示する #144 TestDescription一覧画面の日付指定をカレンダーを用いて行う機能の実装

    Fixed

    なし

    Source code(tar.gz)
    Source code(zip)

  • 0.1.8(Mar 12, 2021)

    Added

    #1212 TD詳細画面-グラフを複数選択して追加したい #1340 TD詳細画面でairflowのログダウンロードURLリンクを表示する #1342 グラフ複数選択時に未登録のものだけを登録したい #1347 [TF3]TD一覧画面でTDを削除できるようにしたい

    Fixed

    #1331 何も選択していない状態で「add to Report」ボタンを押下できてしまう #1337 活性化判定をcheckAddBTNActiveメソッドで対応させるよう処理を統一 #1345 [TD詳細画面]追加グラフの数チェック不整合 #1348 [TD編集画面1]TDの再編集時にTD名のテキストボックスが入力1文字ごとにフォーカスが外れる #1336 docker-compose実行時に、ait-installerが実行されてない

    Source code(tar.gz)
    Source code(zip)

  • 0.1.7(Feb 26, 2021)

  • 0.1.6(Feb 12, 2021)

    Added

    #1200 AIF360の指標を取り込んだAITを作成する #1211 TD詳細画面-どのグラフを選択中か分かるようにしたい #1248 ait.manifest.jsonのreport.measuresにminとmaxを書く

    Fixed

    #1300 jupyter新バージョン3.X以後、AITのset_ait_descriptionにUnicodeEncodeError (漢字、など) #1305 TDでのレポート使用グラフを一つ削除すると、ソートがリセットされる #1312 Dependency alert解消(2/4)

    Source code(tar.gz)
    Source code(zip)

  • 0.1.5(Jan 29, 2021)

    Added

    #1199 TDの品質指標に何を入れれば良いか分かりにくい問題を解消する #1213 [サマリ]manifestのresources,downloadsからpathを削除する

    Fixed

    #1262 eval_bdd100k_aicc_tf2.3のリソース「all_label_accuracy_csv」がタイプ「text」になっている #1272 ローカルにAITイメージがない状態で実行するとairflowでエラーになる #1277 Dependabot alerts解消(2021/1/15)

    Source code(tar.gz)
    Source code(zip)

  • 0.1.4(Jan 15, 2021)

    Added

    #1213 manifestのresources,downloadsからpathを削除する

    Fixed

    #1208 レポートのレーダーチャートが、品質特性2以下だと数量が判別できない #1254 レポートのレーダーチャートの表示範囲が５で固定 #1260 Dependabot alerts解消(2021/1/8)

    Source code(tar.gz)
    Source code(zip)

  • 0.1.3(Dec 25, 2020)

    Added

    #1166 qlib新規作成

    Fixed

    #1183 TestDescriptionの中で大量の画像を扱うと画面が応答しない #1198 フロントエンド誤字修正 #1203 AITでresoucesに大量のデータをセットすると、TestDescription詳細画面やレポート出力が応答しない #1242 measures無しのAITを登録するとQualityDimensionが反映されない

    Source code(tar.gz)
    Source code(zip)

  • 0.1.2(Dec 10, 2020)

    Added

    #1171 インベントリの選択方法を改善する #1173 レポートのサマリでTD0件の品質特性は出力対象にしないようにする

    Fixed

    #1187 レポート出力時に2.1のレーダーチャートの項目名が長すぎると途中で切れる #1184 airflowのdocker buildが失敗する

    Source code(tar.gz)
    Source code(zip)

  • 0.1.1(Nov 27, 2020)

    Added

    #1071 確認ダイアログの多言語化対応 #1126 作成したAITをtestbedにdeployするツールが必要

    Fixed

    #1099 ブラウザバック、リロードでエラーが発生する画面がある #1123 内部品質名称を英語に変更する #1115 2つ以上あるmeasureのうち、一つだけチェックをいれてTDを作成するとエラーが発生する #1125 READMEの記述を修正（qai-testbed → qunomon） #1163 dag配下のフォルダを削除する #1156 docker-airflowのDockerfileの修正 #1157 Github security alert への対応

    Source code(tar.gz)
    Source code(zip)

  • 0.1.0(Nov 13, 2020)

Owner
AIST AIRC
AIST AIRC
GitHub Repository https://aistairc.github.io/qunomon/
This is the repository for the NeurIPS-21 paper [Contrastive Graph Poisson Networks: Semi-Supervised Learning with Extremely Limited Labels].

CGPN This is the repository for the NeurIPS-21 paper [Contrastive Graph Poisson Networks: Semi-Supervised Learning with Extremely Limited Labels]. Req

10 Sep 12, 2022
This is the code for Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning

This is the code for Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning It includes /bert, which is the original BERT repos

Mitchell Gordon 11 Nov 15, 2022
MonoScene: Monocular 3D Semantic Scene Completion

MonoScene: Monocular 3D Semantic Scene Completion MonoScene: Monocular 3D Semantic Scene Completion] [arXiv + supp] | [Project page] Anh-Quan Cao, Rao

298 Jan 08, 2023
Start-to-finish tutorial for interactive music co-creation in PyTorch and Tensorflow.js

Start-to-finish tutorial for interactive music co-creation in PyTorch and Tensorflow.js

Chris Donahue 98 Dec 14, 2022
Neural Logic Inductive Learning

Neural Logic Inductive Learning This is the implementation of the Neural Logic Inductive Learning model (NLIL) proposed in the ICLR 2020 paper: Learn

36 Nov 28, 2022
3D-aware GANs based on NeRF (arXiv).

CIPS-3D This repository will contain the code of the paper, CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis.

Peterou 563 Dec 31, 2022
PyTorch implementation of DeepDream algorithm

neural-dream This is a PyTorch implementation of DeepDream. The code is based on neural-style-pt. Here we DeepDream a photograph of the Golden Gate Br

121 Nov 05, 2022
An implementation demo of the ICLR 2021 paper Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks in PyTorch.

Neural Attention Distillation This is an implementation demo of the ICLR 2021 paper Neural Attention Distillation: Erasing Backdoor Triggers from Deep

Yige-Li 84 Jan 04, 2023
Determined: Deep Learning Training Platform

Determined: Deep Learning Training Platform Determined is an open-source deep learning training platform that makes building models fast and easy. Det

Determined AI 2k Dec 31, 2022
Probabilistic Entity Representation Model for Reasoning over Knowledge Graphs

Implementation for the paper: Probabilistic Entity Representation Model for Reasoning over Knowledge Graphs, Nurendra Choudhary, Nikhil Rao, Sumeet Ka

Nurendra Choudhary 8 Nov 15, 2022
Genetic feature selection module for scikit-learn

sklearn-genetic Genetic feature selection module for scikit-learn Genetic algorithms mimic the process of natural selection to search for optimal valu

Manuel Calzolari 260 Dec 14, 2022
Rename Images with Auto Generated Neural Image Captions

Recaption Images with Generated Neural Image Caption Example Usage: Commandline: Recaption all images from folder /home/feng/Downloads/images to folde

feng wang 3 May 01, 2022
Research on controller area network Intrusion Detection Systems

Group members information Member 1: Lixue Liang Member 2: Yuet Lee Chan Member 3: Xinruo Zhang Member 4: Yifei Han User Manual Generate Attack Packets

Roche 4 Aug 30, 2022
A Research-oriented Federated Learning Library and Benchmark Platform for Graph Neural Networks. Accepted to ICLR'2021 - DPML and MLSys'21 - GNNSys workshops.

FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks A Research-oriented Federated Learning Library and Benchmark Platform

FedML-AI 175 Dec 01, 2022
ICON: Implicit Clothed humans Obtained from Normals (CVPR 2022)

ICON: Implicit Clothed humans Obtained from Normals Yuliang Xiu · Jinlong Yang · Dimitrios Tzionas · Michael J. Black CVPR 2022 News 🚩 [2022/04/26] H

Yuliang Xiu 1.1k Jan 04, 2023
COVID-VIT: Classification of Covid-19 from CT chest images based on vision transformer models

COVID-ViT COVID-VIT: Classification of Covid-19 from CT chest images based on vision transformer models This code is to response to te MIA-COV19 compe

17 Dec 30, 2022
Real-Time-Student-Attendence-System - Real Time Student Attendence System

Real-Time-Student-Attendence-System The Student Attendance Management System Pro

Rounak Das 1 Feb 15, 2022
Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

Apache MXNet (incubating) for Deep Learning Master Docs License Apache MXNet (incubating) is a deep learning framework designed for both efficiency an

ROCm Software Platform 29 Nov 16, 2022
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Created by Charles R. Qi, Hao Su, Kaichun Mo, Leonidas J. Guibas from Sta

Charles R. Qi 4k Dec 30, 2022
Mini Software that give reminder to drink water as per your weight.

Water Notification Desktop Python The Mini Software built in Python (tkinter) that will remind you to drink water on specific time span based on your

Om Jogani 5 Dec 16, 2022