LONG-TERM SERIES FORECASTING WITH QUERYSELECTOR – EFFICIENT MODEL OF SPARSEATTENTION

Overview

Query Selector

Here you can find code and data loaders for the paper https://arxiv.org/pdf/2107.08687v1.pdf . Query Selector is a novel approach to sparse attention Transformer algorithm that is especially suitable for long term time series forecasting

Depencency

Python            3.7.9
deepspeed         0.4.0
numpy             1.20.3
pandas            1.2.4
scipy             1.6.3
tensorboardX      1.8
torch             1.7.1
torchaudio        0.7.2
torchvision       0.8.2
tqdm              4.61.0

Results on ETT dataset

Univariate

Data Prediction len Informer MSE Informer MAE Trans former MSE Trans former MAE Query Selector MSE Query Selector MAE MSE ratio
ETTh1 24 0.0980 0.2470 0.0548 0.1830 0.0436 0.1616 0.445
ETTh1 48 0.1580 0.3190 0.0740 0.2144 0.0721 0.2118 0.456
ETTh1 168 0.1830 0.3460 0.1049 0.2539 0.0935 0.2371 0.511
ETTh1 336 0.2220 0.3870 0.1541 0.3201 0.1267 0.2844 0.571
ETTh1 720 0.2690 0.4350 0.2501 0.4213 0.2136 0.3730 0.794
ETTh2 24 0.0930 0.2400 0.0999 0.2479 0.0843 0.2239 0.906
ETTh2 48 0.1550 0.3140 0.1218 0.2763 0.1117 0.2622 0.721
ETTh2 168 0.2320 0.3890 0.1974 0.3547 0.1753 0.3322 0.756
ETTh2 336 0.2630 0.4170 0.2191 0.3805 0.2088 0.3710 0.794
ETTh2 720 0.2770 0.4310 0.2853 0.4340 0.2585 0.4130 0.933
ETTm1 24 0.0300 0.1370 0.0143 0.0894 0.0139 0.0870 0.463
ETTm1 48 0.0690 0.2030 0.0328 0.1388 0.0342 0.1408 0.475
ETTm1 96 0.1940 0.2030 0.0695 0.2085 0.0702 0.2100 0.358
ETTm1 288 0.4010 0.5540 0.1316 0.2948 0.1548 0.3240 0.328
ETTm1 672 0.5120 0.6440 0.1728 0.3437 0.1735 0.3427 0.338

Multivariate

Data Prediction len Informer MSE Informer MAE Trans former MSE Trans former MAE Query Selector MSE Query Selector MAE MSE ratio
ETTh1 24 0.5770 0.5490 0.4496 0.4788 0.4226 0.4627 0.732
ETTh1 48 0.6850 0.6250 0.4668 0.4968 0.4581 0.4878 0.669
ETTh1 168 0.9310 0.7520 0.7146 0.6325 0.6835 0.6088 0.734
ETTh1 336 1.1280 0.8730 0.8321 0.7041 0.8503 0.7039 0.738
ETTh1 720 1.2150 0.8960 1.1080 0.8399 1.1150 0.8428 0.912
ETTh2 24 0.7200 0.6650 0.4237 0.5013 0.4124 0.4864 0.573
ETTh2 48 1.4570 1.0010 1.5220 0.9488 1.4074 0.9317 0.966
ETTh2 168 3.4890 1.5150 1.6225 0.9726 1.7385 1.0125 0.465
ETTh2 336 2.7230 1.3400 2.6617 1.2189 2.3168 1.1859 0.851
ETTh2 720 3.4670 1.4730 3.1805 1.3668 3.0664 1.3084 0.884
ETTm1 24 0.3230 0.3690 0.3150 0.3886 0.3351 0.3875 0.975
ETTm1 48 0.4940 0.5030 0.4454 0.4620 0.4726 0.4702 0.902
ETTm1 96 0.6780 0.6140 0.4641 0.4823 0.4543 0.4831 0.670
ETTm1 288 1.0560 0.7860 0.6814 0.6312 0.6185 0.5991 0.586
ETTm1 672 1.1920 0.9260 1.1365 0.8572 1.1273 0.8412 0.946

State Of Art

PWC

PWC

PWC

PWC

PWC

PWC

PWC

PWC

PWC

PWC

Citation

@misc{klimek2021longterm,
      title={Long-term series forecasting with Query Selector -- efficient model of sparse attention}, 
      author={Jacek Klimek and Jakub Klimek and Witold Kraskiewicz and Mateusz Topolewski},
      year={2021},
      eprint={2107.08687},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

Contact

If you have any questions please contact us by email - [email protected]

Owner
MORAI
MORAI
[CVPR2021] The source code for our paper 《Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning》.

TBE The source code for our paper "Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Le

Jinpeng Wang 150 Dec 28, 2022
Part-aware Measurement for Robust Multi-View Multi-Human 3D Pose Estimation and Tracking

Part-aware Measurement for Robust Multi-View Multi-Human 3D Pose Estimation and Tracking Part-Aware Measurement for Robust Multi-View Multi-Human 3D P

19 Oct 27, 2022
PyTorch code for 'Efficient Single Image Super-Resolution Using Dual Path Connections with Multiple Scale Learning'

Efficient Single Image Super-Resolution Using Dual Path Connections with Multiple Scale Learning This repository is for EMSRDPN introduced in the foll

7 Feb 10, 2022
TensorFlow-based implementation of "Pyramid Scene Parsing Network".

PSPNet_tensorflow Important Code is fine for inference. However, the training code is just for reference and might be only used for fine-tuning. If yo

HsuanKung Yang 323 Dec 20, 2022
Predicting Event Memorability from Contextual Visual Semantics

Predicting Event Memorability from Contextual Visual Semantics

0 Oct 06, 2021
Meta Language-Specific Layers in Multilingual Language Models

Meta Language-Specific Layers in Multilingual Language Models This repo contains the source codes for our paper On Negative Interference in Multilingu

Zirui Wang 20 Feb 13, 2022
Pytorch implementation of our paper under review -- 1xN Pattern for Pruning Convolutional Neural Networks

1xN Pattern for Pruning Convolutional Neural Networks (paper) . This is Pytorch re-implementation of "1xN Pattern for Pruning Convolutional Neural Net

Mingbao Lin (林明宝) 29 Nov 29, 2022
[NeurIPS-2021] Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data

MosaicKD Code for NeurIPS-21 paper "Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data" 1. Motivation Natural images share common l

ZJU-VIPA 37 Nov 10, 2022
The implementation our EMNLP 2021 paper "Enhanced Language Representation with Label Knowledge for Span Extraction".

LEAR The implementation our EMNLP 2021 paper "Enhanced Language Representation with Label Knowledge for Span Extraction". See below for an overview of

杨攀 93 Jan 07, 2023
ADSPM: Attribute-Driven Spontaneous Motion in Unpaired Image Translation

ADSPM: Attribute-Driven Spontaneous Motion in Unpaired Image Translation This repository provides a PyTorch implementation of ADSPM. Requirements Pyth

24 Jul 24, 2022
Contrastive Multi-View Representation Learning on Graphs

Contrastive Multi-View Representation Learning on Graphs This work introduces a self-supervised approach based on contrastive multi-view learning to l

Kaveh 208 Dec 23, 2022
This repository contains small projects related to Neural Networks and Deep Learning in general.

ILearnDeepLearning.py Description People say that nothing develops and teaches you like getting your hands dirty. This repository contains small proje

Piotr Skalski 1.2k Dec 22, 2022
Magisk module to enable hidden features on Android 12 Developer Preview 1.

Android 12 Extensions This is a Magisk module that enables hidden features on Android 12 Developer Preview 1. Features Scrolling screenshots Wallpaper

Danny Lin 384 Jan 06, 2023
🏖 Keras Implementation of Painting outside the box

Keras implementation of Image OutPainting This is an implementation of Painting Outside the Box: Image Outpainting paper from Standford University. So

Bendang 1.1k Dec 10, 2022
A Decentralized Omnidirectional Visual-Inertial-UWB State Estimation System for Aerial Swar.

Omni-swarm A Decentralized Omnidirectional Visual-Inertial-UWB State Estimation System for Aerial Swarm Introduction Omni-swarm is a decentralized omn

HKUST Aerial Robotics Group 99 Dec 23, 2022
PyTorch-based framework for Deep Hedging

PFHedge: Deep Hedging in PyTorch PFHedge is a PyTorch-based framework for Deep Hedging. PFHedge Documentation Neural Network Architecture for Efficien

139 Dec 30, 2022
Sign Language Transformers (CVPR'20)

Sign Language Transformers (CVPR'20) This repo contains the training and evaluation code for the paper Sign Language Transformers: Sign Language Trans

Necati Cihan Camgoz 164 Dec 30, 2022
Semantic similarity computation with different state-of-the-art metrics

Semantic similarity computation with different state-of-the-art metrics Description • Installation • Usage • License Description TaxoSS is a semantic

6 Jun 22, 2022
Code for the paper "Attention Approximates Sparse Distributed Memory"

Attention Approximates Sparse Distributed Memory - Codebase This is all of the code used to run analyses in the paper "Attention Approximates Sparse D

Trenton Bricken 14 Dec 05, 2022
A minimalist implementation of score-based diffusion model

sdeflow-light This is a minimalist codebase for training score-based diffusion models (supporting MNIST and CIFAR-10) used in the following paper "A V

Chin-Wei Huang 89 Dec 20, 2022