Package towards building Explainable Forecasting and Nowcasting Models with State-of-the-art Deep Neural Networks and Dynamic Factor Model on Time Series data sets with single line of code. Also, provides utilify facility for time-series signal similarities matching, and removing noise from timeseries signals.

Overview

DeepXF: Explainable Forecasting and Nowcasting with State-of-the-art Deep Neural Networks and Dynamic Factor Model

Also, verify TS signal similarities and Filtering of TS signals with single line of code at ease

deep-xf

pypi: https://pypi.org/project/deep_xf

images/logo.png

Related Blog: https://towardsdatascience.com/interpretable-nowcasting-with-deepxf-using-minimal-code-6b16a76ca52f

Related Blog: https://medium.com/analytics-vidhya/building-explainable-forecasting-models-with-state-of-the-art-deep-neural-networks-using-a-ad3fa5844fef

Related Blog: https://towardsdatascience.com/learning-similarities-between-biomedical-signals-with-deep-siamese-network-7684648e2ba0

Related Blog: https://ajay-arunachalam08.medium.com/denoising-ecg-signals-with-ensemble-of-filters-65919d15afe9

About deep-xf

DeepXF is an open source, low-code python library for forecasting and nowcasting tasks. DeepXF helps in designing complex forecasting and nowcasting models with built-in utility for time series data. One can automatically build interpretable deep forecasting and nowcasting models at ease with this simple, easy-to-use and low-code solution. It enables users to perform end-to-end Proof-Of-Concept (POC) quickly and efficiently. One can build models based on deep neural network such as Recurrent Neural Network (RNN), Long Short Term Memory (LSTM), Gated Recurrent Unit (GRU), Bidirectional RNN/LSTM/GRU (BiRNN/BiLSTM/BiGRU), Spiking Neural Network (SNN), Graph Neural Network (GNN), Transformers, Generative Adversarial Network (GAN), Convolutional Neural Network (CNN), and others. It also provides facility to build nowcast model using Dynamic Factor Model.

images/representation.png

DeepXF is conceived and developed by Ajay Arunachalam - https://www.linkedin.com/in/ajay-arunachalam-4744581a/

Please Note:- This is still by large a work in progress, so always open to your comments and things you feel to be included. Also, if you want to be a contributor, you are always most welcome. The RNN/LSTM/GRU/BiRNN/BiLSTM/BiGRU are already part of the initial version roll-out, while the latter ones (SNN, GNN, Transformers, GAN, CNN, etc.) are work in progress, and will be added soon once the testing is completed.

The library provides (not limited too):-

  • Exploratory Data Analysis with services like profiling, filtering outliers, univariate/multivariate plots, plotly interactive plots, rolling window plots, detecting peaks, etc.
  • Data Preprocessing for Time-series data with services like finding missing, imputing missing, date-time extraction, single timestamp generation, removing unwanted features, etc.
  • Descriptive statistics for the provided time-series data, Normality evaluation, etc.
  • Feature engineering with services like generating time lags, date-time features, one-hot encoding, date-time cyclic features, etc.
  • Finding similarity between homogeneous time-series inputs with Siamese Neural Networks.
  • Denoising time-series input signals.
  • Building Deep Forecasting Model with hyperparameters tuning and leveraging available computational resource (CPU/GPU).
  • Forecasting model performance evaluation with several key metrics
  • Game theory based method to interpret forecasting model results.
  • Building Nowcasting model with Expectation–maximization algorithm
  • Explainable Nowcasting

Who can use deep-xf?

DeepXF is an open-source library ideal for:-

  • Citizen Data Scientists who prefer a low code solution.
  • Experienced Data Scientists who want to increase model accuracy and improve productivity.
  • Data Science Professionals and Consultants involved in building proof-of-concept (poc) projects.
  • Researchers for quick poc prototyping and testing.
  • Students and Teachers.
  • ML Enthusiasts.
  • Learners.

Requirements

  • Python 3.6.x
  • torch[>=1.4.0]
  • NumPy[>=1.9.0]
  • SciPy[>=0.14.0]
  • Scikit-learn[>=0.16]
  • statsmodels[0.12.2]
  • Pandas[>=0.23.0]
  • Matplotlib
  • Seaborn[0.9.0]
  • tqdm
  • shap
  • keras[2.6.0]
  • pandas_profiling[3.1.0]
  • py-ecg-detectors

Quickly Setup package with automation scripts

sudo bash setup.sh

Installation

Using pip:

pip install deep-xf or pip3 install deep-xf or pip install git+git://github.com/ajayarunachalam/Deep_XF
$ git clone https://github.com/ajayarunachalam/Deep_XF
$ cd Deep_XF
$ python setup.py install

Using notebook:

!pip install deep-xf

Using conda:

$ conda install -c conda-forge deep-xf

Getting started

  • FORECASTING DEMO:
# set model config
select_model, select_user_path, select_scaler, forecast_window = Forecast.set_model_config(select_model='rnn', select_user_path='./forecast_folder_path/', select_scaler='minmax', forecast_window=1)

# select hyperparameters
hidden_dim, layer_dim, batch_size, dropout, n_epochs, learning_rate, weight_decay = Forecast.hyperparameter_config(hidden_dim=64,                                                                                                                                                               layer_dim = 3, batch_size=64, dropout = 0.2,                                                                                                                                    n_epochs = 30, learning_rate = 1e-3, weight_decay = 1e-6)

# train model
opt, scaler = Forecast.train(df=df_full_features, target_col='value', split_ratio=0.2, select_model=select_model,              select_scaler=select_scaler, forecast_window=forecast_window, batch_size=batch_size, hidden_dim=hidden_dim, layer_dim=layer_dim,dropout=dropout, n_epochs=n_epochs, learning_rate=learning_rate, weight_decay=weight_decay)

# forecast for user selected period
forecasted_data, ff_full_features, ff_full_features_ = Forecast.forecast(model_df, ts, fc, opt, scaler, period=25, fq='1h', select_scaler=select_scaler,)

# interpret the forecasting result
Helper.explainable_forecast(df_full_features, ff_full_features_, fc, specific_prediction_sample_to_explain=df_full_features.shape[0]+2, input_label_index_value=0, num_labels=1)

Example Illustration

__author__ = 'Ajay Arunachalam'
__version__ = '0.0.1'
__date__ = '7.11.2021'


    from deep_xf.main import *
    from deep_xf.dpp import *
    from deep_xf.forecast_ml import *
    from deep_xf.forecast_ml_extension import *
    from deep_xf.stats import *
    from deep_xf.utility import *
    from deep_xf.denoise import *
    from deep_xf.similarity import *
    df = pd.read_csv('../data/PJME_hourly.csv')
    print(df.shape)
    print(df.columns)
    # set variables
    ts, fc = Forecast.set_variable(ts='Datetime', fc='PJME_MW')
    # get variables
    model_df, orig_df = Helper.get_variable(df, ts, fc)
    # EDA
    ExploratoryDataAnalysis.plot_dataset(df=model_df,fc=fc, title='PJM East (PJME) Region: estimated energy consumption in Megawatts (MW)')
    # Feature Engg
    df_full_features = Features.generate_date_time_features_hour(model_df, ['hour','month','day','day_of_week','week_of_year'])
    # generating cyclic features
    df_full_features = Features.generate_cyclic_features(df_full_features, 'hour', 24, 0)
    df_full_features = Features.generate_cyclic_features(df_full_features, 'day_of_week', 7, 0)
    df_full_features = Features.generate_cyclic_features(df_full_features, 'month', 12, 1)
    df_full_features = Features.generate_cyclic_features(df_full_features, 'week_of_year', 52, 0)
    # holiday feature
    df_full_features = Features.generate_other_related_features(df=df_full_features)
    select_model, select_user_path, select_scaler, forecast_window = Forecast.set_model_config(select_model='rnn', select_user_path='./forecast_folder_path/', select_scaler='minmax', forecast_window=1)

    hidden_dim, layer_dim, batch_size, dropout, n_epochs, learning_rate, weight_decay = Forecast.hyperparameter_config(hidden_dim=64,                                                                                                                                                               layer_dim = 3, batch_size=64, dropout = 0.2,                                                                                                                                    n_epochs = 30, learning_rate = 1e-3, weight_decay = 1e-6)

    opt, scaler = Forecast.train(df=df_full_features, target_col='value', split_ratio=0.2, select_model=select_model,              select_scaler=select_scaler, forecast_window=forecast_window, batch_size=batch_size, hidden_dim=hidden_dim, layer_dim=layer_dim,dropout=dropout, n_epochs=n_epochs, learning_rate=learning_rate, weight_decay=weight_decay)

    forecasted_data, ff_full_features, ff_full_features_ = Forecast.forecast(model_df, ts, fc, opt, scaler, period=25, fq='1h', select_scaler=select_scaler,)

    Helper.explainable_forecast(df_full_features, ff_full_features_, fc, specific_prediction_sample_to_explain=df.shape[0]+1, input_label_index_value=0, num_labels=1)
  • NOWCASTING DEMO:
# set model config
select_model, select_user_path, select_scaler, forecast_window = Forecast.set_model_config(select_model='em', select_user_path='./forecast_folder_path/', select_scaler='minmax', forecast_window=5)

# nowcast for user selected window
nowcast_full_data, nowcast_pred_data = EMModel.nowcast(df_full_features, ts, fc, period=5, fq='1h', forecast_window=forecast_window,    select_model=select_model)

# interpret the nowcasting model result
EMModel.explainable_nowcast(df_full_features, nowcast_pred_data, fc, specific_prediction_sample_to_explain=df.shape[0]+2, input_label_index_value=0, num_labels=1)

Example Illustration

__author__ = 'Ajay Arunachalam'
__version__ = '0.0.1'
__date__ = '7.11.2021'

    from deep_xf.main import *
    from deep_xf.dpp import *
    from deep_xf.forecast_ml import *
    from deep_xf.forecast_ml_extension import *
    from deep_xf.stats import *
    from deep_xf.utility import *
    from deep_xf.denoise import *
    from deep_xf.similarity import *
    df = pd.read_csv('./data/PJME_hourly.csv')
    # set variables
    ts, fc = Forecast.set_variable(ts='Datetime', fc='PJME_MW')
    # get variables
    model_df, orig_df = Helper.get_variable(df, ts, fc)
    select_model, select_user_path, select_scaler, forecast_window = Forecast.set_model_config(select_model='em', select_user_path='./forecast_folder_path/', select_scaler='minmax', forecast_window=5)
    df_full_features = Features.generate_date_time_features_hour(model_df, ['hour','month','day','day_of_week','week_of_year'])
    # generating cyclic features
    df_full_features = Features.generate_cyclic_features(df_full_features, 'hour', 24, 0)
    df_full_features = Features.generate_cyclic_features(df_full_features, 'day_of_week', 7, 0)
    df_full_features = Features.generate_cyclic_features(df_full_features, 'month', 12, 1)
    df_full_features = Features.generate_cyclic_features(df_full_features, 'week_of_year', 52, 0)
    df_full_features = Features.generate_other_related_features(df=df_full_features)
    nowcast_full_data, nowcast_pred_data = EMModel.nowcast(df_full_features, ts, fc, period=5, fq='1h', forecast_window=forecast_window, select_model=select_model)
    EMModel.explainable_nowcast(df_full_features, nowcast_pred_data, fc, specific_prediction_sample_to_explain=df.shape[0]+3, input_label_index_value=0, num_labels=1)

Tested Demo

## Important Links

License

Copyright 2021-2022 Ajay Arunachalam <[email protected]>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. © 2021 GitHub, Inc.

Owner
AjayAru
Data Science Manager; Certified Scrum Master; AWS Certified Cloud Solution Architect; AWS Certified Machine Learning Specialist
AjayAru
Train CNNs for the fruits360 data set in NTOU CS「Machine Vision」class.

CNNs fruits360 Train CNNs for the fruits360 data set in NTOU CS「Machine Vision」class. CNN on a pretrained model Build a CNN on a pretrained model, Res

Ricky Chuang 1 Mar 07, 2022
This repo implements several applications of the proposed generalized Bures-Wasserstein (GBW) geometry on symmetric positive definite matrices.

GBW This repo implements several applications of the proposed generalized Bures-Wasserstein (GBW) geometry on symmetric positive definite matrices. Ap

Andi Han 0 Oct 22, 2021
Codes for AAAI 2022 paper: Context-aware Health Event Prediction via Transition Functions on Dynamic Disease Graphs

Context-Aware-Healthcare Codes for AAAI 2022 paper: Context-aware Health Event Prediction via Transition Functions on Dynamic Disease Graphs Download

LuChang 9 Dec 26, 2022
Implementation of the paper: "SinGAN: Learning a Generative Model from a Single Natural Image"

SinGAN This is an unofficial implementation of SinGAN from someone who's been sitting right next to SinGAN's creator for almost five years. Please ref

35 Nov 10, 2022
Jittor implementation of Recursive-NeRF: An Efficient and Dynamically Growing NeRF

Recursive-NeRF: An Efficient and Dynamically Growing NeRF This is a Jittor implementation of Recursive-NeRF: An Efficient and Dynamically Growing NeRF

33 Nov 30, 2022
Fuzzy Overclustering (FOC)

Fuzzy Overclustering (FOC) In real-world datasets, we need consistent annotations between annotators to give a certain ground-truth label. However, in

2 Nov 08, 2022
Pytorch and Keras Implementations of Hyperspectral Image Classification -- Traditional to Deep Models: A Survey for Future Prospects.

The repository contains the implementations for Hyperspectral Image Classification -- Traditional to Deep Models: A Survey for Future Prospects. Model

Ankur Deria 115 Jan 06, 2023
NEATEST: Evolving Neural Networks Through Augmenting Topologies with Evolution Strategy Training

NEATEST: Evolving Neural Networks Through Augmenting Topologies with Evolution Strategy Training

Göktuğ Karakaşlı 16 Dec 05, 2022
PyTorch Implementation of SSTNs for hyperspectral image classifications from the IEEE T-GRS paper "Spectral-Spatial Transformer Network for Hyperspectral Image Classification: A FAS Framework."

PyTorch Implementation of SSTN for Hyperspectral Image Classification Paper links: SSTN published on IEEE T-GRS. Also, you can directly find the imple

Zilong Zhong 54 Dec 19, 2022
This is the repository of shape matching algorithm Iterative Rotations and Assignments (IRA)

Description This is the repository of shape matching algorithm Iterative Rotations and Assignments (IRA), described in the publication [1]. Directory

MAMMASMIAS Consortium 6 Nov 14, 2022
Python Jupyter kernel using Poetry for reproducible notebooks

Poetry Kernel Use per-directory Poetry environments to run Jupyter kernels. No need to install a Jupyter kernel per Python virtual environment! The id

Pathbird 204 Jan 04, 2023
TorchOk - The toolkit for fast Deep Learning experiments in Computer Vision

TorchOk - The toolkit for fast Deep Learning experiments in Computer Vision

52 Dec 23, 2022
This repo holds the code of TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation

TransFuse This repo holds the code of TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation Requirements Pytorch=1.6.0, 1.9.0 (=1.

Rayicer 93 Dec 19, 2022
The official TensorFlow implementation of the paper Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition

Action Transformer A Self-Attention Model for Short-Time Human Action Recognition This repository contains the official TensorFlow implementation of t

PIC4SeRCentre 20 Jan 03, 2023
ICRA 2021 - Robust Place Recognition using an Imaging Lidar

Robust Place Recognition using an Imaging Lidar A place recognition package using high-resolution imaging lidar. For best performance, a lidar equippe

Tixiao Shan 293 Dec 27, 2022
Source Code for Simulations in the Publication "Can the brain use waves to solve planning problems?"

Code for Simulations in the Publication Can the brain use waves to solve planning problems? Installing Required Python Packages Please use Python vers

EMD Group 2 Jul 01, 2022
Subdivision-based Mesh Convolutional Networks

Subdivision-based Mesh Convolutional Networks The official implementation of SubdivNet in our paper, Subdivion-based Mesh Convolutional Networks Requi

Zheng-Ning Liu 181 Dec 28, 2022
Pytorch tutorials for Neural Style transfert

PyTorch Tutorials This tutorial is no longer maintained. Please use the official version: https://pytorch.org/tutorials/advanced/neural_style_tutorial

Alexis David Jacq 135 Jun 26, 2022
Unofficial TensorFlow implementation of the Keyword Spotting Transformer model

Keyword Spotting Transformer This is the unofficial TensorFlow implementation of the Keyword Spotting Transformer model. This model is used to train o

Intelligent Machines Limited 8 May 11, 2022
Efficient Sparse Attacks on Videos using Reinforcement Learning

EARL This repository provides a simple implementation of the work "Efficient Sparse Attacks on Videos using Reinforcement Learning" Example: Demo: Her

12 Dec 05, 2021