A Deep Reinforcement Learning Framework for Stock Market Trading

Overview

DQN-Trading

This is a framework based on deep reinforcement learning for stock market trading. This project is the implementation code for the two papers:

The deep reinforcement learning algorithm used here is Deep Q-Learning.

Acknowledgement

Requirements

Install pytorch using the following commands. This is for CUDA 11.1 and python 3.8:

pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
  • python = 3.8
  • pandas = 1.3.2
  • numpy = 1.21.2
  • matplotlib = 3.4.3
  • cython = 0.29.24
  • scikit-learn = 0.24.2

TODO List

  • Right now this project does not have a code for getting user hyper-parameters from terminal and running the code. We preferred writing a jupyter notebook (Main.ipynb) in which you can set the input data, the model, along with setting the hyper-parameters.

  • The project also does not have a code to do Hyper-parameter search (its easy to implement).

  • You can also set the seed for running the experiments in the original code for training the models.

Developers' Guidelines

In this section, I briefly explain different parts of the project and how to change each. The data for the project downloaded from Yahoo Finance where you can search for a specific market there and download your data under the Historical Data section. Then you create a directory with the name of the stock under the data directory and put the .csv file there.

The DataLoader directory contains files to process the data and interact with the RL agent. The DataLoader.py loads the data given the folder name under Data folder and also the name of the .csv file. For this, you should use the YahooFinanceDataLoader class for using data downloaded from Yahoo Finance.

The Data.py file is the environment that interacts with the RL agent. This file contains all the functionalities used in a standard RL environment. For each agent, I developed a class inherited from the Data class that only differs in the state space (consider that states for LSTM and convolutional models are time-series, while for other models are simple OHLCs). In DataForPatternBasedAgent.py the states are patterns extracted using rule-based methods in technical analysis. In DataAutoPatternExtractionAgent.py states are Open, High, Low, and Close prices (plus some other information about the candle-stick like trend, upper shadow, lower shadow, etc). In DataSequential.py as it is obvious from the name, the state space is time-series which is used in both LSTM and Convolutional models. DataSequencePrediction.py was an idea for feeding states that have been predicted using an LSTM model to the RL agent. This idea is raw and needs to be developed.

Where ever we used encoder-decoder architecture, the decoder is the DQN agent whose neural network is the same across all the models.

The DeepRLAgent directory contains the DQN model without encoder part (VanillaInput) whose data loader corresponds to DataAutoPatternExtractionAgent.py and DataForPatternBasedAgent.py; an encoder-decoder model where the encoder is a 1d convolutional layer added to the decoder which is DQN agent under SimpleCNNEncoder directory; an encoder-decoder model where encoder is a simple MLP model and the decoder is DQN agent under MLPEncoder directory.

Under the EncoderDecoderAgent there exist all the time-series models, including CNN (two-layered 1d CNN as encoder), CNN2D (a single-layered 2d CNN as encoder), CNN-GRU (the encoder is a 1d CNN over input and then a GRU on the output of CNN. The purpose of this model is that CNN extracts features from each candlestick, thenGRU extracts temporal dependency among those extracted features.), CNNAttn (A two-layered 1d CNN with attention layer for putting higher emphasis on specific parts of the features extracted from the time-series data), and a GRU encoder which extracts temporal relations among candles. All of these models use DataSequential.py file as environment.

For running each agent, please refer to the Main.py file for instructions on how to run each agent and feed data. The Main.py file also has code for plotting results.

The Objects directory contains the saved models from our experiments for each agent.

The PatternDetectionCandleStick directory contains Evaluation.py file which has all the evaluation metrics used in the paper. This file receives the actions from the agents and evaluate the result of the strategy offered by each agent. The LabelPatterns.py uses rule-based methods to generate buy or sell signals. Also Extract.py is another file used for detecting wellknown candlestick patterns.

RLAgent directory is the implementation of the traditional RL algorithm SARSA-λ using cython. In order to run that in the Main.ipynb you should first build the cython file. In order to do that, run the following script inside it's directory in terminal:

python setup.py build_ext --inplace

This works for both linux and windows.

For more information on the algorithms and models, please refer to the original paper. You can find them in the references.

If you had any questions regarding the paper, code, or you wanted to contribute, please send me an email: [email protected]

References

@article{taghian2020learning,
  title={Learning financial asset-specific trading rules via deep reinforcement learning},
  author={Taghian, Mehran and Asadi, Ahmad and Safabakhsh, Reza},
  journal={arXiv preprint arXiv:2010.14194},
  year={2020}
}

@article{taghian2021reinforcement,
  title={A Reinforcement Learning Based Encoder-Decoder Framework for Learning Stock Trading Rules},
  author={Taghian, Mehran and Asadi, Ahmad and Safabakhsh, Reza},
  journal={arXiv preprint arXiv:2101.03867},
  year={2021}
}
Non-Homogeneous Poisson Process Intensity Modeling and Estimation using Measure Transport

Non-Homogeneous Poisson Process Intensity Modeling and Estimation using Measure Transport This GitHub page provides code for reproducing the results i

Andrew Zammit Mangion 1 Nov 08, 2021
MIMIC Code Repository: Code shared by the research community for the MIMIC-III database

MIMIC Code Repository The MIMIC Code Repository is intended to be a central hub for sharing, refining, and reusing code used for analysis of the MIMIC

MIT Laboratory for Computational Physiology 1.8k Dec 26, 2022
Automates Machine Learning Pipeline with Feature Engineering and Hyper-Parameters Tuning :rocket:

MLJAR Automated Machine Learning Documentation: https://supervised.mljar.com/ Source Code: https://github.com/mljar/mljar-supervised Table of Contents

MLJAR 2.4k Dec 31, 2022
NeuroLKH: Combining Deep Learning Model with Lin-Kernighan-Helsgaun Heuristic for Solving the Traveling Salesman Problem

NeuroLKH: Combining Deep Learning Model with Lin-Kernighan-Helsgaun Heuristic for Solving the Traveling Salesman Problem Liang Xin, Wen Song, Zhiguang

xinliangedu 33 Dec 27, 2022
CUAD

Contract Understanding Atticus Dataset This repository contains code for the Contract Understanding Atticus Dataset (CUAD), a dataset for legal contra

The Atticus Project 273 Dec 17, 2022
Weak-supervised Visual Geo-localization via Attention-based Knowledge Distillation

Weak-supervised Visual Geo-localization via Attention-based Knowledge Distillation Introduction WAKD is a PyTorch implementation for our ICPR-2022 pap

2 Oct 20, 2022
Anderson Acceleration for Deep Learning

Anderson Accelerated Deep Learning (AADL) AADL is a Python package that implements the Anderson acceleration to speed-up the training of deep learning

Oak Ridge National Laboratory 7 Nov 24, 2022
(NeurIPS 2021) Pytorch implementation of paper "Re-ranking for image retrieval and transductive few-shot classification"

SSR (NeurIPS 2021) Pytorch implementation of paper "Re-ranking for image retrieval and transductivefew-shot classification" [Paper] [Project webpage]

xshen 29 Dec 06, 2022
PyTorch implementation of SmoothGrad: removing noise by adding noise.

SmoothGrad implementation in PyTorch PyTorch implementation of SmoothGrad: removing noise by adding noise. Vanilla Gradients SmoothGrad Guided backpro

SSKH 143 Jan 05, 2023
Generic template to bootstrap your PyTorch project with PyTorch Lightning, Hydra, W&B, and DVC.

NN Template Generic template to bootstrap your PyTorch project. Click on Use this Template and avoid writing boilerplate code for: PyTorch Lightning,

Luca Moschella 520 Dec 30, 2022
Active and Sample-Efficient Model Evaluation

Active Testing: Sample-Efficient Model Evaluation Hi, good to see you here! đź‘‹ This is code for "Active Testing: Sample-Efficient Model Evaluation". P

Jannik Kossen 19 Oct 30, 2022
Code for our paper "Graph Pre-training for AMR Parsing and Generation" in ACL2022

AMRBART An implementation for ACL2022 paper "Graph Pre-training for AMR Parsing and Generation". You may find our paper here (Arxiv). Requirements pyt

xfbai 60 Jan 03, 2023
Object Detection using YOLO from PyImageSearch

Object Detection using YOLO from PyImageSearch By applying object detection, you’ll not only be able to determine what is in an image, but also where

Mohamed NIANG 1 Feb 09, 2022
Adaptive Graph Convolution for Point Cloud Analysis

Adaptive Graph Convolution for Point Cloud Analysis This repository contains the implementation of AdaptConv for point cloud analysis. Adaptive Graph

64 Dec 21, 2022
Official implementation of the paper 'Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolution'

DASR Paper Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolution Jie Liang, Hui Zeng, and Lei Zhang. In arxiv preprint. Abs

81 Dec 28, 2022
Intel® Neural Compressor is an open-source Python library running on Intel CPUs and GPUs

Intel® Neural Compressor targeting to provide unified APIs for network compression technologies, such as low precision quantization, sparsity, pruning, knowledge distillation, across different deep l

Intel Corporation 846 Jan 04, 2023
PyTorch Implementation of CycleGAN and SSGAN for Domain Transfer (Minimal)

MNIST-to-SVHN and SVHN-to-MNIST PyTorch Implementation of CycleGAN and Semi-Supervised GAN for Domain Transfer. Prerequites Python 3.5 PyTorch 0.1.12

Yunjey Choi 401 Dec 30, 2022
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.

TensorFlowOnSpark TensorFlowOnSpark brings scalable deep learning to Apache Hadoop and Apache Spark clusters. By combining salient features from the T

Yahoo 3.8k Jan 04, 2023
Official implementation of "Learning to Discover Cross-Domain Relations with Generative Adversarial Networks"

DiscoGAN Official PyTorch implementation of Learning to Discover Cross-Domain Relations with Generative Adversarial Networks. Prerequisites Python 2.7

SK T-Brain 754 Dec 29, 2022
ColossalAI-Examples - Examples of training models with hybrid parallelism using ColossalAI

ColossalAI-Examples This repository contains examples of training models with Co

HPC-AI Tech 185 Jan 09, 2023