samre12/deep-trading-agent repository overview

⁠Deep Trading Agent

Deep Reinforcement Learning based Trading Agent for Bitcoin using DeepSense⁠ Network for Q function approximation.


For complete details of the input time series, network architecture and implementation, refer to the Wiki⁠ of this repository.

Contents of this document

• Requirements⁠
• Run with Docker⁠
• Usage⁠
• ToDo⁠
• Trading Model⁠
• Implemenation⁠
  • Deep Sense⁠
  • Deep Q Trading⁠

⁠Support

Please give a :star: to this repository to support the project :smile:.

⁠Requirements

• Python 2.7
• Tensorflow⁠
• Pandas⁠ (for pre-processing Bitcoin Price Series)
• TA-lib⁠ (for computing technical indicators over Bitcoin Price Series)
• tqdm⁠ (for displaying progress of training)
• samre12/gym_cryptotrading⁠ (for different trading environments)

⁠Run with Docker

Pull the prebuilt docker image directly from docker hub and run it as

docker pull samre12/deep-trading-agent:latest
docker run -p 6006:6006 -it samre12/deep-trading-agent:latest

Note: Use docker image samre12/deep-trading-agent:latest_dev for the development (dev) branch of the repository.

OR

Build the docker image locally by executing the command and the run the image as

docker build -t deep-trading-agent .
docker run -p 6006:6006 -it deep-trading-agent

This will setup the repository for training the agent and

• mount the current directory into /deep-trading-agent in the container

• to initiate training of the agent, specify suitable parameters in a config file (an example config file is provided at /deep-trading-agent/agent/config/config.cfg) and run the code using /deep-trading-agent/main.py

• training supports logging and monitoring through Tensorboard

• vim and screen are installed in the container to edit the configuration files and run tensorboard

• bind port 6006 of container to 6006 of host machine to monitor training using Tensorboard

⁠Usage

Having cd into the directory, execute the following command to train the model

python2 main.py --config=/path/to/config

To visualize learning using tensorboard, execute the following commands

# cd into tensorboard logging directory
cd logs/tensorboard
tensorboard --logdir=./ --host=0.0.0.0 --port=6006

⁠ToDo

⁠Docker Support

• Add Docker support for a fast and easy start with the project

⁠Improve Model performance

• Extract highest and lowest prices and the volume of Bitcoin traded within a given time interval in the Preprocessor⁠
• Use closing, highest, lowest prices and the volume traded as input channels to the model (remove features calculated just using closing prices)
• Normalize the price tensors using the price of the previous time step
• For the complete state representation, input the remaining number of trades to the model
• Use separate diff price blocks to calculate the unrealized PnL
• Use exponentially decayed weighted unrealized PnL⁠ as a reward function to incorporate current state of investment and stabilize the learning of the agent

⁠Trading Model

is inspired by Deep Q-Trading⁠ where they solve a simplified trading problem for a single asset.
For each trading unit, only one of the three actions: neutral(1), long(2) and short(3) are allowed and a reward is obtained depending upon the current position of agent. Deep Q-Learning agent is trained to maximize the total accumulated rewards.
Current Deep Q-Trading model is modified by using the Deep Sense architecture for Q function approximation.

For more information on the trading model, refer here⁠.

⁠Implementation

Tensorflow is used for the implementation of the Deep Sense network.

⁠Deep Sense

Implementation is adapted from this⁠ Github repository with a few simplifications in the network architecture to incorporate learning over a single time series of the Bitcoin data.

⁠Deep Q Trading

Implementation and preprocessing is inspired from this Medium post⁠. The actual implementation of the Deep Q Network is adapted from DQN-tensorflow⁠.