ABSTRACT



Stock Market Trading Bot using Deep Q Learning15CS496L MAJOR PROJECTA PROJECT REVIEW REPORT - ISubmitted byTejasv SharmaVibhav Agrawal RA16110030100861RA1611003010897Project Batch ID:CSE08610897Name, Signature of guide with date:In partial fulfillment of the requirements for the degree ofBACHELOR OF TECHNOLOGY2947416222142DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERINGFACULTY OF ENGINEERING AND TECHNOLOGYSRM INSTITUTE OF SCIENCE AND TECHNOLOGYKATTANKULATHUR- 603203December 2019ABSTRACTStock trading strategies are developed to find correlation between regulated financial instruments and their future performance. Recent developments in stock trading strategies suggest that future prediction machine learning models are extremely powerful. But are not robust to the dynamic nature of the stock market. In this work, the aim is to do a detailed survey on applying machine learning methods for quantitative trading. Related work has been done on predicting stock market with Support-Vector-Machines and Neural Network. In this project we also review the designing process of online learning algorithms which leverage characteristics historical data, e.g., LSTM. Other than traditional machine learning algorithms we have also reviewed recent development in meta-learning algorithm, they are widely accepted because of their weight first strategy. This project aims at developing a stock market trading bot using deep reinforcement learning and machine learning algorithms which we think hold a lot of promise for the future. In this project we also discusses the untraditional strategies that are leveraged to create more profitable portfolio, such as behavioral and event-driven strategies.iiTABLE OF CONTENTSCHAPTERNO.TITLEPAGE NO. ABSTRACTiiLIST OF TABLESivLIST OF FIGURESvLIST OF SYMBOLS AND ABBREVIATIONS viINTRODUCTION7Past learnings7Implementation7LITERATURE SURVEY8Networks for Predict Trends and Prices 8Analysis on Wavelet neural networks8Research on Public Sentiment Data8Inference from the survey9MODULES10Data pre-processing10Data Visualisation12Neural Network on Financial data15LSTM21CONCLUSION22APPENDIX 23REFERENCES 27LIST OF TABLESTABLE NO.TITLEPAGE NO.2.1Networks for Predict Trends and Prices22.2Analysis on Wavelet neural networks22.3Research on Public Sentiment Data2 ivLIST OF FIGURESFIGURE NO.TITLEPAGENO.3.1Data Frame63.2Column Major Data Visualization73.3Code snippet and data visualization83.4Company wise data visualization93.5Code snippet for visualizing GOOG stock market data103.6Google data frame visualization113.7Basic feature building for neural network123.8Splitting dataset into training and testing dataset133.9Code snippet to split dataset143.10Visualization of split dataset143.11Code snippet for neural network153.12Code snippet of neural network continued16 3.13 Code snippet for LSTM 17vLIST OF SYMBOLS AND ABBREVIATIONSAPPL-Apple’s stock market symbolGOOG-Google’s stock market symbolMSFT-Microsoft’s stock market symbolQQL-Qualcomm’s stock market symbolSVM-Support vector machineUI-User InterfaceUX-User Experience LSTM -Long Short Term MemoryviINTRODUCTIONPast learningsStock market plays a very important role in fast economic growth of the developing country like India. If stock market rises, then countries economic growth would be high. If stock market falls, then countries economic growth would be down. In any country, only 10% of the people engaging themselves with the stock market investment because of the dynamic nature of the stock market. The prediction techniques in stock market can plays a crucial role in bringing more people and existing investors at one place. Predicting stock trends have always been a key work of researchers and analysists. Accurate predictions can lead to huge profits and inaccurate ones can lead to losses. We studied previous research papers which showed how researchers have used machine learning algorithms, wavelet neural networks and public sentiment data to predict stocks and have succeeded to quite an extent but still can improve by using deep reinforcement learning.ImplementationWe are aiming at making a stock market trading bot using deep reinforcement learning which focuses on predicting stock market trends and prices. Stock market is a highly volatile environment and to get high prediction accuracy is challenging. The model will be implemented on the yahoo finance dataset to verify the result. The model can be implemented in real life by leveraging various brokerage services such as Upstox or Zerodha. This reinforcement learning model will help us understand the chaotic market. Our purpose is to develop a self-learning agent which will learn from its mistakes. We are focusing on how various methods like statistical models and machine earning can be applied to stock market trend prediction and stock price estimation, which researchers think will be able to solve the problems of model limitations that are brought on by technical and traditional fundamental analysis, which has been used in the global financial markets, and the problems arising due to the scale and irregularities of the data. The end goal of this project is to optimize the profit for any given portfolio. LITERATURE SURVEYTable 2.1: Networks for Predicting Trends and PricesREF. No. OBJECTIVE ALGORITHM /STRATEGIES USED DATASETS OR INPUT PARAMETERS RELEVANCE PERFORMANCE MEASURE USED [1] To predict stock market trends and prices using neural networks.Neural networks, support vector machine Yahoo! Finance datasetThis algorithm was tested on Shanghai exchange data and showed very high prediction accuracy1. Handles Big data and meets consumer demands. Table 2.2: Analysis on Wavelet neural networks[2] is applied on non-stationary signal leakage such as stock market indicesWavelet decomposition Google finance datasetCurrent academicians see wavelet analysis as an optimization feature which when added to neural networks or support vector machines results in increased prediction accuracy by a big margin.1. Specificity 2. Security 3. Privacy Table 2.3: Research on Public Sentiment Data[3] identifying the investor’s psychology and behavior and also the existence of psychologically unstable investorsPublic sentiment analysis modelYahoo! Finance datasetThe results were very impressive and showed that their model ensures significant gains1. Accuracy 2. Challenges Inference from the surveyTill now applying machine learning algorithms has been applied on financial market data in order to predict stock prices and to maintain the profitability of the portfolio. It is a very challenging task whether done in corporate environment or in an academic field. Although most of the cutting edge machine learning algorithms have outperformed the benchmarks scores in their back tests, it has always been challenging for the researchers to optimize already implemented algorithm and to make them more efficient. Understanding various research works and also summarizing this paper, we would like to draw attention towards both confluent and independent suggestions. During our study we found out that machine learning has two major applications in financial markets. One is to predict stock prices and trends and other is to select a profitable portfolio. This helps corporates make more commercial profits and invest in research. In addition to this work we will also like to add that prediction task done with accurate analysis can help in finding major trend factors which contribute towards the volatility in the stock market. These factors are not just limited to policy but also include sentiment analysis and behavioral finance. The area of application of machine learning algorithms in quantitative finance is very wide. The study also finds out that high performance computers play a major role in stock market trading using advanced machine learning algorithms. 3. MODULESData pre-processingThe dataset was obtained from google finance and yahoo finance dataset. We trimmed the data according to our need and removed unnecessary data columns like stock volume, stock opening date, stock inaugural date etc. We then utilized the pandas python library to trim and restructure our data. The resultant data frame obtained is given below.4292601143000 Fig 3.1: Data Frame Using standard python libraries we removed data with invalid information. For example, where ever the open and close price of a stock is given as NaN we removed those rows. This made our data set more consistent and reliable. We also removed rows from the dataset which included data points which were not of any significance now. To shorten the training process we have chosen the top five US technology companies which are apple, Microsoft, google and qulacomm. Below is a column major orientation of the data set which has been cleaned.Figure 3.2: Column Major Data VisualizationData VisualizationWe have leveraged the inbuilt functions offered by matplotlib python library to visualize the preprocessed data. We visualize the data as a whole and also as company wise stock performance. We use graphs like histogram, line chart and box whisker plot to visualize the data. Below is the code snippet to create a histogram from our dataset. We also provide the output of the given code snippet. Figure 3.3: Code snippet and data visualization In the below figure we have plotted company wise open and close stock price data. We visualize them side by side to have a comprehensive understanding of returns provided by the companies. Each of the four graphs in a single row define the open, close, high and low stock prices of a particular company. The first row represents apple stock details, the second one represents google, the third represents Microsoft and the last one represents qulacomm.2540-25400 Figure 3.4: Company wise data visualizationIn this code snippet we are visualizing google’s (GOOG) stock market data. We utilize matplotlib and our own python script which has been given in the appendix. We have also printed out the top five rows for this data. Below you will first see the code snippet followed by data frames visualization.330202622550 Figure 3.5: Code snippet for visualizing GOOG stock market data-27940-48768000Fig 3.6: Google data frame visualization3.3 Neural Network on Financial DataIn the code snippet given below we try to perform sanity check of the given neural network model. For this we built very simple and insignificant modelling features. This is done just to verify the work done previously. To prevent inconsistencies due to data irregularities we perform the below given modelling features.Fig 3.7: Basic feature building for neural networkThe next task after performing the sanity check is to divide the dataset into two parts namely test data and training data. The ratio between test and train data is 20:80. The testing data and its size has been designed in such a way that any sort of over fitting in the model can be detected, on the other hand training data’s size ensures that enough data is available as input feed for the neural network. It is very important to divide the dataset into testing and training dataset because it keeps the model away from getting over fitted. A model is said to be over fitted when it achieves a near 100% accuracy on the data set but does poorly when a new dataset is introduced. The main reason behind a model being over fitted is insufficiency of data and the test data’s size is too small. Due to the paucity of the test data model finds a very particular local minima which is far away from generality. Hence it performs poorly on newly introduced dataset. Fig 3.8: Splitting dataset into training and testing datasetFig 3.9: Code snippet to split datasetFig: 3.10: Visualization of split dataset Fig 3.11: Code snippet for neural networkFig 3.12: Code snippet of neural network continued LSTM – Long Short Term MemoryLong Short-Term Memory (LSTM) networks are a modified version of recurrent neural networks, which makes it easier to remember past data in memory. The vanishing gradient problem of RNN is resolved here. LSTM is well-suited to classify, process and predict time series given time lags of unknown duration. It trains the model by using back-propagation. In an LSTM network, three gates are present Input gate — discover which value from input should be used to modify the memory. Sigmoid function decides which values to let through 0,1 and tanh function gives weightage to the values which are passed deciding their level of importance ranging from-1 to 1. Forget gate — discover what details to be discarded from the block. It is decided by the sigmoid function. Output gate — the input and the memory of the block is used to decide the output. Sigmoid function decides which values to let through 0,1. and tanh function gives weightage to the values which are passed deciding their level of importance ranging from-1 to 1 and multiplied with output of Sigmoid.Fig 3.13: Code snippet for LSTCONCLUSIONIn this project we aim at building a robust deep reinforcement learning model which can accurately predict stock prices and stock market trends. The project aims at providing a model architecture which provides generality over a vast majority of global financial markets. To know more about this we reviewed study which takes Shanghai exchange and Tokyo exchange into consideration. By including these global markets we expanded our horizon for generality towards major financial markets. Our model strained on google and yahoo finance datasets and performs well enough in the field of asses pricing. We also simulate different market scenarios which made our model more robust and accurate in its prediction. Our ultimate goal is to develop a trading bot which works across various financial markets with good profitable returns.APPENDIXimport pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport loggingfrom time import timefrom datetime import timedelta, datetimefrom sklearn import linear_modelimport tensorflow as tffrom tensorflow import kerasfilenames_time_format='%Y-%m-%d %H-%M-%S'logging_time_format='%H:%M:%S'trading_datetime_format='%Y/%m/%d'logger=logging.getLogger('OzML logger')logging.basicConfig(format='%(asctime)s OzML %(funcName)s (%(levelname)s): %(message)s',level=, # logging messages which are less severe than level will be ignored. Levels: NOTSET<DEBUG<INFO<WARNING<ERROR<CRITICALdatefmt=logging_time_format)('OzML imported, logger created with logging level: INFO (change levels by calling OzML.set_logging_level)')#%% defining functions, classesdef set_logging_level(logger_obj,level):"""created on 2018/09/10"""if level=='debug':logging_level=logging.DEBUGelif level=='info':logging_level=elif level=='warning':logging_level=logging.WARNINGelif level=='error':logger.warn('the logger was set to error = muted, since in errors I raise a RuntimeError and never use the logger')logging_level=logging.ERRORelse:raise RuntimeError('unsupported log_level!')logger_obj.setLevel(logging_level)def import_data(symbols,columns_list_to_use):"""importing symbols data from csv files into a single dataframereturns: (data_df,attribute_names_list)reads symbols from csv files (example: for symbols=['GOOGL'] it readsfrom GOOGL.txt) into pandas dataframes, indexing by 'Date' columns of theoriginal data, taking only columns_list_to_use from each dataframe,merging all into one data_df by intersection of the index of all dataframes,saving all column names (including symbols) to attribute_names_list,re-naming data_df colums to generic names 'x0','x1',..."""df_list=[]for symbol in symbols:df=pd.read_csv(symbol+'.txt',parse_dates=['Date'],index_col=['Date'])df=df[columns_list_to_use]df.columns=[symbol+' '+col for col in df.columns]df_list.append(df)data_df=df_list[0]for df in df_list[1:]:data_df=data_df.merge(df,how='inner',left_index=True,right_index=True)attribute_names_list=list(data_df.columns)data_df.columns=['x%d'%ind for ind in range(len(data_df.columns))]('data imported! total length (number of rows): %d, number of columns: %d'%(len(data_df),len(attribute_names_list)))return (data_df,attribute_names_list)def legend_styler(labels=None,loc='best',framealpha=0.6,fancybox=True,frameon=True,facecolor='white'):"""created on 2018/10/03 to easily control legend style in all executionsfrom one place."""if labels==None:plt.legend(loc=loc,framealpha=framealpha,fancybox=fancybox,frameon=frameon,facecolor=facecolor)else:plt.legend(labels,loc=loc,framealpha=framealpha,fancybox=fancybox,frameon=frameon,facecolor=facecolor)def data_plotting(data_df,attribute_names_list):"""created on 2018/09/13 for data plotting used for verification,before building features.The data is normalized here as a whole without splitting to test,which is not legit for testing, only for data verification"""temp_norm_df=(data_df-data_df.mean(axis=0))/(data_df.max(axis=0)-data_df.min(axis=0))temp_norm_df.columns=attribute_names_listtemp_norm_df.plot()plt.title('normalized data (x-mean(x))/std(x)')legend_styler(loc='upper left')plt.grid(True,which='major',axis='both')def examine_data_jumps(df,processing_mode,suspicious_jump_percents,attribute_names_list=0,plotting=True,bins=100):"""created on 2018/09/16 to examine suspicious jumps in data values(value[1:]/value[0:-1]-1), meaning absolute value jumps of more thansuspicious_jump_percents.if processing_mode=='% differences' then examining df itself, otherwisecreating diff_percents_df to examine.plotting=True: plots jumps histogramif attribute_names_list is given, naming diff_percents_df columns by it(recommended).returns: suspicious_jumps_df"""number_of_row_with_any_zeros=len(df[df.values==0])if number_of_row_with_any_zeros>0:raise RuntimeError("there are %d row with any zeros, get rid of them before executing this (don't include volumes in df!)"%number_of_row_with_any_zeros)if attribute_names_list!=0:df.columns=attribute_names_listif processing_mode=='% differences':diff_percents_df=dfelse:diff_percents_df=pd.DataFrame(100*(df.values[1:,:]/df.values[0:-1,:]-1),index=df.index[1:],# columns=[col+' jumps %' for col in df.columns])columns=df.columns)if plotting:print('histograms of % jumps =100*(value[1:]/value[0:-1]-1)')diff_percents_df.hist(bins=100)plt.show()suspicious_price_jumps_df=diff_percents_df[abs(diff_percents_df.values)>suspicious_jump_percents]('there are %d price jumps higher in abs value than %d%%, saved in suspicious_price_jumps_df'%(len(suspicious_price_jumps_df),suspicious_jump_percents))return suspicious_price_jumps_dfdef basic_feature_building(data_df,target_name,attribute_names_list,include_target_history_as_features=True):"""created on 2018/09/17returns (X_df,y_series) where X_df.values[idx,:] can be used to predicty_series.values[idx], maintaining causality:X_df.values[idx,:] elements are taken from data_df.values[idx-1,:] butindexed by data_df.index[idx],while y_series.values[idx] is taken from data_df.values[idx,:] andindexed by data_df.index[idx]."""target_idx=attribute_names_list.index(target_name)base_feature_names_list=list(data_df.columns)if not include_target_history_as_features:base_feature_names_list.remove(attribute_names_list[target_idx])X_temp=data_df[base_feature_names_list]X_df=pd.DataFrame(X_temp.values[0:-1,:],columns=[col+'(t-dt)' for col in base_feature_names_list],index=data_df.index[1:])X_df.index.name='t'y_series=data_df[base_feature_names_list[target_idx]].filter(items=X_df.index)y_series.name=base_feature_names_list[target_idx]y_series.index.name='t'data_duration_years=timedelta.total_seconds(y_series.index[-1]-y_series.index[0])/(365*24*60*60)('total data duration: %.1f years'%data_duration_yearsREFERENCES[1] Agarap, Abien Fred, 2018. “Deep Learning Using Rectified Linear Units (ReLU),” no. 1: 2–8. [2]Almahdi, Saud, and Steve Y. Yang. ”An adaptive portfolio trading system: A risk-return portfolio optimization using recurrent reinforcement learning with expected maximum drawdown.” IEEE Journal of Expert Systems with Applications 87 (2017): 267-279[3]Souradeep Chakraborty, 2018. “Capturing Financial markets to apply Deep Reinforcement Learning”, arXiv:1907.04373v3 [4]Si, W., Li, J., Ding, P., & Rao, R. (2017). ‘A Multi-objective Deep Reinforcement Learning Approach for Stock Index Futures’ Intraday Trading’ 2017 10th International Symposium on Computational Intelligence and Design (ISCID)[5]Konstantinos Saitas Zarkias, Nikolaos Passalis, Avraam Tsantekidis, Anastasios Tefas, “Deep Reinforcement Learning for Financial Trading Using Price Trailing”, 2019, ICASSP - IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)[6]Yang Li , Wanshan Zheng ,Zibin Zheng, “Deep Robust Reinforcement Learning for Practical Algorithmic Trading”, 2019, IEEE Access Journal, Vol 7 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download