research papers on stock market prediction

• Open access
• Published: 28 August 2020

Short-term stock market price trend prediction using a comprehensive deep learning system

• Jingyi Shen 1 &
• M. Omair Shafiq   ORCID: orcid.org/0000-0002-1859-8296 1  

Journal of Big Data volume  7 , Article number:  66 ( 2020 ) Cite this article

265k Accesses

155 Citations

91 Altmetric

Metrics details

In the era of big data, deep learning for predicting stock market prices and trends has become even more popular than before. We collected 2 years of data from Chinese stock market and proposed a comprehensive customization of feature engineering and deep learning-based model for predicting price trend of stock markets. The proposed solution is comprehensive as it includes pre-processing of the stock market dataset, utilization of multiple feature engineering techniques, combined with a customized deep learning based system for stock market price trend prediction. We conducted comprehensive evaluations on frequently used machine learning models and conclude that our proposed solution outperforms due to the comprehensive feature engineering that we built. The system achieves overall high accuracy for stock market trend prediction. With the detailed design and evaluation of prediction term lengths, feature engineering, and data pre-processing methods, this work contributes to the stock analysis research community both in the financial and technical domains.

Introduction

Stock market is one of the major fields that investors are dedicated to, thus stock market price trend prediction is always a hot topic for researchers from both financial and technical domains. In this research, our objective is to build a state-of-art prediction model for price trend prediction, which focuses on short-term price trend prediction.

As concluded by Fama in [ 26 ], financial time series prediction is known to be a notoriously difficult task due to the generally accepted, semi-strong form of market efficiency and the high level of noise. Back in 2003, Wang et al. in [ 44 ] already applied artificial neural networks on stock market price prediction and focused on volume, as a specific feature of stock market. One of the key findings by them was that the volume was not found to be effective in improving the forecasting performance on the datasets they used, which was S&P 500 and DJI. Ince and Trafalis in [ 15 ] targeted short-term forecasting and applied support vector machine (SVM) model on the stock price prediction. Their main contribution is performing a comparison between multi-layer perceptron (MLP) and SVM then found that most of the scenarios SVM outperformed MLP, while the result was also affected by different trading strategies. In the meantime, researchers from financial domains were applying conventional statistical methods and signal processing techniques on analyzing stock market data.

The optimization techniques, such as principal component analysis (PCA) were also applied in short-term stock price prediction [ 22 ]. During the years, researchers are not only focused on stock price-related analysis but also tried to analyze stock market transactions such as volume burst risks, which expands the stock market analysis research domain broader and indicates this research domain still has high potential [ 39 ]. As the artificial intelligence techniques evolved in recent years, many proposed solutions attempted to combine machine learning and deep learning techniques based on previous approaches, and then proposed new metrics that serve as training features such as Liu and Wang [ 23 ]. This type of previous works belongs to the feature engineering domain and can be considered as the inspiration of feature extension ideas in our research. Liu et al. in [ 24 ] proposed a convolutional neural network (CNN) as well as a long short-term memory (LSTM) neural network based model to analyze different quantitative strategies in stock markets. The CNN serves for the stock selection strategy, automatically extracts features based on quantitative data, then follows an LSTM to preserve the time-series features for improving profits.

The latest work also proposes a similar hybrid neural network architecture, integrating a convolutional neural network with a bidirectional long short-term memory to predict the stock market index [ 4 ]. While the researchers frequently proposed different neural network solution architectures, it brought further discussions about the topic if the high cost of training such models is worth the result or not.

There are three key contributions of our work (1) a new dataset extracted and cleansed (2) a comprehensive feature engineering, and (3) a customized long short-term memory (LSTM) based deep learning model.

We have built the dataset by ourselves from the data source as an open-sourced data API called Tushare [ 43 ]. The novelty of our proposed solution is that we proposed a feature engineering along with a fine-tuned system instead of just an LSTM model only. We observe from the previous works and find the gaps and proposed a solution architecture with a comprehensive feature engineering procedure before training the prediction model. With the success of feature extension method collaborating with recursive feature elimination algorithms, it opens doors for many other machine learning algorithms to achieve high accuracy scores for short-term price trend prediction. It proved the effectiveness of our proposed feature extension as feature engineering. We further introduced our customized LSTM model and further improved the prediction scores in all the evaluation metrics. The proposed solution outperformed the machine learning and deep learning-based models in similar previous works.

The remainder of this paper is organized as follows. “ Survey of related works ” section describes the survey of related works. “ The dataset ” section provides details on the data that we extracted from the public data sources and the dataset prepared. “ Methods ” section presents the research problems, methods, and design of the proposed solution. Detailed technical design with algorithms and how the model implemented are also included in this section. “ Results ” section presents comprehensive results and evaluation of our proposed model, and by comparing it with the models used in most of the related works. “ Discussion ” section provides a discussion and comparison of the results. “ Conclusion ” section presents the conclusion. This research paper has been built based on Shen [ 36 ].

Survey of related works

In this section, we discuss related works. We reviewed the related work in two different domains: technical and financial, respectively.

Kim and Han in [ 19 ] built a model as a combination of artificial neural networks (ANN) and genetic algorithms (GAs) with discretization of features for predicting stock price index. The data used in their study include the technical indicators as well as the direction of change in the daily Korea stock price index (KOSPI). They used the data containing samples of 2928 trading days, ranging from January 1989 to December 1998, and give their selected features and formulas. They also applied optimization of feature discretization, as a technique that is similar to dimensionality reduction. The strengths of their work are that they introduced GA to optimize the ANN. First, the amount of input features and processing elements in the hidden layer are 12 and not adjustable. Another limitation is in the learning process of ANN, and the authors only focused on two factors in optimization. While they still believed that GA has great potential for feature discretization optimization. Our initialized feature pool refers to the selected features. Qiu and Song in [ 34 ] also presented a solution to predict the direction of the Japanese stock market based on an optimized artificial neural network model. In this work, authors utilize genetic algorithms together with artificial neural network based models, and name it as a hybrid GA-ANN model.

Piramuthu in [ 33 ] conducted a thorough evaluation of different feature selection methods for data mining applications. He used for datasets, which were credit approval data, loan defaults data, web traffic data, tam, and kiang data, and compared how different feature selection methods optimized decision tree performance. The feature selection methods he compared included probabilistic distance measure: the Bhattacharyya measure, the Matusita measure, the divergence measure, the Mahalanobis distance measure, and the Patrick-Fisher measure. For inter-class distance measures: the Minkowski distance measure, city block distance measure, Euclidean distance measure, the Chebychev distance measure, and the nonlinear (Parzen and hyper-spherical kernel) distance measure. The strength of this paper is that the author evaluated both probabilistic distance-based and several inter-class feature selection methods. Besides, the author performed the evaluation based on different datasets, which reinforced the strength of this paper. However, the evaluation algorithm was a decision tree only. We cannot conclude if the feature selection methods will still perform the same on a larger dataset or a more complex model.

Hassan and Nath in [ 9 ] applied the Hidden Markov Model (HMM) on the stock market forecasting on stock prices of four different Airlines. They reduce states of the model into four states: the opening price, closing price, the highest price, and the lowest price. The strong point of this paper is that the approach does not need expert knowledge to build a prediction model. While this work is limited within the industry of Airlines and evaluated on a very small dataset, it may not lead to a prediction model with generality. One of the approaches in stock market prediction related works could be exploited to do the comparison work. The authors selected a maximum 2 years as the date range of training and testing dataset, which provided us a date range reference for our evaluation part.

Lei in [ 21 ] exploited Wavelet Neural Network (WNN) to predict stock price trends. The author also applied Rough Set (RS) for attribute reduction as an optimization. Rough Set was utilized to reduce the stock price trend feature dimensions. It was also used to determine the structure of the Wavelet Neural Network. The dataset of this work consists of five well-known stock market indices, i.e., (1) SSE Composite Index (China), (2) CSI 300 Index (China), (3) All Ordinaries Index (Australian), (4) Nikkei 225 Index (Japan), and (5) Dow Jones Index (USA). Evaluation of the model was based on different stock market indices, and the result was convincing with generality. By using Rough Set for optimizing the feature dimension before processing reduces the computational complexity. However, the author only stressed the parameter adjustment in the discussion part but did not specify the weakness of the model itself. Meanwhile, we also found that the evaluations were performed on indices, the same model may not have the same performance if applied on a specific stock.

Lee in [ 20 ] used the support vector machine (SVM) along with a hybrid feature selection method to carry out prediction of stock trends. The dataset in this research is a sub dataset of NASDAQ Index in Taiwan Economic Journal Database (TEJD) in 2008. The feature selection part was using a hybrid method, supported sequential forward search (SSFS) played the role of the wrapper. Another advantage of this work is that they designed a detailed procedure of parameter adjustment with performance under different parameter values. The clear structure of the feature selection model is also heuristic to the primary stage of model structuring. One of the limitations was that the performance of SVM was compared to back-propagation neural network (BPNN) only and did not compare to the other machine learning algorithms.

Sirignano and Cont leveraged a deep learning solution trained on a universal feature set of financial markets in [ 40 ]. The dataset used included buy and sell records of all transactions, and cancellations of orders for approximately 1000 NASDAQ stocks through the order book of the stock exchange. The NN consists of three layers with LSTM units and a feed-forward layer with rectified linear units (ReLUs) at last, with stochastic gradient descent (SGD) algorithm as an optimization. Their universal model was able to generalize and cover the stocks other than the ones in the training data. Though they mentioned the advantages of a universal model, the training cost was still expensive. Meanwhile, due to the inexplicit programming of the deep learning algorithm, it is unclear that if there are useless features contaminated when feeding the data into the model. Authors found out that it would have been better if they performed feature selection part before training the model and found it as an effective way to reduce the computational complexity.

Ni et al. in [ 30 ] predicted stock price trends by exploiting SVM and performed fractal feature selection for optimization. The dataset they used is the Shanghai Stock Exchange Composite Index (SSECI), with 19 technical indicators as features. Before processing the data, they optimized the input data by performing feature selection. When finding the best parameter combination, they also used a grid search method, which is k cross-validation. Besides, the evaluation of different feature selection methods is also comprehensive. As the authors mentioned in their conclusion part, they only considered the technical indicators but not macro and micro factors in the financial domain. The source of datasets that the authors used was similar to our dataset, which makes their evaluation results useful to our research. They also mentioned a method called k cross-validation when testing hyper-parameter combinations.

McNally et al. in [ 27 ] leveraged RNN and LSTM on predicting the price of Bitcoin, optimized by using the Boruta algorithm for feature engineering part, and it works similarly to the random forest classifier. Besides feature selection, they also used Bayesian optimization to select LSTM parameters. The Bitcoin dataset ranged from the 19th of August 2013 to 19th of July 2016. Used multiple optimization methods to improve the performance of deep learning methods. The primary problem of their work is overfitting. The research problem of predicting Bitcoin price trend has some similarities with stock market price prediction. Hidden features and noises embedded in the price data are threats of this work. The authors treated the research question as a time sequence problem. The best part of this paper is the feature engineering and optimization part; we could replicate the methods they exploited in our data pre-processing.

Weng et al. in [ 45 ] focused on short-term stock price prediction by using ensemble methods of four well-known machine learning models. The dataset for this research is five sets of data. They obtained these datasets from three open-sourced APIs and an R package named TTR. The machine learning models they used are (1) neural network regression ensemble (NNRE), (2) a Random Forest with unpruned regression trees as base learners (RFR), (3) AdaBoost with unpruned regression trees as base learners (BRT) and (4) a support vector regression ensemble (SVRE). A thorough study of ensemble methods specified for short-term stock price prediction. With background knowledge, the authors selected eight technical indicators in this study then performed a thoughtful evaluation of five datasets. The primary contribution of this paper is that they developed a platform for investors using R, which does not need users to input their own data but call API to fetch the data from online source straightforward. From the research perspective, they only evaluated the prediction of the price for 1 up to 10 days ahead but did not evaluate longer terms than two trading weeks or a shorter term than 1 day. The primary limitation of their research was that they only analyzed 20 U.S.-based stocks, the model might not be generalized to other stock market or need further revalidation to see if it suffered from overfitting problems.

Kara et al. in [ 17 ] also exploited ANN and SVM in predicting the movement of stock price index. The data set they used covers a time period from January 2, 1997, to December 31, 2007, of the Istanbul Stock Exchange. The primary strength of this work is its detailed record of parameter adjustment procedures. While the weaknesses of this work are that neither the technical indicator nor the model structure has novelty, and the authors did not explain how their model performed better than other models in previous works. Thus, more validation works on other datasets would help. They explained how ANN and SVM work with stock market features, also recorded the parameter adjustment. The implementation part of our research could benefit from this previous work.

Jeon et al. in [ 16 ] performed research on millisecond interval-based big dataset by using pattern graph tracking to complete stock price prediction tasks. The dataset they used is a millisecond interval-based big dataset of historical stock data from KOSCOM, from August 2014 to October 2014, 10G–15G capacity. The author applied Euclidean distance, Dynamic Time Warping (DTW) for pattern recognition. For feature selection, they used stepwise regression. The authors completed the prediction task by ANN and Hadoop and RHive for big data processing. The “ Results ” section is based on the result processed by a combination of SAX and Jaro–Winkler distance. Before processing the data, they generated aggregated data at 5-min intervals from discrete data. The primary strength of this work is the explicit structure of the whole implementation procedure. While they exploited a relatively old model, another weakness is the overall time span of the training dataset is extremely short. It is difficult to access the millisecond interval-based data in real life, so the model is not as practical as a daily based data model.

Huang et al. in [ 12 ] applied a fuzzy-GA model to complete the stock selection task. They used the key stocks of the 200 largest market capitalization listed as the investment universe in the Taiwan Stock Exchange. Besides, the yearly financial statement data and the stock returns were taken from the Taiwan Economic Journal (TEJ) database at www.tej.com.tw/ for the time period from year 1995 to year 2009. They conducted the fuzzy membership function with model parameters optimized with GA and extracted features for optimizing stock scoring. The authors proposed an optimized model for selection and scoring of stocks. Different from the prediction model, the authors more focused on stock rankings, selection, and performance evaluation. Their structure is more practical among investors. But in the model validation part, they did not compare the model with existed algorithms but the statistics of the benchmark, which made it challenging to identify if GA would outperform other algorithms.

Fischer and Krauss in [ 5 ] applied long short-term memory (LSTM) on financial market prediction. The dataset they used is S&P 500 index constituents from Thomson Reuters. They obtained all month-end constituent lists for the S&P 500 from Dec 1989 to Sep 2015, then consolidated the lists into a binary matrix to eliminate survivor bias. The authors also used RMSprop as an optimizer, which is a mini-batch version of rprop. The primary strength of this work is that the authors used the latest deep learning technique to perform predictions. They relied on the LSTM technique, lack of background knowledge in the financial domain. Although the LSTM outperformed the standard DNN and logistic regression algorithms, while the author did not mention the effort to train an LSTM with long-time dependencies.

Tsai and Hsiao in [ 42 ] proposed a solution as a combination of different feature selection methods for prediction of stocks. They used Taiwan Economic Journal (TEJ) database as data source. The data used in their analysis was from year 2000 to 2007. In their work, they used a sliding window method and combined it with multi layer perceptron (MLP) based artificial neural networks with back propagation, as their prediction model. In their work, they also applied principal component analysis (PCA) for dimensionality reduction, genetic algorithms (GA) and the classification and regression trees (CART) to select important features. They did not just rely on technical indices only. Instead, they also included both fundamental and macroeconomic indices in their analysis. The authors also reported a comparison on feature selection methods. The validation part was done by combining the model performance stats with statistical analysis.

Pimenta et al. in [ 32 ] leveraged an automated investing method by using multi-objective genetic programming and applied it in the stock market. The dataset was obtained from Brazilian stock exchange market (BOVESPA), and the primary techniques they exploited were a combination of multi-objective optimization, genetic programming, and technical trading rules. For optimization, they leveraged genetic programming (GP) to optimize decision rules. The novelty of this paper was in the evaluation part. They included a historical period, which was a critical moment of Brazilian politics and economics when performing validation. This approach reinforced the generalization strength of their proposed model. When selecting the sub-dataset for evaluation, they also set criteria to ensure more asset liquidity. While the baseline of the comparison was too basic and fundamental, and the authors did not perform any comparison with other existing models.

Huang and Tsai in [ 13 ] conducted a filter-based feature selection assembled with a hybrid self-organizing feature map (SOFM) support vector regression (SVR) model to forecast Taiwan index futures (FITX) trend. They divided the training samples into clusters to marginally improve the training efficiency. The authors proposed a comprehensive model, which was a combination of two novel machine learning techniques in stock market analysis. Besides, the optimizer of feature selection was also applied before the data processing to improve the prediction accuracy and reduce the computational complexity of processing daily stock index data. Though they optimized the feature selection part and split the sample data into small clusters, it was already strenuous to train daily stock index data of this model. It would be difficult for this model to predict trading activities in shorter time intervals since the data volume would be increased drastically. Moreover, the evaluation is not strong enough since they set a single SVR model as a baseline, but did not compare the performance with other previous works, which caused difficulty for future researchers to identify the advantages of SOFM-SVR model why it outperforms other algorithms.

Thakur and Kumar in [ 41 ] also developed a hybrid financial trading support system by exploiting multi-category classifiers and random forest (RAF). They conducted their research on stock indices from NASDAQ, DOW JONES, S&P 500, NIFTY 50, and NIFTY BANK. The authors proposed a hybrid model combined random forest (RF) algorithms with a weighted multicategory generalized eigenvalue support vector machine (WMGEPSVM) to generate “Buy/Hold/Sell” signals. Before processing the data, they used Random Forest (RF) for feature pruning. The authors proposed a practical model designed for real-life investment activities, which could generate three basic signals for investors to refer to. They also performed a thorough comparison of related algorithms. While they did not mention the time and computational complexity of their works. Meanwhile, the unignorable issue of their work was the lack of financial domain knowledge background. The investors regard the indices data as one of the attributes but could not take the signal from indices to operate a specific stock straightforward.

Hsu in [ 11 ] assembled feature selection with a back propagation neural network (BNN) combined with genetic programming to predict the stock/futures price. The dataset in this research was obtained from Taiwan Stock Exchange Corporation (TWSE). The authors have introduced the description of the background knowledge in detail. While the weakness of their work is that it is a lack of data set description. This is a combination of the model proposed by other previous works. Though we did not see the novelty of this work, we can still conclude that the genetic programming (GP) algorithm is admitted in stock market research domain. To reinforce the validation strengths, it would be good to consider adding GP models into evaluation if the model is predicting a specific price.

Hafezi et al. in [ 7 ] built a bat-neural network multi-agent system (BN-NMAS) to predict stock price. The dataset was obtained from the Deutsche bundes-bank. They also applied the Bat algorithm (BA) for optimizing neural network weights. The authors illustrated their overall structure and logic of system design in clear flowcharts. While there were very few previous works that had performed on DAX data, it would be difficult to recognize if the model they proposed still has the generality if migrated on other datasets. The system design and feature selection logic are fascinating, which worth referring to. Their findings in optimization algorithms are also valuable for the research in the stock market price prediction research domain. It is worth trying the Bat algorithm (BA) when constructing neural network models.

Long et al. in [ 25 ] conducted a deep learning approach to predict the stock price movement. The dataset they used is the Chinese stock market index CSI 300. For predicting the stock price movement, they constructed a multi-filter neural network (MFNN) with stochastic gradient descent (SGD) and back propagation optimizer for learning NN parameters. The strength of this paper is that the authors exploited a novel model with a hybrid model constructed by different kinds of neural networks, it provides an inspiration for constructing hybrid neural network structures.

Atsalakis and Valavanis in [ 1 ] proposed a solution of a neuro-fuzzy system, which is composed of controller named as Adaptive Neuro Fuzzy Inference System (ANFIS), to achieve short-term stock price trend prediction. The noticeable strength of this work is the evaluation part. Not only did they compare their proposed system with the popular data models, but also compared with investment strategies. While the weakness that we found from their proposed solution is that their solution architecture is lack of optimization part, which might limit their model performance. Since our proposed solution is also focusing on short-term stock price trend prediction, this work is heuristic for our system design. Meanwhile, by comparing with the popular trading strategies from investors, their work inspired us to compare the strategies used by investors with techniques used by researchers.

Nekoeiqachkanloo et al. in [ 29 ] proposed a system with two different approaches for stock investment. The strengths of their proposed solution are obvious. First, it is a comprehensive system that consists of data pre-processing and two different algorithms to suggest the best investment portions. Second, the system also embedded with a forecasting component, which also retains the features of the time series. Last but not least, their input features are a mix of fundamental features and technical indices that aim to fill in the gap between the financial domain and technical domain. However, their work has a weakness in the evaluation part. Instead of evaluating the proposed system on a large dataset, they chose 25 well-known stocks. There is a high possibility that the well-known stocks might potentially share some common hidden features.

As another related latest work, Idrees et al. [ 14 ] published a time series-based prediction approach for the volatility of the stock market. ARIMA is not a new approach in the time series prediction research domain. Their work is more focusing on the feature engineering side. Before feeding the features into ARIMA models, they designed three steps for feature engineering: Analyze the time series, identify if the time series is stationary or not, perform estimation by plot ACF and PACF charts and look for parameters. The only weakness of their proposed solution is that the authors did not perform any customization on the existing ARIMA model, which might limit the system performance to be improved.

One of the main weaknesses found in the related works is limited data-preprocessing mechanisms built and used. Technical works mostly tend to focus on building prediction models. When they select the features, they list all the features mentioned in previous works and go through the feature selection algorithm then select the best-voted features. Related works in the investment domain have shown more interest in behavior analysis, such as how herding behaviors affect the stock performance, or how the percentage of inside directors hold the firm’s common stock affects the performance of a certain stock. These behaviors often need a pre-processing procedure of standard technical indices and investment experience to recognize.

In the related works, often a thorough statistical analysis is performed based on a special dataset and conclude new features rather than performing feature selections. Some data, such as the percentage of a certain index fluctuation has been proven to be effective on stock performance. We believe that by extracting new features from data, then combining such features with existed common technical indices will significantly benefit the existing and well-tested prediction models.

The dataset

This section details the data that was extracted from the public data sources, and the final dataset that was prepared. Stock market-related data are diverse, so we first compared the related works from the survey of financial research works in stock market data analysis to specify the data collection directions. After collecting the data, we defined a data structure of the dataset. Given below, we describe the dataset in detail, including the data structure, and data tables in each category of data with the segment definitions.

Description of our dataset

In this section, we will describe the dataset in detail. This dataset consists of 3558 stocks from the Chinese stock market. Besides the daily price data, daily fundamental data of each stock ID, we also collected the suspending and resuming history, top 10 shareholders, etc. We list two reasons that we choose 2 years as the time span of this dataset: (1) most of the investors perform stock market price trend analysis using the data within the latest 2 years, (2) using more recent data would benefit the analysis result. We collected data through the open-sourced API, namely Tushare [ 43 ], mean-while we also leveraged a web-scraping technique to collect data from Sina Finance web pages, SWS Research website.

Data structure

Figure  1 illustrates all the data tables in the dataset. We collected four categories of data in this dataset: (1) basic data, (2) trading data, (3) finance data, and (4) other reference data. All the data tables can be linked to each other by a common field called “Stock ID” It is a unique stock identifier registered in the Chinese Stock market. Table  1 shows an overview of the dataset.

Data structure for the extracted dataset

The Table  1 lists the field information of each data table as well as which category the data table belongs to.

In this section, we present the proposed methods and the design of the proposed solution. Moreover, we also introduce the architecture design as well as algorithmic and implementation details.

Problem statement

We analyzed the best possible approach for predicting short-term price trends from different aspects: feature engineering, financial domain knowledge, and prediction algorithm. Then we addressed three research questions in each aspect, respectively: How can feature engineering benefit model prediction accuracy? How do findings from the financial domain benefit prediction model design? And what is the best algorithm for predicting short-term price trends?

The first research question is about feature engineering. We would like to know how the feature selection method benefits the performance of prediction models. From the abundance of the previous works, we can conclude that stock price data embedded with a high level of noise, and there are also correlations between features, which makes the price prediction notoriously difficult. That is also the primary reason for most of the previous works introduced the feature engineering part as an optimization module.

The second research question is evaluating the effectiveness of findings we extracted from the financial domain. Different from the previous works, besides the common evaluation of data models such as the training costs and scores, our evaluation will emphasize the effectiveness of newly added features that we extracted from the financial domain. We introduce some features from the financial domain. While we only obtained some specific findings from previous works, and the related raw data needs to be processed into usable features. After extracting related features from the financial domain, we combine the features with other common technical indices for voting out the features with a higher impact. There are numerous features said to be effective from the financial domain, and it would be impossible for us to cover all of them. Thus, how to appropriately convert the findings from the financial domain to a data processing module of our system design is a hidden research question that we attempt to answer.

The third research question is that which algorithms are we going to model our data? From the previous works, researchers have been putting efforts into the exact price prediction. We decompose the problem into predicting the trend and then the exact number. This paper focuses on the first step. Hence, the objective has been converted to resolve a binary classification problem, meanwhile, finding an effective way to eliminate the negative effect brought by the high level of noise. Our approach is to decompose the complex problem into sub-problems which have fewer dependencies and resolve them one by one, and then compile the resolutions into an ensemble model as an aiding system for investing behavior reference.

In the previous works, researchers have been using a variety of models for predicting stock price trends. While most of the best-performed models are based on machine learning techniques, in this work, we will compare our approach with the outperformed machine learning models in the evaluation part and find the solution for this research question.

Proposed solution

The high-level architecture of our proposed solution could be separated into three parts. First is the feature selection part, to guarantee the selected features are highly effective. Second, we look into the data and perform the dimensionality reduction. And the last part, which is the main contribution of our work is to build a prediction model of target stocks. Figure  2 depicts a high-level architecture of the proposed solution.

High-level architecture of the proposed solution

There are ways to classify different categories of stocks. Some investors prefer long-term investments, while others show more interest in short-term investments. It is common to see the stock-related reports showing an average performance, while the stock price is increasing drastically; this is one of the phenomena that indicate the stock price prediction has no fixed rules, thus finding effective features before training a model on data is necessary.

In this research, we focus on the short-term price trend prediction. Currently, we only have the raw data with no labels. So, the very first step is to label the data. We mark the price trend by comparing the current closing price with the closing price of n trading days ago, the range of n is from 1 to 10 since our research is focusing on the short-term. If the price trend goes up, we mark it as 1 or mark as 0 in the opposite case. To be more specified, we use the indices from the indices of n  −  1 th day to predict the price trend of the n th day.

According to the previous works, some researchers who applied both financial domain knowledge and technical methods on stock data were using rules to filter the high-quality stocks. We referred to their works and exploited their rules to contribute to our feature extension design.

However, to ensure the best performance of the prediction model, we will look into the data first. There are a large number of features in the raw data; if we involve all the features into our consideration, it will not only drastically increase the computational complexity but will also cause side effects if we would like to perform unsupervised learning in further research. So, we leverage the recursive feature elimination (RFE) to ensure all the selected features are effective.

We found most of the previous works in the technical domain were analyzing all the stocks, while in the financial domain, researchers prefer to analyze the specific scenario of investment, to fill the gap between the two domains, we decide to apply a feature extension based on the findings we gathered from the financial domain before we start the RFE procedure.

Since we plan to model the data into time series, the number of the features, the more complex the training procedure will be. So, we will leverage the dimensionality reduction by using randomized PCA at the beginning of our proposed solution architecture.

Detailed technical design elaboration

This section provides an elaboration of the detailed technical design as being a comprehensive solution based on utilizing, combining, and customizing several existing data preprocessing, feature engineering, and deep learning techniques. Figure  3 provides the detailed technical design from data processing to prediction, including the data exploration. We split the content by main procedures, and each procedure contains algorithmic steps. Algorithmic details are elaborated in the next section. The contents of this section will focus on illustrating the data workflow.

Detailed technical design of the proposed solution

Based on the literature review, we select the most commonly used technical indices and then feed them into the feature extension procedure to get the expanded feature set. We will select the most effective i features from the expanded feature set. Then we will feed the data with i selected features into the PCA algorithm to reduce the dimension into j features. After we get the best combination of i and j , we process the data into finalized the feature set and feed them into the LSTM [ 10 ] model to get the price trend prediction result.

The novelty of our proposed solution is that we will not only apply the technical method on raw data but also carry out the feature extensions that are used among stock market investors. Details on feature extension are given in the next subsection. Experiences gained from applying and optimizing deep learning based solutions in [ 37 , 38 ] were taken into account while designing and customizing feature engineering and deep learning solution in this work.

Applying feature extension

The first main procedure in Fig.  3 is the feature extension. In this block, the input data is the most commonly used technical indices concluded from related works. The three feature extension methods are max–min scaling, polarizing, and calculating fluctuation percentage. Not all the technical indices are applicable for all three of the feature extension methods; this procedure only applies the meaningful extension methods on technical indices. We choose meaningful extension methods while looking at how the indices are calculated. The technical indices and the corresponding feature extension methods are illustrated in Table  2 .

After the feature extension procedure, the expanded features will be combined with the most commonly used technical indices, i.e., input data with output data, and feed into RFE block as input data in the next step.

Applying recursive feature elimination

After the feature extension above, we explore the most effective i features by using the Recursive Feature Elimination (RFE) algorithm [ 6 ]. We estimate all the features by two attributes, coefficient, and feature importance. We also limit the features that remove from the pool by one, which means we will remove one feature at each step and retain all the relevant features. Then the output of the RFE block will be the input of the next step, which refers to PCA.

Applying principal component analysis (PCA)

The very first step before leveraging PCA is feature pre-processing. Because some of the features after RFE are percentage data, while others are very large numbers, i.e., the output from RFE are in different units. It will affect the principal component extraction result. Thus, before feeding the data into the PCA algorithm [ 8 ], a feature pre-processing is necessary. We also illustrate the effectiveness and methods comparison in “ Results ” section.

After performing feature pre-processing, the next step is to feed the processed data with selected i features into the PCA algorithm to reduce the feature matrix scale into j features. This step is to retain as many effective features as possible and meanwhile eliminate the computational complexity of training the model. This research work also evaluates the best combination of i and j, which has relatively better prediction accuracy, meanwhile, cuts the computational consumption. The result can be found in the “ Results ” section, as well. After the PCA step, the system will get a reshaped matrix with j columns.

Fitting long short-term memory (LSTM) model

PCA reduced the dimensions of the input data, while the data pre-processing is mandatory before feeding the data into the LSTM layer. The reason for adding the data pre-processing step before the LSTM model is that the input matrix formed by principal components has no time steps. While one of the most important parameters of training an LSTM is the number of time steps. Hence, we have to model the matrix into corresponding time steps for both training and testing dataset.

After performing the data pre-processing part, the last step is to feed the training data into LSTM and evaluate the performance using testing data. As a variant neural network of RNN, even with one LSTM layer, the NN structure is still a deep neural network since it can process sequential data and memorizes its hidden states through time. An LSTM layer is composed of one or more LSTM units, and an LSTM unit consists of cells and gates to perform classification and prediction based on time series data.

The LSTM structure is formed by two layers. The input dimension is determined by j after the PCA algorithm. The first layer is the input LSTM layer, and the second layer is the output layer. The final output will be 0 or 1 indicates if the stock price trend prediction result is going down or going up, as a supporting suggestion for the investors to perform the next investment decision.

Design discussion

Feature extension is one of the novelties of our proposed price trend predicting system. In the feature extension procedure, we use technical indices to collaborate with the heuristic processing methods learned from investors, which fills the gap between the financial research area and technical research area.

Since we proposed a system of price trend prediction, feature engineering is extremely important to the final prediction result. Not only the feature extension method is helpful to guarantee we do not miss the potentially correlated feature, but also feature selection method is necessary for pooling the effective features. The more irrelevant features are fed into the model, the more noise would be introduced. Each main procedure is carefully considered contributing to the whole system design.

Besides the feature engineering part, we also leverage LSTM, the state-of-the-art deep learning method for time-series prediction, which guarantees the prediction model can capture both complex hidden pattern and the time-series related pattern.

It is known that the training cost of deep learning models is expansive in both time and hardware aspects; another advantage of our system design is the optimization procedure—PCA. It can retain the principal components of the features while reducing the scale of the feature matrix, thus help the system to save the training cost of processing the large time-series feature matrix.

Algorithm elaboration

This section provides comprehensive details on the algorithms we built while utilizing and customizing different existing techniques. Details about the terminologies, parameters, as well as optimizers. From the legend on the right side of Fig.  3 , we note the algorithm steps as octagons, all of them can be found in this “ Algorithm elaboration ” section.

Before dive deep into the algorithm steps, here is the brief introduction of data pre-processing: since we will go through the supervised learning algorithms, we also need to program the ground truth. The ground truth of this research is programmed by comparing the closing price of the current trading date with the closing price of the previous trading date the users want to compare with. Label the price increase as 1, else the ground truth will be labeled as 0. Because this research work is not only focused on predicting the price trend of a specific period of time but short-term in general, the ground truth processing is according to a range of trading days. While the algorithms will not change with the prediction term length, we can regard the term length as a parameter.

The algorithmic detail is elaborated, respectively, the first algorithm is the hybrid feature engineering part for preparing high-quality training and testing data. It corresponds to the Feature extension, RFE, and PCA blocks in Fig.  3 . The second algorithm is the LSTM procedure block, including time-series data pre-processing, NN constructing, training, and testing.

Algorithm 1: Short-term stock market price trend prediction—applying feature engineering using FE + RFE + PCA

The function FE is corresponding to the feature extension block. For the feature extension procedure, we apply three different processing methods to translate the findings from the financial domain to a technical module in our system design. While not all the indices are applicable for expanding, we only choose the proper method(s) for certain features to perform the feature extension (FE), according to Table  2 .

Normalize method preserves the relative frequencies of the terms, and transform the technical indices into the range of [0, 1]. Polarize is a well-known method often used by real-world investors, sometimes they prefer to consider if the technical index value is above or below zero, we program some of the features using polarize method and prepare for RFE. Max-min (or min-max) [ 35 ] scaling is a transformation method often used as an alternative to zero mean and unit variance scaling. Another well-known method used is fluctuation percentage, and we transform the technical indices fluctuation percentage into the range of [− 1, 1].

The function RFE () in the first algorithm refers to recursive feature elimination. Before we perform the training data scale reduction, we will have to make sure that the features we selected are effective. Ineffective features will not only drag down the classification precision but also add more computational complexity. For the feature selection part, we choose recursive feature elimination (RFE). As [ 45 ] explained, the process of recursive feature elimination can be split into the ranking algorithm, resampling, and external validation.

For the ranking algorithm, it fits the model to the features and ranks by the importance to the model. We set the parameter to retain i numbers of features, and at each iteration of feature selection retains Si top-ranked features, then refit the model and assess the performance again to begin another iteration. The ranking algorithm will eventually determine the top Si features.

The RFE algorithm is known to have suffered from the over-fitting problem. To eliminate the over-fitting issue, we will run the RFE algorithm multiple times on randomly selected stocks as the training set and ensure all the features we select are high-weighted. This procedure is called data resampling. Resampling can be built as an optimization step as an outer layer of the RFE algorithm.

The last part of our hybrid feature engineering algorithm is for optimization purposes. For the training data matrix scale reduction, we apply Randomized principal component analysis (PCA) [ 31 ], before we decide the features of the classification model.

Financial ratios of a listed company are used to present the growth ability, earning ability, solvency ability, etc. Each financial ratio consists of a set of technical indices, each time we add a technical index (or feature) will add another column of data into the data matrix and will result in low training efficiency and redundancy. If non-relevant or less relevant features are included in training data, it will also decrease the precision of classification.

The above equation represents the explanation power of principal components extracted by PCA method for original data. If an ACR is below 85%, the PCA method would be unsuitable due to a loss of original information. Because the covariance matrix is sensitive to the order of magnitudes of data, there should be a data standardize procedure before performing the PCA. The commonly used standardized methods are mean-standardization and normal-standardization and are noted as given below:

Mean-standardization: \(X_{ij}^{*} = X_{ij} /\overline{{X_{j} }}\) , which \(\overline{{X_{j} }}\) represents the mean value.

Normal-standardization: \(X_{ij}^{*} = (X_{ij} - \overline{{X_{j} }} )/s_{j}\) , which \(\overline{{X_{j} }}\) represents the mean value, and \(s_{j}\) is the standard deviation.

The array fe_array is defined according to Table  2 , row number maps to the features, columns 0, 1, 2, 3 note for the extension methods of normalize, polarize, max–min scale, and fluctuation percentage, respectively. Then we fill in the values for the array by the rule where 0 stands for no necessity to expand and 1 for features need to apply the corresponding extension methods. The final algorithm of data preprocessing using RFE and PCA can be illustrated as Algorithm 1.

Algorithm 2: Price trend prediction model using LSTM

After the principal component extraction, we will get the scale-reduced matrix, which means i most effective features are converted into j principal components for training the prediction model. We utilized an LSTM model and added a conversion procedure for our stock price dataset. The detailed algorithm design is illustrated in Alg 2. The function TimeSeriesConversion () converts the principal components matrix into time series by shifting the input data frame according to the number of time steps [ 3 ], i.e., term length in this research. The processed dataset consists of the input sequence and forecast sequence. In this research, the parameter of LAG is 1, because the model is detecting the pattern of features fluctuation on a daily basis. Meanwhile, the N_TIME_STEPS is varied from 1 trading day to 10 trading days. The functions DataPartition (), FitModel (), EvaluateModel () are regular steps without customization. The NN structure design, optimizer decision, and other parameters are illustrated in function ModelCompile () .

Some procedures impact the efficiency but do not affect the accuracy or precision and vice versa, while other procedures may affect both efficiency and prediction result. To fully evaluate our algorithm design, we structure the evaluation part by main procedures and evaluate how each procedure affects the algorithm performance. First, we evaluated our solution on a machine with 2.2 GHz i7 processor, with 16 GB of RAM. Furthermore, we also evaluated our solution on Amazon EC2 instance, 3.1 GHz Processor with 16 vCPUs, and 64 GB RAM.

In the implementation part, we expanded 20 features into 54 features, while we retain 30 features that are the most effective. In this section, we discuss the evaluation of feature selection. The dataset was divided into two different subsets, i.e., training and testing datasets. Test procedure included two parts, one testing dataset is for feature selection, and another one is for model testing. We note the feature selection dataset and model testing dataset as DS_test_f and DS_test_m, respectively.

We randomly selected two-thirds of the stock data by stock ID for RFE training and note the dataset as DS_train_f; all the data consist of full technical indices and expanded features throughout 2018. The estimator of the RFE algorithm is SVR with linear kernels. We rank the 54 features by voting and get 30 effective features then process them using the PCA algorithm to perform dimension reduction and reduce the features into 20 principal components. The rest of the stock data forms the testing dataset DS_test_f to validate the effectiveness of principal components we extracted from selected features. We reformed all the data from 2018 as the training dataset of the data model and noted as DS_train_m. The model testing dataset DS_test_m consists of the first 3 months of data in 2019, which has no overlap with the dataset we utilized in the previous steps. This approach is to prevent the hidden problem caused by overfitting.

Term length

To build an efficient prediction model, instead of the approach of modeling the data to time series, we determined to use 1 day ahead indices data to predict the price trend of the next day. We tested the RFE algorithm on a range of short-term from 1 day to 2 weeks (ten trading days), to evaluate how the commonly used technical indices correlated to price trends. For evaluating the prediction term length, we fully expanded the features as Table  2 , and feed them to RFE. During the test, we found that different length of the term has a different level of sensitive-ness to the same indices set.

We get the close price of the first trading date and compare it with the close price of the n _ th trading date. Since we are predicting the price trend, we do not consider the term lengths if the cross-validation score is below 0.5. And after the test, as we can see from Fig.  4 , there are three-term lengths that are most sensitive to the indices we selected from the related works. They are n  = {2, 5, 10}, which indicates that price trend prediction of every other day, 1 week, and 2 weeks using the indices set are likely to be more reliable.

How do term lengths affect the cross-validation score of RFE

While these curves have different patterns, for the length of 2 weeks, the cross-validation score increases with the number of features selected. If the prediction term length is 1 week, the cross-validation score will decrease if selected over 8 features. For every other day price trend prediction, the best cross-validation score is achieved by selecting 48 features. Biweekly prediction requires 29 features to achieve the best score. In Table  3 , we listed the top 15 effective features for these three-period lengths. If we predict the price trend of every other day, the cross-validation score merely fluctuates with the number of features selected. So, in the next step, we will evaluate the RFE result for these three-term lengths, as shown in Fig.  4 .

We compare the output feature set of RFE with the all-original feature set as a baseline, the all-original feature set consists of n features and we choose n most effective features from RFE output features to evaluate the result using linear SVR. We used two different approaches to evaluate feature effectiveness. The first method is to combine all the data into one large matrix and evaluate them by running the RFE algorithm once. Another method is to run RFE for each individual stock and calculate the most effective features by voting.

Feature extension and RFE

From the result of the previous subsection, we can see that when predicting the price trend for every other day or biweekly, the best result is achieved by selecting a large number of features. Within the selected features, some features processed from extension methods have better ranks than original features, which proves that the feature extension method is useful for optimizing the model. The feature extension affects both precision and efficiency, while in this part, we only discuss the precision aspect and leave efficiency part in the next step since PCA is the most effective method for training efficiency optimization in our design. We involved an evaluation of how feature extension affects RFE and use the test result to measure the improvement of involving feature extension.

We further test the effectiveness of feature extension, i.e., if polarize, max–min scale, and calculate fluctuation percentage works better than original technical indices. The best case to leverage this test is the weekly prediction since it has the least effective feature selected. From the result we got from the last section, we know the best cross-validation score appears when selecting 8 features. The test consists of two steps, and the first step is to test the feature set formed by original features only, in this case, only SLOWK, SLOWD, and RSI_5 are included. The next step is to test the feature set of all 8 features we selected in the previous subsection. We leveraged the test by defining the simplest DNN model with three layers.

The normalized confusion matrix of testing the two feature sets are illustrated in Fig.  5 . The left one is the confusion matrix of the feature set with expanded features, and the right one besides is the test result of using original features only. Both precisions of true positive and true negative have been improved by 7% and 10%, respectively, which proves that our feature extension method design is reasonably effective.

Confusion matrix of validating feature extension effectiveness

Feature reduction using principal component analysis

PCA will affect the algorithm performance on both prediction accuracy and training efficiency, while this part should be evaluated with the NN model, so we also defined the simplest DNN model with three layers as we used in the previous step to perform the evaluation. This part introduces the evaluation method and result of the optimization part of the model from computational efficiency and accuracy impact perspectives.

In this section, we will choose bi-weekly prediction to perform a use case analysis, since it has a smoothly increasing cross-validation score curve, moreover, unlike every other day prediction, it has excluded more than 20 ineffective features already. In the first step, we select all 29 effective features and train the NN model without performing PCA. It creates a baseline of the accuracy and training time for comparison. To evaluate the accuracy and efficiency, we keep the number of the principal component as 5, 10, 15, 20, 25. Table  4 recorded how the number of features affects the model training efficiency, then uses the stack bar chart in Fig.  6 to illustrate how PCA affects training efficiency. Table  6 shows accuracy and efficiency analysis on different procedures for the pre-processing of features. The times taken shown in Tables  4 , 6 are based on experiments conducted in a standard user machine to show the viability of our solution with limited or average resource availability.

Relationship between feature number and training time

We also listed the confusion matrix of each test in Fig.  7 . The stack bar chart shows that the overall time spends on training the model is decreasing by the number of selected features, while the PCA method is significantly effective in optimizing training dataset preparation. For the time spent on the training stage, PCA is not as effective as the data preparation stage. While there is the possibility that the optimization effect of PCA is not drastic enough because of the simple structure of the NN model.

How does the number of principal components affect evaluation results

Table  5 indicates that the overall prediction accuracy is not drastically affected by reducing the dimension. However, the accuracy could not fully support if the PCA has no side effect to model prediction, so we looked into the confusion matrices of test results.

From Fig.  7 we can conclude that PCA does not have a severe negative impact on prediction precision. The true positive rate and false positive rate are barely be affected, while the false negative and true negative rates are influenced by 2% to 4%. Besides evaluating how the number of selected features affects the training efficiency and model performance, we also leveraged a test upon how data pre-processing procedures affect the training procedure and predicting result. Normalizing and max–min scaling is the most commonly seen data pre-procedure performed before PCA, since the measure units of features are varied, and it is said that it could increase the training efficiency afterward.

We leveraged another test on adding pre-procedures before extracting 20 principal components from the original dataset and make the comparison in the aspects of time elapse of training stage and prediction precision. However, the test results lead to different conclusions. In Table  6 we can conclude that feature pre-processing does not have a significant impact on training efficiency, but it does influence the model prediction accuracy. Moreover, the first confusion matrix in Fig.  8 indicates that without any feature pre-processing procedure, the false-negative rate and true negative rate are severely affected, while the true positive rate and false positive rate are not affected. If it performs the normalization before PCA, both true positive rate and true negative rate are decreasing by approximately 10%. This test also proved that the best feature pre-processing method for our feature set is exploiting the max–min scale.

Confusion matrices of different feature pre-processing methods

In this section, we discuss and compare the results of our proposed model, other approaches, and the most related works.

Comparison with related works

From the previous works, we found the most commonly exploited models for short-term stock market price trend prediction are support vector machine (SVM), multilayer perceptron artificial neural network (MLP), Naive Bayes classifier (NB), random forest classifier (RAF) and logistic regression classifier (LR). The test case of comparison is also bi-weekly price trend prediction, to evaluate the best result of all models, we keep all 29 features selected by the RFE algorithm. For MLP evaluation, to test if the number of hidden layers would affect the metric scores, we noted layer number as n and tested n  = {1, 3, 5}, 150 training epochs for all the tests, found slight differences in the model performance, which indicates that the variable of MLP layer number hardly affects the metric scores.

From the confusion matrices in Fig.  9 , we can see all the machine learning models perform well when training with the full feature set we selected by RFE. From the perspective of training time, training the NB model got the best efficiency. LR algorithm cost less training time than other algorithms while it can achieve a similar prediction result with other costly models such as SVM and MLP. RAF algorithm achieved a relatively high true-positive rate while the poor performance in predicting negative labels. For our proposed LSTM model, it achieves a binary accuracy of 93.25%, which is a significantly high precision of predicting the bi-weekly price trend. We also pre-processed data through PCA and got five principal components, then trained for 150 epochs. The learning curve of our proposed solution, based on feature engineering and the LSTM model, is illustrated in Fig.  10 . The confusion matrix is the figure on the right in Fig.  11 , and detailed metrics scores can be found in Table  9 .

Model prediction comparison—confusion matrices

Learning curve of proposed solution

Proposed model prediction precision comparison—confusion matrices

The detailed evaluate results are recorded in Table  7 . We will also initiate a discussion upon the evaluation result in the next section.

Because the resulting structure of our proposed solution is different from most of the related works, it would be difficult to make naïve comparison with previous works. For example, it is hard to find the exact accuracy number of price trend prediction in most of the related works since the authors prefer to show the gain rate of simulated investment. Gain rate is a processed number based on simulated investment tests, sometimes one correct investment decision with a large trading volume can achieve a high gain rate regardless of the price trend prediction accuracy. Besides, it is also a unique and heuristic innovation in our proposed solution, we transform the problem of predicting an exact price straight forward to two sequential problems, i.e., predicting the price trend first, focus on building an accurate binary classification model, construct a solid foundation for predicting the exact price change in future works. Besides the different result structure, the datasets that previous works researched on are also different from our work. Some of the previous works involve news data to perform sentiment analysis and exploit the SE part as another system component to support their prediction model.

The latest related work that can compare is Zubair et al. [ 47 ], the authors take multiple r-square for model accuracy measurement. Multiple r-square is also called the coefficient of determination, and it shows the strength of predictor variables explaining the variation in stock return [ 28 ]. They used three datasets (KSE 100 Index, Lucky Cement Stock, Engro Fertilizer Limited) to evaluate the proposed multiple regression model and achieved 95%, 89%, and 97%, respectively. Except for the KSE 100 Index, the dataset choice in this related work is individual stocks; thus, we choose the evaluation result of the first dataset of their proposed model.

We listed the leading stock price trend prediction model performance in Table  8 , from the comparable metrics, the metric scores of our proposed solution are generally better than other related works. Instead of concluding arbitrarily that our proposed model outperformed other models in related works, we first look into the dataset column of Table  8 . By looking into the dataset used by each work [ 18 ], only trained and tested their proposed solution on three individual stocks, which is difficult to prove the generalization of their proposed model. Ayo [ 2 ] leveraged analysis on the stock data from the New York Stock Exchange (NYSE), while the weakness is they only performed analysis on closing price, which is a feature embedded with high noise. Zubair et al. [ 47 ] trained their proposed model on both individual stocks and index price, but as we have mentioned in the previous section, index price only consists of the limited number of features and stock IDs, which will further affect the model training quality. For our proposed solution, we collected sufficient data from the Chinese stock market, and applied FE + RFE algorithm on the original indices to get more effective features, the comprehensive evaluation result of 3558 stock IDs can reasonably explain the generalization and effectiveness of our proposed solution in Chinese stock market. However, the authors of Khaidem and Dey [ 18 ] and Ayo [ 2 ] chose to analyze the stock market in the United States, Zubair et al. [ 47 ] performed analysis on Pakistani stock market price, and we obtained the dataset from Chinese stock market, the policies of different countries might impact the model performance, which needs further research to validate.

Proposed model evaluation—PCA effectiveness

Besides comparing the performance across popular machine learning models, we also evaluated how the PCA algorithm optimizes the training procedure of the proposed LSTM model. We recorded the confusion matrices comparison between training the model by 29 features and by five principal components in Fig.  11 . The model training using the full 29 features takes 28.5 s per epoch on average. While it only takes 18 s on average per epoch training on the feature set of five principal components. PCA has significantly improved the training efficiency of the LSTM model by 36.8%. The detailed metrics data are listed in Table  9 . We will leverage a discussion in the next section about complexity analysis.

Complexity analysis of proposed solution

This section analyzes the complexity of our proposed solution. The Long Short-term Memory is different from other NNs, and it is a variant of standard RNN, which also has time steps with memory and gate architecture. In the previous work [ 46 ], the author performed an analysis of the RNN architecture complexity. They introduced a method to regard RNN as a directed acyclic graph and proposed a concept of recurrent depth, which helps perform the analysis on the intricacy of RNN.

The recurrent depth is a positive rational number, and we denote it as \(d_{rc}\) . As the growth of \(n\) \(d_{rc}\) measures, the nonlinear transformation average maximum number of each time step. We then unfold the directed acyclic graph of RNN and denote the processed graph as \(g_{c}\) , meanwhile, denote \(C(g_{c} )\) as the set of directed cycles in this graph. For the vertex \(v\) , we note \(\sigma_{s} (v)\) as the sum of edge weights and \(l(v)\) as the length. The equation below is proved under a mild assumption, which could be found in [ 46 ].

They also found that another crucial factor that impacts the performance of LSTM, which is the recurrent skip coefficients. We note \(s_{rc}\) as the reciprocal of the recurrent skip coefficient. Please be aware that \(s_{rc}\) is also a positive rational number.

According to the above definition, our proposed model is a 2-layers stacked LSTM, which \(d_{rc} = 2\) and \(s_{rc} = 1\) . From the experiments performed in previous work, the authors also found that when facing the problems of long-term dependency, LSTMs may benefit from decreasing the reciprocal of recurrent skip coefficients and from increasing recurrent depth. The empirical findings above mentioned are useful to enhance the performance of our proposed model further.

This work consists of three parts: data extraction and pre-processing of the Chinese stock market dataset, carrying out feature engineering, and stock price trend prediction model based on the long short-term memory (LSTM). We collected, cleaned-up, and structured 2 years of Chinese stock market data. We reviewed different techniques often used by real-world investors, developed a new algorithm component, and named it as feature extension, which is proved to be effective. We applied the feature expansion (FE) approaches with recursive feature elimination (RFE), followed by principal component analysis (PCA), to build a feature engineering procedure that is both effective and efficient. The system is customized by assembling the feature engineering procedure with an LSTM prediction model, achieved high prediction accuracy that outperforms the leading models in most related works. We also carried out a comprehensive evaluation of this work. By comparing the most frequently used machine learning models with our proposed LSTM model under the feature engineering part of our proposed system, we conclude many heuristic findings that could be future research questions in both technical and financial research domains.

Our proposed solution is a unique customization as compared to the previous works because rather than just proposing yet another state-of-the-art LSTM model, we proposed a fine-tuned and customized deep learning prediction system along with utilization of comprehensive feature engineering and combined it with LSTM to perform prediction. By researching into the observations from previous works, we fill in the gaps between investors and researchers by proposing a feature extension algorithm before recursive feature elimination and get a noticeable improvement in the model performance.

Though we have achieved a decent outcome from our proposed solution, this research has more potential towards research in future. During the evaluation procedure, we also found that the RFE algorithm is not sensitive to the term lengths other than 2-day, weekly, biweekly. Getting more in-depth research into what technical indices would influence the irregular term lengths would be a possible future research direction. Moreover, by combining latest sentiment analysis techniques with feature engineering and deep learning model, there is also a high potential to develop a more comprehensive prediction system which is trained by diverse types of information such as tweets, news, and other text-based data.

Abbreviations

Long short term memory

Principal component analysis

Recurrent neural networks

Artificial neural network

Deep neural network

Dynamic Time Warping

Recursive feature elimination

Support vector machine

Convolutional neural network

Stochastic gradient descent

Rectified linear unit

Multi layer perceptron

Atsalakis GS, Valavanis KP. Forecasting stock market short-term trends using a neuro-fuzzy based methodology. Expert Syst Appl. 2009;36(7):10696–707.

Article   Google Scholar  

Ayo CK. Stock price prediction using the ARIMA model. In: 2014 UKSim-AMSS 16th international conference on computer modelling and simulation. 2014. https://doi.org/10.1109/UKSim.2014.67 .

Brownlee J. Deep learning for time series forecasting: predict the future with MLPs, CNNs and LSTMs in Python. Machine Learning Mastery. 2018. https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/

Eapen J, Bein D, Verma A. Novel deep learning model with CNN and bi-directional LSTM for improved stock market index prediction. In: 2019 IEEE 9th annual computing and communication workshop and conference (CCWC). 2019. pp. 264–70. https://doi.org/10.1109/CCWC.2019.8666592 .

Fischer T, Krauss C. Deep learning with long short-term memory networks for financial market predictions. Eur J Oper Res. 2018;270(2):654–69. https://doi.org/10.1016/j.ejor.2017.11.054 .

Article   MathSciNet   MATH   Google Scholar  

Guyon I, Weston J, Barnhill S, Vapnik V. Gene selection for cancer classification using support vector machines. Mach Learn 2002;46:389–422.

Hafezi R, Shahrabi J, Hadavandi E. A bat-neural network multi-agent system (BNNMAS) for stock price prediction: case study of DAX stock price. Appl Soft Comput J. 2015;29:196–210. https://doi.org/10.1016/j.asoc.2014.12.028 .

Halko N, Martinsson PG, Tropp JA. Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev. 2001;53(2):217–88.

Article   MathSciNet   Google Scholar  

Hassan MR, Nath B. Stock market forecasting using Hidden Markov Model: a new approach. In: Proceedings—5th international conference on intelligent systems design and applications 2005, ISDA’05. 2005. pp. 192–6. https://doi.org/10.1109/ISDA.2005.85 .

Hochreiter S, Schmidhuber J. Long short-term memory. J Neural Comput. 1997;9(8):1735–80. https://doi.org/10.1162/neco.1997.9.8.1735 .

Hsu CM. A hybrid procedure with feature selection for resolving stock/futures price forecasting problems. Neural Comput Appl. 2013;22(3–4):651–71. https://doi.org/10.1007/s00521-011-0721-4 .

Huang CF, Chang BR, Cheng DW, Chang CH. Feature selection and parameter optimization of a fuzzy-based stock selection model using genetic algorithms. Int J Fuzzy Syst. 2012;14(1):65–75. https://doi.org/10.1016/J.POLYMER.2016.08.021 .

Huang CL, Tsai CY. A hybrid SOFM-SVR with a filter-based feature selection for stock market forecasting. Expert Syst Appl. 2009;36(2 PART 1):1529–39. https://doi.org/10.1016/j.eswa.2007.11.062 .

Idrees SM, Alam MA, Agarwal P. A prediction approach for stock market volatility based on time series data. IEEE Access. 2019;7:17287–98. https://doi.org/10.1109/ACCESS.2019.2895252 .

Ince H, Trafalis TB. Short term forecasting with support vector machines and application to stock price prediction. Int J Gen Syst. 2008;37:677–87. https://doi.org/10.1080/03081070601068595 .

Jeon S, Hong B, Chang V. Pattern graph tracking-based stock price prediction using big data. Future Gener Comput Syst. 2018;80:171–87. https://doi.org/10.1016/j.future.2017.02.010 .

Kara Y, Acar Boyacioglu M, Baykan ÖK. Predicting direction of stock price index movement using artificial neural networks and support vector machines: the sample of the Istanbul Stock Exchange. Expert Syst Appl. 2011;38(5):5311–9. https://doi.org/10.1016/j.eswa.2010.10.027 .

Khaidem L, Dey SR. Predicting the direction of stock market prices using random forest. 2016. pp. 1–20.

Kim K, Han I. Genetic algorithms approach to feature discretization in artificial neural networks for the prediction of stock price index. Expert Syst Appl. 2000;19:125–32. https://doi.org/10.1016/S0957-4174(00)00027-0 .

Lee MC. Using support vector machine with a hybrid feature selection method to the stock trend prediction. Expert Syst Appl. 2009;36(8):10896–904. https://doi.org/10.1016/j.eswa.2009.02.038 .

Lei L. Wavelet neural network prediction method of stock price trend based on rough set attribute reduction. Appl Soft Comput J. 2018;62:923–32. https://doi.org/10.1016/j.asoc.2017.09.029 .

Lin X, Yang Z, Song Y. Expert systems with applications short-term stock price prediction based on echo state networks. Expert Syst Appl. 2009;36(3):7313–7. https://doi.org/10.1016/j.eswa.2008.09.049 .

Liu G, Wang X. A new metric for individual stock trend prediction. Eng Appl Artif Intell. 2019;82(March):1–12. https://doi.org/10.1016/j.engappai.2019.03.019 .

Liu S, Zhang C, Ma J. CNN-LSTM neural network model for quantitative strategy analysis in stock markets. 2017;1:198–206. https://doi.org/10.1007/978-3-319-70096-0 .

Long W, Lu Z, Cui L. Deep learning-based feature engineering for stock price movement prediction. Knowl Based Syst. 2018;164:163–73. https://doi.org/10.1016/j.knosys.2018.10.034 .

Malkiel BG, Fama EF. Efficient capital markets: a review of theory and empirical work. J Finance. 1970;25(2):383–417.

McNally S, Roche J, Caton S. Predicting the price of bitcoin using machine learning. In: Proceedings—26th Euromicro international conference on parallel, distributed, and network-based processing, PDP 2018. pp. 339–43. https://doi.org/10.1109/PDP2018.2018.00060 .

Nagar A, Hahsler M. News sentiment analysis using R to predict stock market trends. 2012. http://past.rinfinance.com/agenda/2012/talk/Nagar+Hahsler.pdf . Accessed 20 July 2019.

Nekoeiqachkanloo H, Ghojogh B, Pasand AS, Crowley M. Artificial counselor system for stock investment. 2019. ArXiv Preprint arXiv:1903.00955 .

Ni LP, Ni ZW, Gao YZ. Stock trend prediction based on fractal feature selection and support vector machine. Expert Syst Appl. 2011;38(5):5569–76. https://doi.org/10.1016/j.eswa.2010.10.079 .

Pang X, Zhou Y, Wang P, Lin W, Chang V. An innovative neural network approach for stock market prediction. J Supercomput. 2018. https://doi.org/10.1007/s11227-017-2228-y .

Pimenta A, Nametala CAL, Guimarães FG, Carrano EG. An automated investing method for stock market based on multiobjective genetic programming. Comput Econ. 2018;52(1):125–44. https://doi.org/10.1007/s10614-017-9665-9 .

Piramuthu S. Evaluating feature selection methods for learning in data mining applications. Eur J Oper Res. 2004;156(2):483–94. https://doi.org/10.1016/S0377-2217(02)00911-6 .

Qiu M, Song Y. Predicting the direction of stock market index movement using an optimized artificial neural network model. PLoS ONE. 2016;11(5):e0155133.

Scikit-learn. Scikit-learn Min-Max Scaler. 2019. https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html . Retrieved 26 July 2020.

Shen J. Thesis, “Short-term stock market price trend prediction using a customized deep learning system”, supervised by M. Omair Shafiq, Carleton University. 2019.

Shen J, Shafiq MO. Deep learning convolutional neural networks with dropout—a parallel approach. ICMLA. 2018;2018:572–7.

Google Scholar  

Shen J, Shafiq MO. Learning mobile application usage—a deep learning approach. ICMLA. 2019;2019:287–92.

Shih D. A study of early warning system in volume burst risk assessment of stock with Big Data platform. In: 2019 IEEE 4th international conference on cloud computing and big data analysis (ICCCBDA). 2019. pp. 244–8.

Sirignano J, Cont R. Universal features of price formation in financial markets: perspectives from deep learning. Ssrn. 2018. https://doi.org/10.2139/ssrn.3141294 .

Article   MATH   Google Scholar  

Thakur M, Kumar D. A hybrid financial trading support system using multi-category classifiers and random forest. Appl Soft Comput J. 2018;67:337–49. https://doi.org/10.1016/j.asoc.2018.03.006 .

Tsai CF, Hsiao YC. Combining multiple feature selection methods for stock prediction: union, intersection, and multi-intersection approaches. Decis Support Syst. 2010;50(1):258–69. https://doi.org/10.1016/j.dss.2010.08.028 .

Tushare API. 2018. https://github.com/waditu/tushare . Accessed 1 July 2019.

Wang X, Lin W. Stock market prediction using neural networks: does trading volume help in short-term prediction?. n.d.

Weng B, Lu L, Wang X, Megahed FM, Martinez W. Predicting short-term stock prices using ensemble methods and online data sources. Expert Syst Appl. 2018;112:258–73. https://doi.org/10.1016/j.eswa.2018.06.016 .

Zhang S. Architectural complexity measures of recurrent neural networks, (NIPS). 2016. pp. 1–9.

Zubair M, Fazal A, Fazal R, Kundi M. Development of stock market trend prediction system using multiple regression. Computational and mathematical organization theory. Berlin: Springer US; 2019. https://doi.org/10.1007/s10588-019-09292-7 .

Book   Google Scholar  

Download references

Acknowledgements

This research is supported by Carleton University, in Ottawa, ON, Canada. This research paper has been built based on the thesis [ 36 ] of Jingyi Shen, supervised by M. Omair Shafiq at Carleton University, Canada, available at https://curve.carleton.ca/52e9187a-7f71-48ce-bdfe-e3f6a420e31a .

NSERC and Carleton University.

Author information

Authors and affiliations.

School of Information Technology, Carleton University, Ottawa, ON, Canada

Jingyi Shen & M. Omair Shafiq

You can also search for this author in PubMed   Google Scholar

Contributions

Yes. All authors read and approved the final manuscript.

Corresponding author

Correspondence to M. Omair Shafiq .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Shen, J., Shafiq, M.O. Short-term stock market price trend prediction using a comprehensive deep learning system. J Big Data 7 , 66 (2020). https://doi.org/10.1186/s40537-020-00333-6

Download citation

Received : 24 January 2020

Accepted : 30 July 2020

Published : 28 August 2020

DOI : https://doi.org/10.1186/s40537-020-00333-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Deep learning
• Stock market trend
• Feature engineering

Predicting stock market using machine learning: best and accurate way to know future stock prices

• REVIEW PAPER
• Published: 09 January 2023
• Volume 14 , pages 1–18, ( 2023 )

Cite this article

• Dhruhi Sheth 1 &
• Manan Shah   ORCID: orcid.org/0000-0002-8665-5010 2  

2561 Accesses

10 Citations

Explore all metrics

Dissatisfaction is the first step of progress, this statement serves to be the base of using Artifcial Intelligence in predicting stock prices. A great deal of people dreamed of predicting stock prices faultlessly but it remained only as a dream for those visionaries at that time. The legacy of those visionaries led to the discovery of something concrete and made that dream come to reality, and due to this we can use machine learning methods in today’s era for predicting accurate stock prices. These methods have proved to be extremely beneficial and an easy way for common man to earn quick money if done appropriately. These methods still have drawbacks that are being worked upon and it confirmations immense improvement in the future unlike the prior methods of predicting stock market prices like time-series forecasting that didn’t provide results that satisfying the needs of an investor. As a result, to deal with the volatile and dynamic nature of the market, a link between stock market and Artificial Intelligence was founded that brought about wonders. The three methods that were implemented in the prediction process were Artificial Neural Network (ANN), Support Vector Machine (SVM) and Long Short-Term Memory (LSTM). ANN works on neural network, SVM works using Kernel method and LSTM works using Keras LSTM. Various techniques offered by each methodology are carefully analyzed and it was found that ANN based on neural network provides best results because it considers complex, non-linear relationships and recognizes patterns. While SVM is comparatively a new method and capable of providing better results in the future and LSTM gives good results only when large dataset is given which can be considered a drawback.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Machine learning and deep learning

Christian Janiesch, Patrick Zschech & Kai Heinrich

Artificial intelligence in Finance: a comprehensive review through bibliometric and content analysis

Salman Bahoo, Marco Cucculelli, … Jasmine Mondolo

Deep learning for time series classification: a review

Hassan Ismail Fawaz, Germain Forestier, … Pierre-Alain Muller

Data availability

All relevant data and material are presented in the main paper.

Abiodun O, Jantan A, Omolara A, Heliyon KD (2018). State-of-the-art in artificial neural network applications: A survey. Elsevier. https://www.sciencedirect.com/science/article/pii/S2405844018332067

Ahn JJ, Oh KJ, Kim TY, Kim DH (2011) Usefulness of support vector machine to develop an early warning system for financial crisis. Expert Syst Appl 38(4):2966–2973. https://doi.org/10.1016/J.ESWA.2010.08.085

Article   Google Scholar  

Aldin MM, Dehnavi HD, Entezari S (2012) Evaluating the employment of technical indicators in predicting stock price index variations using artificial neural networks (Case study: Tehran stock exchange). Int J Bus Manage 7:15. https://doi.org/10.5539/IJBM.v7n15p25

Ang KK, Quek C (2006) Stock trading using RSPOP: a novel rough set-based neuro-fuzzy approach. IEEE Trans Neural Networks 17(5):1301–1315. https://doi.org/10.1109/TNN.2006.875996

Angra S, Ahuja S (2017) Machine learning and its applications: a review. Proceedings of the 2017 international conference on big data analytics and computational intelligence, 57–60

Arestis P, Demetriades PO, Luintel KB (2001) Financial development and economic growth: the role of stock markets. J Money, Credit, Bank 33(1):16. https://doi.org/10.2307/2673870

Atiya AF, El-Shoura SM, Shaheen SI, El-Sherif MS (1999) A comparison between neural-network forecasting techniques-case study: river flow forecasting. IEEE Trans Neural Networks 10(2):402–409. https://doi.org/10.1109/72.750569

Bench-Capon TJM, Dunne PE (2007) Argumentation in artificial intelligence. Artif Intell 171(10):619–641. https://doi.org/10.1016/J.ARTINT.2007.05.001

Article   MATH   Google Scholar  

Billmeier A, Massa I, Billmeier A, Massa I (2009) What drives stock market development in emerging markets--institutions, remittances, or natural resources? Emerging Markets Review, 10(1):23–35. https://econpapers.repec.org/RePEc:eee:ememar:v:10:y:2009:i:1:p:23-35

Binoy Varkey S, Belfin RV, Paul GR (2020) Machine learning algorithms using stock market dataset-a comparative study. J Crit Rev 7(15):3517–3526

Google Scholar  

Bonde G, R. K. the I. C. on G., & 2012, undefined. (n.d.). Stock price prediction using genetic algorithms and evolution strategies. World-Comp.Org. Retrieved December 16, 2022, from http://world-comp.org/p2012/GEM4716.pdf

Borovkova S, Tsiamas I (2019) An ensemble of LSTM neural networks for high-frequency stock market classification. J Forecast 38(6):600–619. https://doi.org/10.1002/FOR.2585

Budiharto W (2021) Data science approach to stock prices forecasting in Indonesia during Covid-19 using long short-term memory (LSTM). J Big Data 2021(8–1):1–9. https://doi.org/10.1186/S40537-021-00430-0

Caporale GM, Howells PGA, Soliman AM (2004). Stock Market Development And Economic Growth: The Causal Linkage. Journal of Economic Development , 29 (1), 33–50. https://ideas.repec.org/a/jed/journl/v29y2004i1p33-50.html

Chong E, Han C, Park FC (2017) Deep learning networks for stock market analysis and prediction: methodology, data representations, and case studies. Expert Syst Appl 83:187–205. https://doi.org/10.1016/J.ESWA.2017.04.030

Chopra S, Yadav D, Chopra AN (2019) Artificial neural networks based Indian stock market price prediction: before and after demonetization. Int J Swarm Intell Evolut Comput 8(1):1–7

Cocianu CL, Grigoryan H (2015) An artificial neural network for data forecasting purposes. Informatica Economica 20(2):34–45. https://doi.org/10.12948/issn14531305/19.2.2015.04

Cooray A (n.d.). Cooray, & Arusha. (2010). Do Stock Markets Lead to Economic Growth? J Policy Model, 32(4):448–460. https://econpapers.repec.org/RePEc:eee:jpolmo:v:32:y::i:4:p:448-460

Cristianini N, Shawe-Taylor J (2000) An introduction to support vector machines and other kernel-based learning methods, doi https://doi.org/10.1017/CBO9780511801389

Damrongsakmethee T, Neagoe VE (2020). Stock market prediction using a deep learning approach. Proceedings of the 12th international conference on electronics

Das SP, Padhy S (2012) Support vector machines for prediction of futures prices in indian stock market. Int J Comp Appl 41(3):975–8887

de Oliveira FA, Nobre CN, Zárate LE (2013) Applying Artificial Neural Networks to prediction of stock price and improvement of the directional prediction index – Case study of PETR4, Petrobras. Brazil Exp Sys Appl 40(18):7596–7606. https://doi.org/10.1016/J.ESWA.2013.06.071

Dhenuvakonda P, Anandan R, Kumar N (2020) Stock price prediction using artificial neural networks. J Crit Rev 7(11):846–850. https://doi.org/10.31838/JCR.07.11.152

Di X (2014) Stock trend prediction with technical indicators using SVM. Stanford University. http://finance.yahoo.com

Dike HU, Zhou Y, Deveerasetty KK, Wu Q (2019) Unsupervised learning based on artificial neural network: a review. 2018 IEEE International conference on cyborg and bionic systems. CBS , 2018, 322–327. https://doi.org/10.1109/CBS.2018.8612259

Ding S, Zhu Z, Zhang X (2015) An overview on semi-supervised support vector machine. Neural Comput Appl 2015(28–5):969–978. https://doi.org/10.1007/S00521-015-2113-7

Du J, Liu Q, Chen K, Wang J (2019) Forecasting stock prices in two ways based on LSTM neural network. In: E. Networking, A. C. Conference (Eds.), Proceedings of 2019 IEEE 3rd Information Technology (pp. 1083–1086). ITNEC 2019

Elango NM, Sureshkumar KK (2012). Performance analysis of stock price prediction using artificial neural network. Glob J Comp Sci Tech http://computerresearch.org/index.php/computer/article/view/426

Enisan AA, Olufisayo AO, Enisan AA, Olufisayo AO (2009) Stock market development and economic growth: Evidence from seven sub-Sahara African countries. J Econ Bus, 61(2), 162–171. https://econpapers.repec.org/RePEc:eee:jebusi:v:61:y:2009:i:2:p:162-171

Farahani MS, Hajiagha SHR (2021) Forecasting stock price using integrated artificial neural network and metaheuristic algorithms compared to time series models. Soft Comput 25(13):8483–8513. https://doi.org/10.1007/S00500-021-05775-5

Fischer T, Krauss C (2017) Deep learning with long short-term memory networks for financial market predictions. https://ideas.repec.org/p/zbw/iwqwdp/112017.html

Gholami R, Fakhari N (2017) Support vector machine: principles, parameters, and applications. Handb Neur Comp. https://doi.org/10.1016/B978-0-12-811318-9.00027-2

Graves A (2012) Long short-term memory. 37–45. https://doi.org/10.1007/978-3-642-24797-2_4

Grigoryan H (2016) A stock market prediction method based on support vector machines (SVM) and independent component analysis (ICA). Database Syst J 7(1):12–21

Gupta A (2014) An SVM Based Approach for {I}ndian Benchmark Index Prediction. In: F. Economics & S. S. www.globalbizresearch.org (Eds.), Proceedings of the Third International Conference on Global Business

Guresen E, Kayakutlu G, Daim TU (2011) Using artificial neural network models in stock market index prediction. Exp Sys Appl: Int J 38(8):10389–10397. https://doi.org/10.1016/J.ESWA.2011.02.068

Gururaj V, Shriya V, Ashwini K (2019) Stock market prediction using linear regression and support vector machines. Int J Appl Eng Res, 14(8), 1931–1934. http://www.ripublication.com/ijaer19/ijaerv14n8_24.pdf

Haddad Z, Chaker A, Rahmani A (2017) Improving the basin type solar still performances using a vertical rotating wick. Desalination, Elsevier. https://www.sciencedirect.com/science/article/pii/S0011916416317702

Henrique BM, Sobreiro VA, Kimura H (2018) Stock price prediction using support vector regression on daily and up to the minute prices. J Finan Data Sci 4(3):183–201. https://doi.org/10.1016/J.JFDS.2018.04.003

Henrique BM, Sobreiro VA, Kimura H (2019) Literature review: machine learning techniques applied to financial market prediction. Expert Syst Appl 124:226–251. https://doi.org/10.1016/J.ESWA.2019.01.012

Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput, Ieeexplore.Ieee.Org , 9 (8), 1735–1780. https://ieeexplore.ieee.org/abstract/document/6795963/

Hossain MS, Rokonuzzaman M (2018) Impact of stock market. Trade and bank on economic growth for Latin American Countries: An econometrics approach, 6, 1. http://www.sciencepublishinggroup.com

Hou H, Cheng S-Y (2010) The roles of stock market in the finance-growth nexus: time series cointegration and causality evidence from Taiwan. Appl Financial Econ 20(12):975–981. https://doi.org/10.1080/09603101003724331

Joseph E (2019) Forecast on close stock market prediction using support vector machine (SVM). Int J Eng Res. https://doi.org/10.17577/ijertv8is020031

Kara Y, Boyacioglu MA, Baykan ÖK (2011) Predicting direction of stock price index movement using artificial neural networks and support vector machines: the sample of the Istanbul stock. Exchange 38(5):5311–5319. https://doi.org/10.1016/J.ESWA.2010.10.027

Kecman V (2001) Learning and soft computing. 2001. MIT Press/Bradford …. https://www.researchgate.net/publication/31727392_Learning_and_Soft_Computing_V_Kecman

Khan ZH, Alin TS, Hussain MA (2011) Price prediction of share market using artificial neural network (ANN). Int J Comp Appl 22(2):42–47

Lai CY, Chen RC, Caraka RE (2019) Prediction stock price based on different index factors using LSTM. Proceedings - International conference on machine learning and cybernetics

Lertyingyod W, Benjamas N (2017) Stock price trend prediction using artificial neural network techniques: case study: thailand stock exchange. 20th International computer science and engineering conference: smart ubiquitos computing and knowledge. ICSEC, 2016. https://doi.org/10.1109/ICSEC.2016.7859878

Levine R, Zervos S (1996) Stock market development and long-run growth on JSTOR. The World Bank Economic Review. https://www.jstor.org/stable/3990065

Levine R, Zervos S (1998) Stock markets, banks, and economic growth. American Economic Review, 88, 3. https://www.researchgate.net/publication/4901422_Stock_Markets_Banks_and_Economic_Growth

Li X, Li Y, Yang H, Yang L, Liu XY (2019). DP-LSTM: differential privacy-inspired LSTM for stock prediction using financial news. Arxiv.Org . https://arxiv.org/abs/1912.10806

Liagkouras K Metaxiotis K (2020) No title. Stock market forecasting by using support vector machines (p, 259–271. https://doi.org/10.1007/978-3-030-49724-8_11

Litta AJ, Idicula MS, Mohanty UC (2013) Artificial neural network model in prediction of meteorological parameters during premonsoon thunderstorms. Int J Atmosph Sci. https://doi.org/10.1155/2013/525383

Liu J, Kong X, Xia F, Bai X, Wang L, Qing Q, Lee I (2018) Artificial intelligence in the 21st century. IEEE Access 6:34403–34421. https://doi.org/10.1109/ACCESS.2018.2819688

Madge S, Bhatt S (2015) Predicting stock price direction using support vector machines. https://github.com/SaahilMadge/Spring-2015-IW

Marr D (1976) Artificial Intelligence -- A Personal View. MIT Libraries. https://dspace.mit.edu/handle/1721.1/5776

Marty AL (1961) Gurley and Shaw on Money in a Theory of Finance. J Polit Econ. https://www.jstor.org/stable/1829227

Masoud NMH (2013) The impact of stock market performance upon economic growth. Int J Econ Financ Issues 3(4):788–798

Meesad P, Rasel RI (2017) No Title. Predicting Stock Market Price Using Support Vector Regression, https://doi.org/10.1109/ICIEV.2013.6572570

Min F, Hu Q, Zhu W (2014) Feature selection with test cost constraint. Int J Appr Rea 55(1):167–179. https://doi.org/10.1016/J.IJAR.2013.04.003

Moghaddam AH, Moghaddam MH, Esfandyari M (2016) Stock market index prediction using artificial neural network. J Econ, Finance Administ Sci 21(41):89–93. https://doi.org/10.1016/J.JEFAS.2016.07.002

Moghar A, Hamiche M (2020) Stock market prediction using LSTM recurrent neural network. Elsevier. https://www.sciencedirect.com/science/article/pii/S1877050920304865

Mubeena SK, Kumar MA, Ramya U, Sujatha P, Tech, B. (2020). Forecasting stock market movement direction using sentiment analysis and support vector machine. Int Res J Eng Tech. www.irjet.net

Naik N, Mohan BR (2019) Stock price movements classification using machine and deep learning techniques-the case study of indian stock market. Commun Comp Inf Sci 1000:445–452. https://doi.org/10.1007/978-3-030-20257-6_38

Nandakumar R, Uttamraj KR, Vishal R, Lokeswari YV (2018) Stock price prediction using long short term memory. Int Res J Eng Technol 5(3):342–3348

Nti IK, Adekoya AF, Weyori BA (2020a) Efficient stock-market prediction using ensemble support vector machine. Open Comp Sci 10(1):153–163. https://doi.org/10.1515/COMP-2020-0199

O’Leary DE (2013) Artificial intelligence and big data. IEEE Intell Syst 28(2):96–99. https://doi.org/10.1109/MIS.2013.39

Pan J, Zhuang Y, Fong S (2016) The impact of data normalization on stock market prediction: using SVM and technical indicators. Commun Comp Inf Sci 652:72–88. https://doi.org/10.1007/978-981-10-2777-2_7

Pang X, Zhou Y, Wang P, Lin W, Chang V (2018) An innovative neural network approach for stock market prediction. J Supercomput 2018(76–3):2098–2118. https://doi.org/10.1007/S11227-017-2228-Y

Parmar I, Agarwal N, Saxena S, Arora R, Gupta S, Dhiman H, & Chouhan L (2018a) Stock market prediction using machine learning. ICSCCCst international conference on secure cyber computing and communications, 2011–2018a

Patil SS, Patidar K, Jain M (2016) Stock market trend prediction using support vector machine. Int J Curr Trends Eng Technol, 2(1), 18–25. http://casopisi.junis.ni.ac.rs/index.php/FUAutContRob/article/view/585

Pedrozo D, Barajas F, Estupiñán A, Cristiano KL, Triana DA (2020) Development and implementation of a predictive method for the stock market analysis, using the long short-term memory machine learning method. J Phys: Conf Ser 1514(1):012009. https://doi.org/10.1088/1742-6596/1514/1/012009

Perwej Y, Perwej A, Perwej Y, Perwej A (2012) Prediction of the bombay stock exchange (BSE) market returns using artificial neural network and genetic algorithm. J Intell Learn Syst Appl 4(2):108–119. https://doi.org/10.4236/JILSA.2012.42010

Pradhan A, Model, S. (2012). Support Vector Machine-A Survey. In undefined

Pradhan RP, Arvin MB, Samadhan B, Taneja S (2013) The impact of stock market development on inflation and economic growth of 16 asian countries: a panel VAR Approach. Appl Econom Int Devel, 13(1), 203–218. https://ideas.repec.org/a/eaa/aeinde/v13y2013i1_16.html

Qiu M, Song Y (2016) Predicting the direction of stock market index movement using an optimized artificial neural network model. PLoS ONE 11:5. https://doi.org/10.1371/JOURNAL.PONE.0155133

Qiu J, Wang B, Zhou C (2020) Forecasting stock prices with long-short term memory neural network based on attention mechanism. PLoS ONE 15:1. https://doi.org/10.1371/JOURNAL.PONE.0227222

Rahul, Subrat S, Priyansh, K Monika 2020 Analysis of various approaches for stock market prediction. J Stat Manag Syst, 23(2):285–293, https://doi.org/10.1080/09720510.2020.1724627

Reddy VKS (2018) Stock market prediction using machine learning. Int Res J Eng Technol (IRJET). https://doi.org/10.13140/RG.2.2.12300.77448

Reddy Nadikattu R (2017) The supremacy of artificial intelligence and neural networks. Int J Creat Res Thoughts 5(1):2320–2882

Roondiwala M, Patel H (2017) Predicting stock prices using LSTM. Int J Sci Research (IJSR). https://doi.org/10.21275/ART20172755

Rosillo R, Giner J, la Fuente DD (2014) Stock Market simulation using support vector machines. J Forecast 33(6):488–500. https://doi.org/10.1002/FOR.2302

Samek D, Vařacha P (2013) Time series prediction using artificial neural networks: single and multi-dimensional data Request PDF. Int J Math Model Meth Appl Sci, 7(1):38–46. https://www.researchgate.net/publication/288530573_Time_series_prediction_using_artificial_neural_networks_Single_and_multi-dimensional_data

Saud AS, Shakya S (2020) Analysis of look back period for stock price prediction with RNN variants: a case study on banking sector of NEPSE. Procedia Comp Sci 167:788–798. https://doi.org/10.1016/J.PROCS.2020.03.419

Sch"olkopf B, Smola AJ (2002) Support vector machines and kernel algorithms. In: The handbook of brain theory and neural networks, pp 1119–1125

Schapire RE (2003) The boosting approach to machine learning: an overview. Springer, Berlin, pp 149–171. https://doi.org/10.1007/978-0-387-21579-2_9

Book   MATH   Google Scholar  

Seetanah B, Subadar U, Sannassee RV, Lamport M, Ajageer V (2012) Stock market development and economic growth: Evidence from least developed countries. Competence centre on Money. https://ideas.repec.org/p/mtf/wpaper/1205.html

Selvamuthu D, Kumar V, Mishra A (2019) {I}ndian stock market prediction using artificial neural networks on tick data. Financial Innov 2019(5–1):1–12. https://doi.org/10.1186/S40854-019-0131-7

Selvin S, Vinayakumar R, Gopalakrishnan EA, Menon VK, Soman KP (2017) Stock price prediction using LSTM. RNN and CNN-sliding window model, 1643–1647

Shanmuganathan S (2016) Artificial neural network modelling: an introduction. Stud Computat Intell 628:1–14. https://doi.org/10.1007/978-3-319-28495-8_1

Sharma V, Rai S, Dev A (2012) A comprehensive study of artificial neural networks. Int J Adv Res Comp Sci Softw Eng 2:10

Sidhu P, Aggarwal H, Lal M (2021) stock market prediction using LSTM. https://doi.org/10.4108/EAI.27-2-2020.2303545

Simon S, Raoot A, Professor A (2012) Accuracy driven artificial neural networks in stock market prediction. Int J Soft Comp (IJSC) 3:2. https://doi.org/10.5121/ijsc.2012.3203

Smagulova K, James AP (2020) Overview of long short-term memory neural networks. Model Optimiz Sci Technol 14:139–153. https://doi.org/10.1007/978-3-030-14524-8_11

Stiglitz JE (1985) Credit markets and the control of capital. J Money, Credit Bank, 17(2):133–152. https://econpapers.repec.org/RePEc:mcb:jmoncb:v:17:y:1985:i:2:p:133-52

Tripathy N (2019) Stock price prediction using support vector machine approach. https://doi.org/10.33422/conferenceme.2019.11.641

Vaiz J, Ramaswami M (2016) a hybrid model to forecast stock trend using support vector machine and neural networks. Int J Eng Res Develop (IJERD). https://www.academia.edu/download/54665553/H130925259.pdf

van Houdt G, Mosquera C, Nápoles G (2020) A review on the long short-term memory model. Artif Intell Rev 2020(53–8):5929–5955. https://doi.org/10.1007/S10462-020-09838-1

Vapnik V (1998) The support vector method of function estimation. Nonlin Model. Springer, Boston, pp 55–85. https://doi.org/10.1007/978-1-4615-5703-6_3

Chapter   Google Scholar  

Wanjawa BW (2016). Evaluating the performance of ANN prediction system at Shanghai Stock market in the period, 21. https://www.researchgate.net/publication/311514572_Evaluating_the_Performance_of_ANN_Prediction_System_at_Shanghai_Stock_Market_in_the_Period_21-Sep-2016_to_11-Oct-2016

Yang R, Yu L, Zhao Y, Yu H, Xu G, Wu Y, Liu Z (2020) Big data analytics for financial Market volatility forecast based on support vector machine. Int J Inf Manage 50:452–462. https://doi.org/10.1016/J.IJINFOMGT.2019.05.027

Zeng Y, Liu X (2018) A-stock price fluctuation forecast model based on LSTM. Proceedings - 2018 14th international conference on semantics, 261–264

Zhang L, Pan Y, Wu X, Skibniewski MJ (2021) Introduction to artificial intelligence. In Lecture notes in civil engineering. Vol. 163, Cham, pp. 1–15

Zou Z, Qu Z (2020) Using LSTM in stock prediction and quantitative trading. Deep Learning

Download references

Acknowledgements

The authors are grateful to Delhi Public School and Department of Chemical Engineering, School of Energy Technology, Pandit Deendayal Energy University for the permission to publish this research.

Not Applicable.

Author information

Authors and affiliations.

Delhi Public School, Bopal, Ahmedabad, Gujarat, India

Dhruhi Sheth

Department of Chemical Engineering, School of Energy Technology, Pandit Deendayal Energy University, Gandhinagar, Gujarat, 382426, India

You can also search for this author in PubMed   Google Scholar

Contributions

All the authors make a substantial contribution to this manuscript. DS and MS participated in drafting the manuscript. DS and MS wrote the main manuscript. All the authors discussed the results and implication on the manuscript at all stages.

Corresponding author

Correspondence to Manan Shah .

Ethics declarations

Conflict of interest.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Sheth, D., Shah, M. Predicting stock market using machine learning: best and accurate way to know future stock prices. Int J Syst Assur Eng Manag 14 , 1–18 (2023). https://doi.org/10.1007/s13198-022-01811-1

Download citation

Received : 18 October 2021

Revised : 15 August 2022

Accepted : 24 November 2022

Published : 09 January 2023

Issue Date : February 2023

DOI : https://doi.org/10.1007/s13198-022-01811-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Stock market
• Machine learning (ML)
• Find a journal
• Publish with us
• Track your research

Help | Advanced Search

Quantitative Finance > Statistical Finance

Title: stock price prediction using sentiment analysis and deep learning for indian markets.

Abstract: Stock market prediction has been an active area of research for a considerable period. Arrival of computing, followed by Machine Learning has upgraded the speed of research as well as opened new avenues. As part of this research study, we aimed to predict the future stock movement of shares using the historical prices aided with availability of sentiment data. Two models were used as part of the exercise, LSTM was the first model with historical prices as the independent variable. Sentiment Analysis captured using Intensity Analyzer was used as the major parameter for Random Forest Model used for the second part, some macro parameters like Gold, Oil prices, USD exchange rate and Indian Govt. Securities yields were also added to the model for improved accuracy of the model. As the end product, prices of 4 stocks viz. Reliance, HDFC Bank, TCS and SBI were predicted using the aforementioned two models. The results were evaluated using RMSE metric.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Stock Market Prediction Using Machine Learning

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Subscribe to the PwC Newsletter

Join the community, add a new evaluation result row, stock market prediction.

41 papers with code • 3 benchmarks • 4 datasets

Benchmarks Add a Result

Most implemented papers, bert: pre-training of deep bidirectional transformers for language understanding.

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers.

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging.

SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis

In particular, the prediction of aspect-sentiment pairs is converted into multi-label classification, aiming to capture the dependency between words in a pair.

Sentiment Analysis of Twitter Data for Predicting Stock Market Movements

In this paper, we have applied sentiment analysis and supervised machine learning principles to the tweets extracted from twitter and analyze the correlation between stock market movements of a company and sentiments in tweets.

Revisiting Pre-Trained Models for Chinese Natural Language Processing

Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and consecutive variants have been proposed to further improve the performance of the pre-trained language models.

FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance

In this paper, we introduce a DRL library FinRL that facilitates beginners to expose themselves to quantitative finance and to develop their own stock trading strategies.

Listening to Chaotic Whispers: A Deep Learning Framework for News-oriented Stock Trend Prediction

gkeng/Listening-to-Chaotic-Whishpers--Code • 6 Dec 2017

Stock trend prediction plays a critical role in seeking maximized profit from stock investment.

Twitter mood predicts the stock market

peanutshawny/lstm-stock-predictor • 14 Oct 2010

A Granger causality analysis and a Self-Organizing Fuzzy Neural Network are then used to investigate the hypothesis that public mood states, as measured by the OpinionFinder and GPOMS mood time series, are predictive of changes in DJIA closing values.

Stock Price Correlation Coefficient Prediction with ARIMA-LSTM Hybrid Model

imhgchoi/Corr_Prediction_ARIMA_LSTM_Hybrid • 5 Aug 2018

Predicting the price correlation of two assets for future time periods is important in portfolio optimization.

Temporal Relational Ranking for Stock Prediction

Our RSR method advances existing solutions in two major aspects: 1) tailoring the deep learning models for stock ranking, and 2) capturing the stock relations in a time-sensitive manner.

Wall Street Debut of Trump's Truth Social Network Could Net Him Stock Worth Billions on Paper

The Wall Street debut of Donald Trump’s Truth Social network could give him stock worth billions of dollars on paper

Wall Street Debut of Trump's Truth Social Network Could Net Him Stock Worth Billions on Paper

John Minchillo

The Truth Social account for former President Donald Trump is seen on a mobile device, Wednesday, March 20, 2024, in New York. Trump’s Truth Social looks set to hit Wall Street in a move that could give him stock worth billions of dollars on paper. But the former president likely can’t cash it out right away, unless some things change. (AP Photo/John Minchillo)

NEW YORK (AP) — The Wall Street debut of Donald Trump’s Truth Social network could give him stock worth billions of dollars on paper. But the former president probably will not be able to cash it out right away, unless some things change.

The longer-term outlook for the business is highly questionable. Trump's company has said it expects to continue losing money for a while, and at least one expert says it's likely worth far less than the stock market suggests.

Trump's pending return to Wall Street comes down to a vote scheduled for Friday by shareholders of a company named Digital World Acquisition Corp., which at the moment is essentially just a pile of cash. The corporation hopes to merge with Trump Media & Technology Group, the company behind Truth Social that goes by TMTG. If the shareholders approve the deal, TMTG could soon see its stock trading on the Nasdaq in Digital World’s place.

Here’s a look at the proposal and Trump's role in it.

WHAT HAPPENS FRIDAY?

Shareholders of Digital World are scheduled to vote on whether to approve a merger with TMTG, where Trump is the chairman. Digital World is what’s called a special purpose acquisition company, or SPAC, or “blank-check company.”

Photos You Should See

SPACs raise cash and then hunt for companies to merge with. Such deals give the target companies a potentially quicker and easier way to get their stocks onto the New York Stock Exchange or Nasdaq. The arrangement lets them avoid some of the paperwork associated with traditional initial public offerings of stock, or IPOs.

For investors, SPACs offer a way to get into hyped, potentially faster-growing companies such as TMTG, the DraftKings betting service or SoFi banking.

DO SHAREHOLDERS EVER SAY NO?

It happens, but only rarely. This vote looks likely to pass given how high Digital World’s stock has jumped on excitement about Trump. It finished Thursday at $42.81 per share. It's already up nearly 145% so far this year, towering over the roughly 10% gain for the S&P 500 index.

Many of Digital World’s investors are small-time investors who are either fans of Trump or trying to cash in on the mania, instead of big institutional and professional investors.

WHAT HAPPENS IF THE SHAREHOLDERS APPROVE?

Digital World will merge with TMTG. The stock will continue to trade under Digital World’s ticker, DWAC, possibly for a couple of days to a couple of weeks, experts say. Then at some point, companies in SPAC deals usually announce that their stock will begin trading under the new ticker symbol.

Trump's company hopes to trade under the ticker symbol DJT, the former president’s initials. The same ticker symbol was used by Trump Hotels & Casino Resorts before it filed for Chapter 11 bankruptcy protection in 2004.

HOW MUCH WILL TRUMP GET?

Trump will own most of the new, combined company, or nearly 78.8 million shares, which would account for at least 58%. Multiply that by Digital World’s current stock price of more than $40, and the total value could surpass $3 billion.

TRUMP NEEDS CASH, RIGHT? CAN HE SELL RIGHT AWAY?

Trump faces a $454 million judgment in a fraud lawsuit, among other financial burdens. But he cannot sell easily for at least six months. That’s because major TMTG shareholders will be under what’s called a “lock-up” provision, a common restriction on Wall Street that keeps big, early investors from immediately dumping their shares. Such sales could tank the stock’s price.

Investors under the lock-up deal cannot sell, lend, donate or encumber their shares for six months after the close of the deal. Legal experts say “encumber” is a powerful word that could prevent Trump from using the stock as collateral to raise cash before six months have elapsed.

There are a few exceptions, such as by transferring stock to immediate family members. But in such cases, the recipients would also have to agree to abide by the lock-up agreement.

SO DEFINITELY NO CASH RIGHT AWAY?

Digital World could waive the lock-up agreement before the deal closes. Or, in what some legal experts say could be a more likely path, the new company’s board could decide to alter the lock-up agreement after the deal closes.

Such a decision by the board could open those directors up to legal scrutiny. They would need to show they’re doing it to benefit shareholders.

But if the value of Trump’s brand is key to the company’s success, and if easing the lock-up agreements could preserve that brand, it could make for a case that would at least spare board members' lawyers from getting laughed out of court immediately.

Some companies' boards in the past have altered lock-up agreements to allow investors to sell earlier.

WHO WILL BE ON THIS COMPANY'S BOARD?

Mostly people put forth by TMTG, including the former president’s son, Donald Trump Jr., if all goes as expected. Former Republican Rep. Devin Nunes would be a director and the company’s CEO.

Also on the board would be Robert Lighthizer, who served as Trump’s U.S. trade representative, and Linda McMahon, who ran the Small Business Administration under Trump.

IS THIS A SAFE INVESTMENT?

Every stock has risks. Digital World has filed 84 pages with U.S. regulators to list many of its risks and those of TMTG.

One risk, the company said, was that as a controlling stockholder, Trump would be entitled to vote his shares in his own interest, which may not always be in the interests of all the shareholders generally.

It also cited the high rate of failure for new social media platforms, as well as TMTG’s expectation that the company will lose money on its operations “for the foreseeable future.” The company lost $49 million in the first nine months of last year, when it brought in just $3.4 million in revenue and had to pay $37.7 million in interest expenses.

“It's losing money, there's no way the company is worth anything like" what the stock price suggests, said Jay Ritter, an IPO specialist at the University of Florida’s Warrington College of Business.

“Here, given the stock price is so divorced from fundamental value, it’s kind of the same issue that came up with meme stocks,” he said, recalling companies whose share prices once soared far beyond what professionals considered rational. “With AMC and GameStop, the price was way above fundamental value, and there's the question of: Can you get out before the music stops?”

Copyright 2024 The  Associated Press . All rights reserved. This material may not be published, broadcast, rewritten or redistributed.

Tags: Associated Press , business , politics

America 2024

Subscribe to our daily newsletter to get investing advice, rankings and stock market news.

See a newsletter example .

You May Also Like

The 10 worst presidents.

U.S. News Staff Feb. 23, 2024

Cartoons on President Donald Trump

Feb. 1, 2017, at 1:24 p.m.

Photos: Obama Behind the Scenes

April 8, 2022

Photos: Who Supports Joe Biden?

March 11, 2020

The Baltimore Bridge Collapse, Explained

Elliott Davis Jr. March 27, 2024

In Ala., More GOP Trouble on Abortion

Susan Milligan March 27, 2024

Robert F. Kennedy Jr. Names VP

Susan Milligan March 26, 2024

3 SCOTUS Abortion Pill Takeaways

Cecelia Smith-Schoenwalder March 26, 2024

Consumers Sour About Economy’s Future

Tim Smart March 26, 2024

The Week in Cartoons March 25-29

March 26, 2024, at 10:49 a.m.

• Election 2024
• Entertainment
• Newsletters
• Photography
• Personal Finance
• AP Buyline Personal Finance
• Press Releases
• Israel-Hamas War
• Russia-Ukraine War
• Global elections
• Asia Pacific
• Latin America
• Middle East
• March Madness
• AP Top 25 Poll
• Movie reviews
• Book reviews
• Personal finance
• Financial Markets
• Business Highlights
• Financial wellness
• Artificial Intelligence
• Social Media

Wall Street debut of Trump’s Truth Social network could net him stock worth billions on paper

The Truth Social account for former President Donald Trump is seen on a mobile device, Wednesday, March 20, 2024, in New York. Trump’s Truth Social looks set to hit Wall Street in a move that could give him stock worth billions of dollars on paper. But the former president likely can’t cash it out right away, unless some things change. (AP Photo/John Minchillo)

• Copy Link copied

NEW YORK (AP) — The Wall Street debut of Donald Trump’s Truth Social network could give him stock worth billions of dollars on paper. But the former president probably will not be able to cash it out right away, unless some things change.

The longer-term outlook for the business is highly questionable. Trump’s company has said it expects to continue losing money for a while, and at least one expert says it’s likely worth far less than the stock market suggests.

Trump’s pending return to Wall Street comes down to a vote scheduled for Friday by shareholders of a company named Digital World Acquisition Corp., which at the moment is essentially just a pile of cash. The corporation hopes to merge with Trump Media & Technology Group, the company behind Truth Social that goes by TMTG. If the shareholders approve the deal, TMTG could soon see its stock trading on the Nasdaq in Digital World’s place.

Here’s a look at the proposal and Trump’s role in it.

WHAT HAPPENS FRIDAY?

Shareholders of Digital World are scheduled to vote on whether to approve a merger with TMTG, where Trump is the chairman. Digital World is what’s called a special purpose acquisition company, or SPAC, or “blank-check company.”

SPACs raise cash and then hunt for companies to merge with. Such deals give the target companies a potentially quicker and easier way to get their stocks onto the New York Stock Exchange or Nasdaq. The arrangement lets them avoid some of the paperwork associated with traditional initial public offerings of stock, or IPOs.

For investors, SPACs offer a way to get into hyped, potentially faster-growing companies such as TMTG, the DraftKings betting service or SoFi banking.

DO SHAREHOLDERS EVER SAY NO?

It happens, but only rarely. This vote looks likely to pass given how high Digital World’s stock has jumped on excitement about Trump. It finished Thursday at $42.81 per share. It’s already up nearly 145% so far this year, towering over the roughly 10% gain for the S&P 500 index.

Many of Digital World’s investors are small-time investors who are either fans of Trump or trying to cash in on the mania, instead of big institutional and professional investors.

WHAT HAPPENS IF THE SHAREHOLDERS APPROVE?

Digital World will merge with TMTG. The stock will continue to trade under Digital World’s ticker, DWAC, possibly for a couple of days to a couple of weeks, experts say. Then at some point, companies in SPAC deals usually announce that their stock will begin trading under the new ticker symbol.

Trump’s company hopes to trade under the ticker symbol DJT, the former president’s initials. The same ticker symbol was used by Trump Hotels & Casino Resorts before it filed for Chapter 11 bankruptcy protection in 2004.

HOW MUCH WILL TRUMP GET?

Trump will own most of the new, combined company, or nearly 78.8 million shares, which would account for at least 58%. Multiply that by Digital World’s current stock price of more than $40, and the total value could surpass $3 billion.

TRUMP NEEDS CASH, RIGHT? CAN HE SELL RIGHT AWAY?

Trump faces a $454 million judgment in a fraud lawsuit, among other financial burdens. But he cannot sell easily for at least six months. That’s because major TMTG shareholders will be under what’s called a “lock-up” provision, a common restriction on Wall Street that keeps big, early investors from immediately dumping their shares. Such sales could tank the stock’s price.

Investors under the lock-up deal cannot sell, lend, donate or encumber their shares for six months after the close of the deal. Legal experts say “encumber” is a powerful word that could prevent Trump from using the stock as collateral to raise cash before six months have elapsed.

There are a few exceptions, such as by transferring stock to immediate family members. But in such cases, the recipients would also have to agree to abide by the lock-up agreement.

SO DEFINITELY NO CASH RIGHT AWAY?

Digital World could waive the lock-up agreement before the deal closes. Or, in what some legal experts say could be a more likely path, the new company’s board could decide to alter the lock-up agreement after the deal closes.

Such a decision by the board could open those directors up to legal scrutiny. They would need to show they’re doing it to benefit shareholders.

But if the value of Trump’s brand is key to the company’s success, and if easing the lock-up agreements could preserve that brand, it could make for a case that would at least spare board members’ lawyers from getting laughed out of court immediately.

Some companies’ boards in the past have altered lock-up agreements to allow investors to sell earlier.

WHO WILL BE ON THIS COMPANY’S BOARD?

Mostly people put forth by TMTG, including the former president’s son, Donald Trump Jr., if all goes as expected. Former Republican Rep. Devin Nunes would be a director and the company’s CEO.

Also on the board would be Robert Lighthizer, who served as Trump’s U.S. trade representative, and Linda McMahon, who ran the Small Business Administration under Trump.

IS THIS A SAFE INVESTMENT?

Every stock has risks. Digital World has filed 84 pages with U.S. regulators to list many of its risks and those of TMTG.

One risk, the company said, was that as a controlling stockholder, Trump would be entitled to vote his shares in his own interest, which may not always be in the interests of all the shareholders generally.

It also cited the high rate of failure for new social media platforms, as well as TMTG’s expectation that the company will lose money on its operations “for the foreseeable future.” The company lost $49 million in the first nine months of last year, when it brought in just $3.4 million in revenue and had to pay $37.7 million in interest expenses.

“It’s losing money, there’s no way the company is worth anything like” what the stock price suggests, said Jay Ritter, an IPO specialist at the University of Florida’s Warrington College of Business.

“Here, given the stock price is so divorced from fundamental value, it’s kind of the same issue that came up with meme stocks,” he said, recalling companies whose share prices once soared far beyond what professionals considered rational. “With AMC and GameStop, the price was way above fundamental value, and there’s the question of: Can you get out before the music stops?”

Watch CBS News

Trump's Truth Social platform soars in second day of trading on Nasdaq

By Aimee Picchi

Edited By Anne Marie Lee , Alain Sherter

Updated on: March 27, 2024 / 11:51 AM EDT / CBS News

Former President Donald Trump's Truth Social began trading under the ticker "DJT" on Tuesday, putting the real estate tycoon — and his initials — at the helm of a publicly traded company once again. 

Trump Media & Technology Group shares soared in their second day of trading, rising $7.41, or 13%, to $65.40 on Wednesday morning. That follows a gain of 16% on Monday, when the company began trading on Nasdaq.

The gains give Trump Media & Technology Group a market value of $9.3 billion. Trump, who owns 58% of the newly public company, now has a stake valued at $5.2 billion — at least on paper.

The company, whose main asset is the social media service Truth Social, has captured the attention of both critics and supporters, with some investors buying stock to express their support for the former president. Others are retail investors who want to cash in on the mania, rather than big institutional and professional investors. 

"DJT has all the makings of a meme stock, given the Trump news factor," noted Ben Emons, senior portfolio manager and head of fixed income at NewEdge Wealth, in a Tuesday research note. "For global macro investors, DJT will be a proxy for how markets price Trump 2.0 policies."

Trump Media is now the most expensive stock to short in the U.S., according to Bloomberg News. Short-selling, which involves betting that a specific stock will decline, is pricey for Trump's company because there are few shares available to borrow and there's high interest in betting against the company, the report noted.

On Truth Social Tuesday, users were posting about being shareholders or seeking tips on how to buy shares. One user urged conservatives to "get behind the DJT stock and sent it over $100 per share" to "drive the liberals insane!" Another declared: "Get yourself a piece of #DJT stock if your a true MAGA supporter."

On Monday, Trump told reporters that "Truth Social is doing very well. It's hot as a pistol and doing great." On Tuesday, he posted "I LOVE TRUTH SOCIAL, I LOVE THE TRUTH!," on the platform. A day earlier, Trump Media CEO Devin Nunes, a former Republican congressman, said that going public will allow the company "to build a movement to reclaim the Internet from Big Tech censors."

Despite the enthusiasm, investors could experience a bumpy ride. For one, they're betting on a company with uncertain financial prospects. Trump Media lost $49 million in the first nine months of last year, when it brought in just $3.4 million in revenue and had to pay $37.7 million in interest expenses.

DJT: an "on brand" ticker

Trump Media & Technology Group said in a statement Monday  that the ticker symbol would be active on Tuesday following its merger with a so-called blank-check company , also known as a special purchase acquisition company (SPAC). SPACs are shell companies created to take a private business public without going through an initial public offering.

In the case of Trump's media business, the shareholders of the SPAC, called Digital World Acquisition Corp.,  voted Friday  in favor of the merger, ushering in the next step of taking the new Truth Social company public without an IPO. The merged company officially changed its name to Trump Media & Technology Group after the deal was completed on Monday, the statement said. 

The eponymous symbol "is so on brand" for Trump, noted Kristi Marvin, chief executive of SPACInsider.com, a service that provides news and data about the SPAC industry.

Ahead of the debut of the new DJT ticker, shares of Digital World Acquisition Corp. soared $13.01, or 35%, to $49.95 on Monday.

Trump: Will he sell DJT shares?

Trump Media & Technology Group's multibillion valuation provides Trump with access to liquidity at a time when he's increasingly under financial pressure from a string of lawsuits. On Monday, he got a major break when an appeals court reduced a $464 million civil fraud judgment to $175 million, yet he still faces mounting legal bills related to other cases.

Trump could sell some of DJT stock to help pay for his legal bills, although the company currently has a "lock up" period, effectively barring its executives from selling shares for six months.

However, the company's board — comprised of Trump associates such as Kash Patel, an official during the Trump administration; and son, Donald Trump Jr. — could waive or shorten the lock-up period, experts said.

But there's a risk if Trump sells his stock, Emons noted. Because he owns a sizable chunk of the company, selling his shares could undermine its trading stability. For instance, "If he goes ahead [with selling], it could sink DJT by at least 15% to 40% based on option pricing," Emons calculated.

Truth Social: Losing money

To be sure, plenty of tech companies have gone public while in the red, yet typically investors want to see that a business can grow its user base and ramp up sales quickly by appealing to a broad range of advertisers. 

Truth Social, which doesn't release its user numbers, had roughly 5 million active members in February, according to research firm Similarweb estimates. 

By comparison, Reddit, which went public last week , had about 73.1 million daily active users last year, while revenue jumped 21% to $804 million in 2023, it reported last month ahead of the IPO filing.

Previous DJT ticker: From IPO to penny stock

It's also not the first time that Trump has overseen a publicly traded company with the ticker DJT. 

The previous iteration of the DJT ticker occurred in 1995, when Trump took his Trump Hotels & Casino Resorts public in an IPO. The idea was to raise money in the public markets that would help Trump expand his casino businesses, according to the New York Times' account of the IPO. 

The shares initially performed well, increasing from its IPO price of $14 to a high of $35 a share soon after. But the stock plunged over the next few years, eventually trading for pennies, according to the Washington Post. 

Trump Hotels & Casino Resorts filed for Chapter 11 bankruptcy in 2004. 

—With reporting by the Associated Press.

• Wall Street
• Donald Trump
• Truth Social

Aimee Picchi is the associate managing editor for CBS MoneyWatch, where she covers business and personal finance. She previously worked at Bloomberg News and has written for national news outlets including USA Today and Consumer Reports.

More from CBS News

Shoun Thao appointed to Sacramento City Council interim District 2 seat left vacant after Loloee resignation

Sacramento airport janitors, some with disabilities, could soon be out of work

Caldor Fire victims collect claims to sue Forest Service for Grizzly Flats destruction

Who is Nicole Shanahan, RFK Jr.'s new running mate?