Expert Stock Picker: The Wisdom of (Experts in) Crowds

Expert Stock Picker: The Wisdom of (Experts in) Crowds

Shawndra Hill and Noah Ready-Campbell

ABSTRACT: The phrase "the wisdom of crowds" suggests that good verdicts can be achieved by averaging the opinions and insights of large, diverse groups of people who possess varied types of information. Online user-generated content enables researchers to view the opinions of large numbers of users publicly. These opinions, in the form of reviews and votes, can be used to automatically generate remarkably accurate verdicts--collective estimations of future performance--about companies, products, and people on the Web to resolve very tough problems. The wealth and richness of user-generated content may enable firms and individuals to aggregate consumer-think for better business understanding. Our main contribution, here applied to user-generated stock pick votes from a widely used online financial newsletter, is a genetic algorithm approach that can be used to identify the appropriate vote weights for users based on their prior individual voting success. Our method allows us to identify and rank "experts" within the crowd, enabling better stock pick decisions than the S&P 500. We show that the online crowd performs better, on average, than the S&P 500 for two test time periods, 2008 and 2009, in terms of both overall returns and risk-adjusted returns, as measured by the Sharpe ratio. Furthermore, we show that giving more weight to the votes of the experts in the crowds increases the accuracy of the verdicts, yielding an even greater return in the same time periods. We test our approach by utilizing more than three years of publicly available stock pick data. We compare our method to approaches derived from both the computer science and finance literature. We believe that our approach can be generalized to other domains where user opinions are publicly available early and where those opinions can be evaluated. For example, YouTube video ratings may be used to predict downloads, or online reviewer ratings on Digg may be used to predict the success or popularity of a story.

KEY WORDS AND PHRASES: data mining, prediction markets, social media, usergenerated content, wisdom of crowds.

In this paper we show that user-generated content (UGC) is an acceptable theater in which crowd wisdom can be used to identify good verdicts--in this case, accurate stock picks. Furthermore, we show that when we identify, or at least reveal, experts and weight their votes accordingly, we perform more accurately than when we use everyone in the crowd to vote for stocks. Our contribution is that we provide a method based on a genetic algorithm (GA) to learn the appropriate contributions of independent users through the use of observed past individual performance. We compare and evaluate our approach in the context of criteria used in past research to generate stock portfolios.

The authors thank Anthony Crawford for research assistance, Theodoros Evgeniou, Steve Kimbrough, and Vasant Dhar for valuable comments, and Motley Fool CAPS for allowing us to use their valuable data.

International Journal of Electronic Commerce / Spring 2011, Vol. 15, No. 3, pp. 73?102. Copyright ? 2011 M.E. Sharpe, Inc. All rights reserved. 1086-4415/2011 $9.50 + 0.00. DOI 10.2753/JEC1086-4415150304

04 hill.indd 73

2/21/2011 11:23:14 PM

74 Hill and Ready-Campbell

In prior research, stock market analysts that worked in financial firms were evaluated to find "start analysts" [7]--the top performing analysts, to make stock market predictions. In addition, people that make stock market picks, as a leisure activity, online [11], have been evaluated in aggregate to find good stock portfolios. Gu et al. [11] weigh the different posts by the author's credibility based on the accuracy of the author's past post and most credible authors are considered experts. Likewise, we use a mixture of experts approach [11] informed by the method of Jordan and Jacobs [14] In our work, we use historical individual level stock pick data from online users to identify users that have been successful at the task of picking stocks in the past--we call these successful users "experts." The ability to identify experts as part of the crowd enables us to take better advantage of the "wisdom of crowds" [22] by restricting the crowd to a set of experts.

The purpose of this research is to implement a stock-trading strategy using the publicly available Motley Fool CAPS data (). If the trading strategy proves to be modestly successful, it could be of broad interest to investment managers looking for alternative investment strategies, at least in the short term. In addition, the improved scoring system could be of considerable interest to the CAPS team and other firms making stock voting data available. Most important, however, showing that the wisdom of crowds is effective for decision making may have implications for how firms and social systems should be organized around group voting for tough decision making.

There were approximately 116 million consumers of UGC and 82.5 million content creators in February 2009, according to market research and analysis firm eMarketer [24]. The bottom line: groups and crowds are contributing their opinions online in public venues at a spectacular rate. In this research we take advantage of publicly available UGC for decision making--specifically, stock picks.

There are many sources of online UGC submitted by millions of creators; for example, social networking, blogs, online reviews, question-answer, pictures, video, and wikis. In addition, votes and aggregate opinion are available from voting and information/prediction market sites, which are most often used to predict financial, election, and sports outcomes. UGC has been used in aggregate to predict recommendation system ratings [15], music sales [6], and blockbuster performance [8]. User-generated text has also been used to predict stock market performance [2, 18, 25].

The types of UGC sites we are interested in are the many online prediction and voting markets, such as BetFair, NewsFutures, Hollywood Stock Exchange, and Popular Science Predictions Exchange. As their names suggest, these sites enable users to bet on and make predictions about the outcomes of future events. Some use virtual money, and others use real money on their exchanges, with varying missions from profit to philanthropy.

Most relevant to our research are the major players in the stock voting game: Piqqem, Cake Financial, Covestor, Predictify, and the Motley Fool CAPS. These sites fall into two categories--quasi-prediction markets (where "quasi" means data are aggregated from disparate sources on the Web, but explicit voting did not necessarily occur) and prediction markets (where explicit voting did

04 hill.indd 74

2/21/2011 11:23:14 PM

International journal of electronic commerce 75

occur). In both cases, the sites offer solutions to help aid in financial decision making. To our knowledge no one has applied the wisdom of (expert) crowds theory to a large-scale stock pick data set.

The challenge in successfully picking stocks, however, is that, obviously, it is difficult. Even though there are challenges to the efficient market hypothesis [12], methods to challenge the hypothesis historically required in-house experts or proprietary models based on financial indicators and environmental factors to identify good stock picks, and even the most sophisticated resources and tools are unreliable. In this paper we propose to augment existing approaches with UGC for this problem. Using aggregate-level expert votes is not new-- in fact, it has been found that in aggregate expert financial analysts tend to perform better than they do alone [9, 13, 19, 20]. But using large-scale user data from online sources is new--especially if it is not known whether the users are experts or not.

We believe the results of this research to be of considerable intellectual and practical value not only to the financial discipline but also to domains where problems are hard and voting on the solutions is possible. In fact, because of our findings in this study, we advocate for reputation mechanisms for all UGC to enable firms and individuals to identify experts and therefore make more accurate predictions and decisions. Identifying the experts in the crowd, or the wisdom of the few, has already been shown to be useful in domains outside finance.

For example, in collaborative filtering, a nearest neighbor collaboration approach was augmented with external expert validation data in order to identify the users that should be considered for the nearest neighbor approach. The approach filters expert users based on their expertise in making accurate recommendations for users [1]. In addition, harnessing the wisdom of the few in Wikipedia has shown to be useful [16, 17]. For Wikipedia, increasing the number of users beyond a threshold is costly because it leads to noisy posts and high coordination needs among users. Finally, most recently, researchers have integrated social network data to study influence in the context of different assumptions about trust of network neighbors on the network [10]. To date, mechanisms for voting and identifying trust in different contexts are few. The landscape of UGC contributions is changing, however.

In the field of investment management and quantitative investing, for example, there is no known prior strategy using broad stock voting data. Until recently, such data were impossible to acquire, because the online voting systems, like CAPS, are unlike any financial voting system previously created. These systems, for the first time, allow us to measure expertise externally. Additionally, this study builds on and extends the work of others that have applied machine learning techniques to portfolio management [5].

We pursue two hypotheses in this work. First we test the hypothesis that using the stock picks of a large sample of online users from Motley Fool CAPS enables us to outperform the S&P 500. Second, we explore the hypothesis that applying our approach to identify experts in the crowds of online stock pickers on the Motley Fool CAPS site will help us do better than the baseline of the S&P 500 for stock price as well as better than letting the entire online crowd vote.

04 hill.indd 75

2/21/2011 11:23:14 PM

76 Hill and Ready-Campbell

This research aims to capitalize on the tension between needing the crowd and needing expert decisions for prediction. Specifically, we evaluate different methods to rank these online experts based on our custom approach against variants of two established methods in the literature: (1) a mixture of experts' strategy [14] that was used on message board posts and the experts' associated sentiments [11] and (2) a method proposed by Fang and Yasuda [7] that was used on real trading data from stock picks of real-world star analysts.

The remainder of the paper is organized as follows: In the next section we discuss our testbed. In the third we discuss our method of utilizing the largescale data to reach a verdict in reference to which stocks to pick. We discuss our results in the fourth section, and in the fifth we conclude with a discussion of future work.

Testbed

The Motley Fool is a well-respected financial newsletter publisher with a strong online presence. The firm created a new service in 2006 called CAPS, a stock voting system whereby each user can make predictions about the performance of stocks--namely, whether they will under- or outperform the market. Users are ranked according to the accuracy of their predictions, and stocks are ranked according to the quantity and quality of the users voting for and against them. In this way, each stock is ranked on a five-star system, with the theory being that stocks with five stars will perform better than stocks with one star. The exact rating equation used by the CAPS site is not public. However, the ranking system described in general terms on the CAPS site is the inspiration for our approach to identify experts from their prior stock pick performance. Our user ranking system is described in detail in the Method section. In the section below, we describe the data used from CAPS to identify the experts.

Data Acquisition

We sourced the publicly available votes directly from the CAPS Web site.1 The data stored do not contain any identifying information on voters, nor are they used for our analysis. We were able to track the votes from January 2007 through December 2009. Altogether, we use over 2 million stock picks in our analysis.

We combined the CAPS data with stock price data from the Center for Research in Securities Prices (CRSP), which was downloaded through Wharton Research Data Services (WRDS). These data were used to calculate returns for stocks (and hence scores for users). CRSP was preferable to other price providers, as it has a history of reliability, and it also provides a "holding period return" value for each stock and trading day. This number differs from a simple ratio of prices in that it takes into account splits, dividends, and other pricing anomalies. The CRSP data also provided S&P 500 prices, which were used for evaluating our overall method as well as for evaluating the expert voters in our data set.

04 hill.indd 76

2/21/2011 11:23:14 PM

International journal of electronic commerce 77

Stock Pick Data

By the end of 2008, there were at least 773,861 registered users. We determined that the number of picks per month appears to be increasing. For each pick, we can collect a number of attributes. An example of user stock picks is shown in Table 1. The picks data were saved in .csv files. The column identities were as follows: the date of the pick; whether the pick was added or removed by the user on this date; the ticker symbol of the applicable stock; whether the pick predicted under- or outperformance; the predicted time horizon in which the under- or outperformance would be realized; the price of the stock when the pick was made; and the hashed ID of the user who made the pick. There were approximately 2 million user-generated stock picks in our data set.

Descriptive Statistics

Over the first testing period (June 2, 2008 through December 24, 2008), we took a simple approach and just let the entire crowd vote on the stock picks. The whole-crowd approach fared very poorly on overall return, ?23.2 percent. However, the S&P 500 total return over the same period was even worse, ?35.5 percent. The difference suggests that we can learn something from the CAPS data.

Users participated in voting at different rates. A significant number of people made few picks, and others made a significant number: the minimum number of picks was 0; the maximum was 13,104. The distribution of number of stock picks is shown in Figure 1.

In Figure 2 (left) we show the distribution of average performance of users with respect to their prediction accuracy. In this plot a user gets a prediction right if the user says that a stock will outperform the market and the stock price for that stock goes up the next day. Likewise, if the user predicts underperformance and the stock goes down, we count it as an accurate prediction in Figure 2. For an individual user we take the number of correct picks divided by the total number of picks. On average, users are 49.1 percent correct, by this definition of correct, over the entire test data set, January 2007 through December 2008. On the surface the plots look as if the stock picks of the users are just a coin flip--the users get the stock movement direction right 50 percent of the time.

In Figure 2 we see that there are outliers at 0 and 100 percent correct. This is due in part to a significant number of people making only one pick. This one pick is either right or wrong, leading to the outliers. With Laplace correction, we are just advocating that one should take the number of picks a user makes when assigning a probability to how right the user might be in the future based on the user's picks.

If we apply Laplace correction to adjust for the variation in the number of picks, we get the distribution of "probability estimates"--the likelihood the user is correct--corrected for the number of picks the user made shown in Figure 2 (right). The plot indicates that indeed some people perform better than others with respect to just getting the direction of stock movement right.

04 hill.indd 77

2/21/2011 11:23:14 PM

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download