Networked Insights recently presented a finding that your tweet could be worth an average of $560 at the box office. Naturally, all of the news outlets read this headline and began regurgitating the finding. Folks, I’m sorry, but this finding is most likely bullshit. I have a graduate degree in Applied Statistics, so I have been trained and educated to understand analyses such as this, and because of such, it is easier to figure out what has some credibility and what is heavy and ponderous poo.
First, let’s go over what Network Insights said they DID do in order to realize that, when doing some critical thinking about the methods used, that their conclusions are largely incorrect. The data used were taken from over 400 movies since 2012, the number of tweets about those movies up to five weeks before the premiere, and added information about those movies such as movie genre, competing movies that weekend, and the number of theaters showing that movie. According to the article about this analysis in Variety, they also removed 90% of all automated accounts, promotions, giveaways and other tweets that didn’t represent true consumer reactions, and tweets from a film’s stars.
The rest of their methods are, conveniently for them, a black box since they don’t explicitly say what they did. It sounds as though they ran a regression to predict a movie’s box office revenue using the number of tweets posted about the movie along with the other control variables previously mentioned. Their take home conclusion is that the average value of a tweet is $560.
Let us now go into nauseating mathematical detail about how this is wrong: as the economics literature will suggest, using a straightforward average (i.e. the arithmetic mean) when analyzing money is stupid. For example: if nine of your friends have one video game and the tenth friend has 991 video games, you could conclude that your friends have an average of 100 games each. Under the assumption that they are using regression, specifically Ordinary Least Squares regression (OLS), any movie that is included in their data that had a very successful box office run will skew the data towards them so as to minimize the squared error. It just so happens that 2012 by itself had three movies which did just that: Marvel’s The Avengers (total gross $623M), The Dark Knight Rises ($448M), and The Hunger Games ($408M). Adding together the box office totals for movies #10 through #12 (Madagascar 3: Europe’s Most Wanted, Dr. Seuss’ The Lorax, and Wreck-It Ralph) would roughly be the same that #1 (Avengers) made. Some remedies for analyzing money include logarithmic transformations to make it closer to a Normal distribution than a skewed one.
The lesson I’m trying to teach is that the summary number of the average is a terrible measure and is not a robust statistic. If just one movie had a different box office result, the average changes. If Avengers instead made $1B instead of $642M, the resulting average per tweet would go way up. A better basic summary measure is a median, which would only change if at least half of the data changed. Because the idea is to adjust for other factors that would influence the box office revenue via regression, something like a robust regression would at least down-weight outliers so that they influence the final regression less than a typical result.
Their regression is using two main components; it is using the number of tweets about a movie to predict the box office revenue. They are prescribing a linear relationship between the number of tweets a movie receives and the money it makes, subject to the adjustment of movie genre, competing movies, and number of theaters. Therein lies part of the logical nightmare that this spurious conclusion provides. You can imagine that there was a very high number of tweets for The Avengers and it also made a lot of money. There was a lot of excitement and press about Marvel’s biggest production to date. You know what people probably didn’t tweet too much about: Wreck-it Ralph.
The regression has basic inputs that mostly come down to yes/no (binary variable) or a classification for genre of movie. The number of tweets is troubling because if The Avengers had 5x as many tweets as every other movie, that would make it a high leverage point and influence the final regression reults towards it. This could also be a reason that the final $560 conclusion may be much higher if The Avengers had a much higher number of tweets along with its high box office take.
The trouble with using Twitter as a causal link to box office performance is that Twitter has a high volume of tweets coming from a particular age group, which was probably right in the sweet spot to go see The Avengers and did not sync with Wreck-it Ralph. Both movies opened to roughly the same number of theaters and both stayed in theaters about the same duration. Granted, their models do account for genre, but that’s where another problem lurks. Using genre most likely acts as a confounder since it is almost assuredly related to both box office revenue and the number of tweets it would have. The result of having a confounding variable in the regression is that the estimate attached to the money per tweet will be different because of the association between the three variables (box office, tweets, genre).
If I had to assign an overall thesis to this diatribe towards the findings that Networked Insights is advertising, it is that not enough due diligence is being given to analyzing these data with proper methods and not enough education to reading and interpreting statistical findings. Everyone who read the findings should have had alarm bells or at least mild skepticism when seeing the word “average” and should have objected to very the sparse results provided by Networked Insights. Why weren’t there any summary tables? Why weren’t the model results shown? What tweets were actually counted? It’s hard to give credibility to these results when the methods are concealed behind the curtain. News outlets should also exercise more caution since they blindly repost a result like this without their own due diligence to see that it has some credence.
“Making you a better geek, one post at a time!”