Please note the following System Requirements. Further, please limit the number of open applications (particularly price streaming applications) while logged in to Alpha Theory™.

Recommended System Specifications
Processor: Dual Core or Quad-Core 2.4GHz or faster
Browser: Google Chrome 30+
Screen Resolution: 1280 x 1024 or greater
Internet Access: Business Class High-Speed

Minimum System Requirements
Processor: Intel Pentium-M 2.0Ghz or equivalent
Browser: Google Chrome, Mozilla Firefox, Internet Explorer 9+ (without Compatibility View), Safari
Screen Resolution: 1024 x 768 or greater
Internet Access: High-Speed

Subscribe to Alpha Theory content

Alpha Theory Blog - News and Insights

3 posts categorized "Superforecasting"

September 12, 2016

The Day the Music Died

For the past few years, I have had a significant edge in my fantasy football leagues because of my pre-draft preparation.  Each year, I would go online to do my preparation before the draft and pull data from multiple sources and look around for novel analytical approaches to player selection which I could combine with the data I was able to source.  Unfortunately, it looks like the years of my fantasy football spreadsheet’s domination has come to an end. I knew this day would come. Each year the data and analysis I was running sourcing seemed to get better and better. This year, I found 95% of my bespoke spreadsheet already prepared by someone else online at: http://apps.fantasyfootballanalytics.net/.  Undoubtedly, several of my competitors found it too.

Key attributes of my fantasy football sheet:

1. Improvement over Average (also called Value over Replacement) – this is by far the most important attribute. It basically states how much better any particular player is to the average player that will be drafted at the same position. Without IOA, the players with the highest potential points will always pop to the top, which is inaccurate because you can’t fill a team with just one position. For example, a QB may be forecast to score 350 points over the year and a WR is forecast to score 325. On that basis, the QB looks like the favorite. But the average drafted QB will score 280 and the average drafted WR will score 180 points. That means that the WR will add 145 points (325-180) against other teams vs the QB which will only add 70 points (350-280).

2. Multiple Sources – I pull projections from multiple sites to create a “wisdom of crowds” improvement.

3. Forecast Risk – measure standard deviation of projections from multiple sites to see how likely a player was to hit the average I was using in my model.

4. Average Draft Position – pulled Average Draft Position from multiple sources to compare my ranking to what was likely to happen.

5. Over/Undervalued to Average Draft Position – this allowed me to see if a player that my sheet really liked, could be selected in a later round because folks on average were drafting him later.

6. Dropoff – used to quickly see how big the difference is between the player and the next best player at the same position.


Now let’s compare to FFAnalytics.net:


The only thing my sheet has is a Strength of Schedule adjustment (#7) and I can just add that in after I download the data.

My reasons for writing this blog are two-fold. One, I want everyone to feel sorry for me that my edge is gone. Two, we all should recognize that with each passing year, data becomes easier to access and manipulate. If there are things in your business you consider your “edge” (competitive advantages), do a careful assessment of the landscape and forecast the probability that it can be undermined by data ubiquity, data analytics tools, or machine learning (artificial intelligence). Old businesses are being disintermediated in a matter of a few years by the likes of Google, Alibaba, Amazon, Uber, AirBNB, etc. Asset managers would be well served to take a hard look at what parts of their spreadsheets/processes and analytics are no longer novel and which are likely to persist and contribute as their true “edge”.    Without an edge in your processes or analytics, there is a limited likelihood that they will see an “edge” in their returns. 

July 18, 2016

Superforecasting – Alpha Theory Book Club

Alpha Theory hosted its first ever book club on July 12th with over 40 portfolio managers, analysts, and allocators coming together to discuss “Superforecasting” by Phil Tetlock. We were lucky enough to have two Superforecasters, Warren Hatch and Steve Roth, moderate and perform forecasting exercises with the group. We spent 2 ½ hours together and only scratched the surface on applying Superforecasting to investing.

Here are a few key takeaways:

1. RAW TALENT: On average, our group had the attributes of Superforecasters with high Active Open-Mindedness (3.99 out of 5) and high Fluid Intelligence (8 out of 10 – this is the highest score that the Good Judgement folks have seen).

Active Open Mindedness



Fluid Intelligence


2. IDENTIFYING TALENT: There are identifiable attributes that can be used in hiring and have a profound impact on forecasting skill (40% - see chart below).

Screen Shot 2016-07-18 at 4.02.10 PM

3. DEVIL’S ADVOCATE: Firms should appoint a Devil’s Advocate for each investment to expand critical thinking (someone to ask the question, “I see your downside is $40. How is that if the 52-Week Low is $22 and the trough multiple would put it at $25?”)

4. OUTSIDE VIEW: Firms should require an Outside View for every investment idea (“While everyone I’ve spoken to says this deal will close, only 20% of deals with one party under SEC investigation close.”)

5. REFINEMENT: New information should always be incorporated in forecast (think Bayesian).

6. POSTMORTEM: An Accuracy Score should be calculated for every investment and should frame the conversation of “what did we do well?” and “what did we do poorly?”.

7. TEAMS MAKE BETTER FORECASTS: Team dialog generally improves forecasting accuracy.

8. FORECAST CULTURE: Firms should embrace “forecast” as part of their vernacular and conversations should revolve around how information impacts the forecast.

9. MEASURE TO BE BETTER: We all forecast, but we rarely measure. That fact needs to change if we really want to improve.


The October Alpha Theory Book Club topic will be “Success Equation: Untangling Skill and Luck” by Michael Mauboussin. Mr. Mauboussin will moderate and highlight the book’s application to investing. Contact your Alpha Theory representative if interested in attending.


June 21, 2016

How Good Are My Analysts? Building a Better Hedge Fund Through Moneyball & Superforecasting

Traditionally, measuring hedge fund analyst skill has been an opaque process mired in ambiguity and subjectivity.  It is often misconstrued and tainted by portfolio manager influence in the form of sizing decisions, liquidity constraints and other non-analyst determinants.  But, in the same way Moneyball revolutionized evaluating baseball player value by prioritizing on-base percentage over batting average, Alpha Theory has distilled the key indicator for predictive aptitude. Alpha Theory invented the Alpha Theory Accuracy Score to introduce radical transparency into the rating of forecasting skill for hedge fund analysts.

P&L is Yesterday’s Batting Average

Using the Moneyball analogy, quantitative disruption of baseball player evaluation changed the way players are paid by isolating the player skill that contributes most to team wins. Using that data, managers now pay athletes in proportion to the amount of that winning skill they individually possess.  As such, the key metric for baseball player value evolved from batting average, to the more predictive on-base percentage, or OBP. 

Specifically, OBP has a 92 percent correlation with runs scored compared to batting’s 81 percent, making it more predictive.  Also, OBP’s 44 percent correlation year-to-year is more persistent than the 32 percent correlation of batting.  The predictive reliability and performance consistency make OBP a superior metric to forecast wins for baseball teams.  OBP’s disruption of batting average is an apt metaphor for the way Alpha Theory’s Accuracy Score will transform analyst ranking and assessment today.      

In 2016, analysts are still primarily rated by the profits and losses their investments generate for the fund, or P&L.  But making money on an investment is a misleading measure of analyst skill.  Beyond its tendency to be distorted by portfolio manager discretion, P&L performance, both good and bad, often masks the integrity and quality of investment processes.  Thus, P&L often misleads portfolio managers into thinking lucky analysts are actually skilled and vice versa.

For example, take these two analysts:

How good is my

Looking at the table above and using P&L to measure skill, Analyst #1 would be exceptional and Analyst #2 would be sub-par.  But Analyst #1 and #2 had the same forecasts, so their forecasting skill is actually identical.  P&L does not translate into forecast skill because analysts do not have ultimate control over position sizing; the portfolio manager does!

More Science, Less Art                                                                                                                         

Inspired by the ideas presented in the groundbreaking book, Superforecasting: The Art and Science of Prediction, Alpha Theory’s Accuracy Score delivers quantitative insight into a qualitative blind spot for portfolio managers.  Authored by Wharton Professor Phillip Tetlock and Dan Gardner in 2015, Superforecasting applies a Brier Score-inspired approach to quantifying predictive skill.  The Brier Score was created by meteorological statistician, Glenn Brier, in 1950 and measures the accuracy of probabilistic outcomes.  Superforecasting applies Brier’s methodology to only binary, or yes/no, outcomes.  

The New Standard

Alpha Theory’s Accuracy Score is an algorithmic solution that measures analysts’ predictive skill over a 0 - 100 percent range, where 100 is the best.  Scores are calculated on a per-forecast basis and then averaged per analyst.  The Accuracy Score algorithm transforms point estimate price targets and probability forecasts into an implied probability distribution, enabling each forecast to be independently scored.  By distributing multi-faceted outcomes across a range of probabilities, the Accuracy Score can measure forecasting skill for any price along the distribution.

The distribution of scores across our Alpha Theory clients is shown below.  The results follow a normal distribution, which further validates the Accuracy Score’s efficacy in rating analysts’ ability to forecast future price movements.

Screen Shot 2016-06-21 at 9.41.12 AM

Good forecasts are the most essential component of fund success and critical when portfolio managers are sizing positions.  Using a data-driven approach to determine which analysts make the best forecasts allows managers to apply those forecasts with greater confidence, leading to better position sizing and superior performance.

The Good Judgement Project

In 2011, the Intelligence Advanced Research Projects Activity, a U.S. government research organization, sponsored a geopolitical forecasting tournament that would span 4 years. The IARPA tournament enlisted tens of thousands of forecasters and solicited more than 1 million forecasts across nearly 500 questions related to U.S. national security.

A group called the Good Judgement Project entered the competition, engaged tens of thousands of ordinary people to make predictions, and the won the tournament. The GJP’s forecast accuracy was so persistent that IARPA closed the tournament early to focus exclusively on them. In fact, GJP was able to find a select group of “Superforecasters” that generated forecasts that were "30 percent better than intelligence officers with access to actual classified information.” 

Ways to Improve Forecasting Skill

The main findings of the GJP and the book that followed are especially relevant to investors. The research in Superforecasting indicates that predictive accuracy doesn’t require sophisticated algorithms or artificial intelligence.  Instead, forecast reliability is the result of process-oriented discipline.  

This process entails collecting evidence from a wide variety of sources, thinking probabilistically, working collaboratively, keeping score and being flexible in the face of error. According to the book, the 10 traits that most Superforecasters possess are: 

    1.  Intelligence - above average, but genius isn’t required

    2.  Quantitative - not only understand math but apply it to everyday life

    3.  Foxes, not hedgehogs - speak in terms of possibilities, not absolutes

    4.  Humility - understand the limits of their knowledge

    5.  System 2 Driven - use the logic-driven instead of instinct-driven portion of their brain

    6.  Refute fatalism - life is not preordained

    7.  Make frequent and small updates to their forecast based on new information

    8.  Believe that history is one of many possible paths that could have occurred

    9.  Incorporate internal and external views


Accountability = Profitability

Organizations cannot improve without systematic and data-driven assessments of their personnel.  Take Bridgewater Associates, for example.  One of the primary factors driving the persistent outperformance of Ray Dalio’s storied fund has been the institutional commitment to radical transparency and accountability.  Similarly, Alpha Theory’s Accuracy Score illuminates blind spots and holds analysts accountable through the precise measurement of predictive skill. For funds that lack the time, inclination or internal resources to create their own probabilistic forecast-grading models, Alpha Theory’s Accuracy Score fills the void.

To this end, Alpha Theory is exploring areas of collaboration with the leadership of Good Judgment Inc. (a spin-off from the Good Judgement Project in “Superforecasting”).  As the competitive landscape for investment capital tightens, discretionary managers must leverage probabilistic data to survive.  Alpha Theory’s Accuracy Score is a mission-critical asset that can help funds compete in the current investment landscape, improving operating inefficiencies and better aligning analyst pay with their intrinsic value to the firm.