Please note the following System Requirements. Further, please limit the number of open applications (particularly price streaming applications) while logged in to Alpha Theory™.

Recommended System Specifications
Processor: Dual Core or Quad-Core 2.4GHz or faster
Browser: Google Chrome 30+
Screen Resolution: 1280 x 1024 or greater
Internet Access: Business Class High-Speed

Minimum System Requirements
Processor: Intel Pentium-M 2.0Ghz or equivalent
Browser: Google Chrome, Mozilla Firefox, Internet Explorer 9+ (without Compatibility View), Safari
Screen Resolution: 1024 x 768 or greater
Internet Access: High-Speed

Alpha Theory Blog - News and Insights

July 18, 2016

Superforecasting – Alpha Theory Book Club

Alpha Theory hosted its first ever book club on July 12th with over 40 portfolio managers, analysts, and allocators coming together to discuss “Superforecasting” by Phil Tetlock. We were lucky enough to have two Superforecasters, Warren Hatch and Steve Roth, moderate and perform forecasting exercises with the group. We spent 2 ½ hours together and only scratched the surface on applying Superforecasting to investing.

Here are a few key takeaways:

1. RAW TALENT: On average, our group had the attributes of Superforecasters with high Active Open-Mindedness (3.99 out of 5) and high Fluid Intelligence (8 out of 10 – this is the highest score that the Good Judgement folks have seen).

Active Open Mindedness



Fluid Intelligence


2. IDENTIFYING TALENT: There are identifiable attributes that can be used in hiring and have a profound impact on forecasting skill (40% - see chart below).

Screen Shot 2016-07-18 at 4.02.10 PM

3. DEVIL’S ADVOCATE: Firms should appoint a Devil’s Advocate for each investment to expand critical thinking (someone to ask the question, “I see your downside is $40. How is that if the 52-Week Low is $22 and the trough multiple would put it at $25?”)

4. OUTSIDE VIEW: Firms should require an Outside View for every investment idea (“While everyone I’ve spoken to says this deal will close, only 20% of deals with one party under SEC investigation close.”)

5. REFINEMENT: New information should always be incorporated in forecast (think Bayesian).

6. POSTMORTEM: An Accuracy Score should be calculated for every investment and should frame the conversation of “what did we do well?” and “what did we do poorly?”.

7. TEAMS MAKE BETTER FORECASTS: Team dialog generally improves forecasting accuracy.

8. FORECAST CULTURE: Firms should embrace “forecast” as part of their vernacular and conversations should revolve around how information impacts the forecast.

9. MEASURE TO BE BETTER: We all forecast, but we rarely measure. That fact needs to change if we really want to improve.


The October Alpha Theory Book Club topic will be “Success Equation: Untangling Skill and Luck” by Michael Mauboussin. Mr. Mauboussin will moderate and highlight the book’s application to investing. Contact your Alpha Theory representative if interested in attending.


June 21, 2016

How Good Are My Analysts? Building a Better Hedge Fund Through Moneyball & Superforecasting

Traditionally, measuring hedge fund analyst skill has been an opaque process mired in ambiguity and subjectivity.  It is often misconstrued and tainted by portfolio manager influence in the form of sizing decisions, liquidity constraints and other non-analyst determinants.  But, in the same way Moneyball revolutionized evaluating baseball player value by prioritizing on-base percentage over batting average, Alpha Theory has distilled the key indicator for predictive aptitude. Alpha Theory invented the Alpha Theory Accuracy Score to introduce radical transparency into the rating of forecasting skill for hedge fund analysts.

P&L is Yesterday’s Batting Average

Using the Moneyball analogy, quantitative disruption of baseball player evaluation changed the way players are paid by isolating the player skill that contributes most to team wins. Using that data, managers now pay athletes in proportion to the amount of that winning skill they individually possess.  As such, the key metric for baseball player value evolved from batting average, to the more predictive on-base percentage, or OBP. 

Specifically, OBP has a 92 percent correlation with runs scored compared to batting’s 81 percent, making it more predictive.  Also, OBP’s 44 percent correlation year-to-year is more persistent than the 32 percent correlation of batting.  The predictive reliability and performance consistency make OBP a superior metric to forecast wins for baseball teams.  OBP’s disruption of batting average is an apt metaphor for the way Alpha Theory’s Accuracy Score will transform analyst ranking and assessment today.      

In 2016, analysts are still primarily rated by the profits and losses their investments generate for the fund, or P&L.  But making money on an investment is a misleading measure of analyst skill.  Beyond its tendency to be distorted by portfolio manager discretion, P&L performance, both good and bad, often masks the integrity and quality of investment processes.  Thus, P&L often misleads portfolio managers into thinking lucky analysts are actually skilled and vice versa.

For example, take these two analysts:

How good is my

Looking at the table above and using P&L to measure skill, Analyst #1 would be exceptional and Analyst #2 would be sub-par.  But Analyst #1 and #2 had the same forecasts, so their forecasting skill is actually identical.  P&L does not translate into forecast skill because analysts do not have ultimate control over position sizing; the portfolio manager does!

More Science, Less Art                                                                                                                         

Inspired by the ideas presented in the groundbreaking book, Superforecasting: The Art and Science of Prediction, Alpha Theory’s Accuracy Score delivers quantitative insight into a qualitative blind spot for portfolio managers.  Authored by Wharton Professor Phillip Tetlock and Dan Gardner in 2015, Superforecasting applies a Brier Score-inspired approach to quantifying predictive skill.  The Brier Score was created by meteorological statistician, Glenn Brier, in 1950 and measures the accuracy of probabilistic outcomes.  Superforecasting applies Brier’s methodology to only binary, or yes/no, outcomes.  

The New Standard

Alpha Theory’s Accuracy Score is an algorithmic solution that measures analysts’ predictive skill over a 0 - 100 percent range, where 100 is the best.  Scores are calculated on a per-forecast basis and then averaged per analyst.  The Accuracy Score algorithm transforms point estimate price targets and probability forecasts into an implied probability distribution, enabling each forecast to be independently scored.  By distributing multi-faceted outcomes across a range of probabilities, the Accuracy Score can measure forecasting skill for any price along the distribution.

The distribution of scores across our Alpha Theory clients is shown below.  The results follow a normal distribution, which further validates the Accuracy Score’s efficacy in rating analysts’ ability to forecast future price movements.

Screen Shot 2016-06-21 at 9.41.12 AM

Good forecasts are the most essential component of fund success and critical when portfolio managers are sizing positions.  Using a data-driven approach to determine which analysts make the best forecasts allows managers to apply those forecasts with greater confidence, leading to better position sizing and superior performance.

The Good Judgement Project

In 2011, the Intelligence Advanced Research Projects Activity, a U.S. government research organization, sponsored a geopolitical forecasting tournament that would span 4 years. The IARPA tournament enlisted tens of thousands of forecasters and solicited more than 1 million forecasts across nearly 500 questions related to U.S. national security.

A group called the Good Judgement Project entered the competition, engaged tens of thousands of ordinary people to make predictions, and the won the tournament. The GJP’s forecast accuracy was so persistent that IARPA closed the tournament early to focus exclusively on them. In fact, GJP was able to find a select group of “Superforecasters” that generated forecasts that were "30 percent better than intelligence officers with access to actual classified information.” 

Ways to Improve Forecasting Skill

The main findings of the GJP and the book that followed are especially relevant to investors. The research in Superforecasting indicates that predictive accuracy doesn’t require sophisticated algorithms or artificial intelligence.  Instead, forecast reliability is the result of process-oriented discipline.  

This process entails collecting evidence from a wide variety of sources, thinking probabilistically, working collaboratively, keeping score and being flexible in the face of error. According to the book, the 10 traits that most Superforecasters possess are: 

    1.  Intelligence - above average, but genius isn’t required

    2.  Quantitative - not only understand math but apply it to everyday life

    3.  Foxes, not hedgehogs - speak in terms of possibilities, not absolutes

    4.  Humility - understand the limits of their knowledge

    5.  System 2 Driven - use the logic-driven instead of instinct-driven portion of their brain

    6.  Refute fatalism - life is not preordained

    7.  Make frequent and small updates to their forecast based on new information

    8.  Believe that history is one of many possible paths that could have occurred

    9.  Incorporate internal and external views


Accountability = Profitability

Organizations cannot improve without systematic and data-driven assessments of their personnel.  Take Bridgewater Associates, for example.  One of the primary factors driving the persistent outperformance of Ray Dalio’s storied fund has been the institutional commitment to radical transparency and accountability.  Similarly, Alpha Theory’s Accuracy Score illuminates blind spots and holds analysts accountable through the precise measurement of predictive skill. For funds that lack the time, inclination or internal resources to create their own probabilistic forecast-grading models, Alpha Theory’s Accuracy Score fills the void.

To this end, Alpha Theory is exploring areas of collaboration with the leadership of Good Judgment Inc. (a spin-off from the Good Judgement Project in “Superforecasting”).  As the competitive landscape for investment capital tightens, discretionary managers must leverage probabilistic data to survive.  Alpha Theory’s Accuracy Score is a mission-critical asset that can help funds compete in the current investment landscape, improving operating inefficiencies and better aligning analyst pay with their intrinsic value to the firm.

May 31, 2016

All in a Day’s Work – Mental Capital Allocation

A Portfolio Manager acts as the final arbiter of what is a good idea and what is not. The Portfolio Manager’s acumen is what investors are ultimately paying for. They need to know each position almost as well as the analyst to be able to properly position it against the rest of the positions in the portfolio. The problem is that most PMs don’t have the time to carry that responsibility because they’re invested in too many positions.

We can prove that PMs don’t know their positions well enough by breaking down a PM’s performance over the course of a year. Let’s assume that the average portfolio manager has 2,500 work hours a year to dedicate to stock analysis (50 weeks times 50 hours). The amount of time dedicated to stock analysis is determined by calculating the percentage of time dedicated to raising capital, communicating with existing investors, running the business, and staring at the P&L.

If the typical portfolio consists of 100 positions and has 50% turnover, then the PM had to understand 150 positions over the course of the year. Now, let’s assume that a PM evaluates two ideas for each position they put in the portfolio. That doubles the number of positions analyzed to 300.

All in a day

The result is 6 hours (basically a day) per year for each position in the portfolio. A PM is ultimately in charge of answering a simple question, “Do I believe my analyst’s assumptions?” 6 hours is hardly enough time for due diligence of that sort.

Don’t take my word for it. Do the math for yourself:

    1. HOURS: Measure your yearly hours using the steps above

    2. ACTUAL POSITIONS: Measure historical average positions per year

    3. ANALYSIS TIME: Measure the time it takes to properly analyze a company (take an example of work you felt really good about)


    5. RESULT: If Actual Positions is greater than Theoretical Max Number of Positions, then you have too many.

Self-reflection is a key to success because it shows us the proper steps to improve. Reflect on how you allocate your Mental Capital and take the time to figure out if you have the time.

April 25, 2016

The Misperception of False Precision for Price Targets and Probabilities

“Objectivity is gained by making assumptions explicit so that they may be examined and challenged, not by vain efforts to eliminate them from analysis.” – Richards Heuer, Psychology of Intelligence Analysis.

Some investors chafe at price targets because they smack of “false precision.”  These investors are missing the point because the key to price targets is not their absolute validity, but their explicit nature which allows for objective conversation about the assumptions that went into them.  Said another way, price targets improve the investment process because they foster great questions and force the team to be able to defend the methodology behind their calculations.

If “false precision” is the concern, then probabilities are doubly damned. They not only require precise estimation, but are considerably more subjective than price targets. The problem with this argument is that probability is an assumption in the process even if you don’t make it explicit. No investor believes they’re 100% right, so there is always a chance of success and failure. In an implicit process, probabilities are expressed with words like “pretty confident” or “high likelihood”, but they are still an estimate of probability. One that is easily misunderstood and difficult to judge. For the same reason that you should require explicit price targets, you should require explicit probabilities.

Not only do explicit price targets and probabilities help investors make better decisions in the short term, they also allow firms to use historical analytics to measure where they are making mistakes and find ways to avoid them.  Great PM’s should be able to leverage information like:  My analysts assume they’ll be right 74% of the time but are actually only right 51% of the time. I will adjust ask them to adjust their assumptions accordingly. 

To make the differences more concrete, I’ve created an example conversation. Problems are better framed when price targets and probabilities are explicitly defined. Compare these conversations and choose the one that you’d prefer as a portfolio manager (I’ve bolded the differences in the two pitches):

Implicit: I think we should add Baidu to the portfolio because overall earnings are being depressed because their non-search business lines are losing money. The current search business is solid, as Baidu holds a natural monopoly in China like Google does in the U.S.A. and their prospects are even better because their users are rapidly moving from 3G to 4G mobile, which will dramatically increase search demand. Their two money-losing lines are independently worth $10+ billion, but are currently not assigned value. The risk is that they can’t monetize the two non-search businesses and they become big money losers. I think there is a high likelihood this is double based on the sum-of-the-parts over the next year or so and has limited downside based on net cash and conservative earnings and multiple. The Risk-Reward is compelling.

Explicit: I think we should add Baidu to the portfolio because overall earnings are being depressed because their non-search business lines are losing money. The current search business is solid, as Baidu holds a natural monopoly in China like Google does in the U.S.A. and their prospects are even better because their users are rapidly moving from 3G to 4G mobile, which will dramatically increase search demand. Their two money-losing lines are independently worth $10+ billion but are currently not assigned value. The risk is that they can’t monetize the two non-search businesses and they become big money losers. I think there is a 70% chance the stock could hit $350 based on sum-of-the-parts over the next year and has downside to $150 based on net cash and conservative earnings and multiple. The Probability-Weighted Return is 50%.

Your discretion as a portfolio manager is not lessened by explicit assumptions. In fact, the explicit assumptions are examined and challenged in a way that is not allowed by the vagaries of implicit assumptions. What “high likelihood” means to you may be different than your analyst. Your interpretation of “limited downside” may be different. Explicit assumptions aren’t designed to make you guess at the unknowable. They are designed to make sure the investment case can withstand the stress of being explicitly defined. If they can’t, you are investing at your own peril.

This post is an extension of the 2014 post – Explicit Lyrics – Why Implicit Assumptions Are Dangerous.

April 04, 2016

Active Managers: The Time is Now

Clare Flynn Levy, the CEO of Essentia Analytics and a friend of Alpha Theory, wrote an article that I wish I could claim. It sums up many of the reasons why the “time is now” to optimize process to stay competitive.

Selected Quotes:

At the end of the day, whether the strategy is hedge or long-only, active management involves a portfolio manager who reviews the information presented by analysts, risk systems and external sources, and then makes what is ultimately a qualitative judgement call.

Now, more than ever, is the time to stop and consider how you could be learning from past success and mistakes in a more efficient way, refocusing your team’s energy on doing more of what you’re good at, and less of what you’re not.

For even though investment management is an industry whose true value proposition is the way decisions are made, very few managers are doing anything to make the decision-making process a provable competitive advantage for themselves.

Click here for a link to the article.

March 21, 2016

Ruminations on Risk

We present this month an interview with Alpha Theory Advisors head Benn Dunn.  Alpha Theory Advisors is the consulting arm of Alpha Theory, providing investment process engineering and thought leadership, outsourced risk management, research leadership and tactical portfolio management guidance to numerous alternative investment firms currently managing approximately $6Bn in AUM across all asset classes. Prior to joining Alpha Theory, Benn served as the Head of Risk Management at the CR Intrinsic Investors unit at S.A.C. Capital Advisors and Chief Risk Officer at Weiss Multi-Strategy.

  1. What is making this market environment so different from others where mid- to high-single digit pullbacks have occurred? Index performance does not appear outright disastrous but we have heard of many hedge funds seeing drawdowns disproportionate and unexpected relative to their exposure levels and typically conservative stance vis-à-vis directional bets.

Things have been changing rapidly versus last year, in that there are many hedge fund-specific issues, even outside of macro and market dynamics, starting to occur.  We are seeing a perfect storm where multi-manager platforms that tend to run (by mandate) market-neutral had levered up in some cases very substantially, and then saw their risk models come apart given an increasingly damaging – and self-perpetuating – unwind in momentum but also other segments.

Crowded growth names like LinkedIn, Tableau and others during earnings season saw declines of as much as 50% in a day – unprecedented in their stock histories – after reporting ugly results.  A widely circulated note from Cornerstone in January essentially fit this increasingly talked-about narrative (that momentum was set to unwind), and that only contributed to the perfect storm.  The short of it is that many risk models, which had until last week been predicated on being market-neutral, factor-neutral, sector-neutral, etc., no longer held up as such.

When this kind of thing starts to happen, the primary and pervasive response is to cut gross exposure.  A big part of the stealth correction and behind-the-scenes damage in the early part of this year can be attributed to reductions in gross, and the result was that crowded names with “small exits” became very dangerous.

  1. What advice, as a result, are you giving clients?

While the contrarian in me suggested some buying of beaten-up but quality assets, I also suggested clients be somewhat defensive, and indeed keep gross exposure in check, for a few reasons.  There is still a very unclear macro environment at the moment, whether it’s the Fed, China, energy company debt problems, Mid-East geopolitics, etc.  The list of reasons the market could go down remains much longer than the list of reasons the markets should go up.

In addition, asset allocators, who tend to avoid over-staying their welcomes and sometimes redeem first and ask questions later, seem in some cases to be pulling out of the multi-manager platforms that were supposed to behave more neutrally.  So that has added to what seemed for too much of this year to be a self-perpetuating risk environment, and the reduction in gross exposure that can have knock-on effects.  One note I saw out of Morgan Stanley in February highlighted that gross exposure started the year at the highest it had been since 2008.  And yet we have been hearing some very well-known and substantial market-neutral platforms have experienced drawdowns that are very atypical for their style and approach.

  1. What can stop the bleeding or draws a line in the sand?

The downside is sustained until CROs (Chief Risk Officers) get their funds’ risk levels back in line and some of these multi-managers are done cutting gross exposure.  Some funds will eliminate individual sector groups for violating drawdown limits, so capital exposed comes down to risk levels that are within appropriate limits.  And finally, some stocks may go to equity values that are below any rational level associated with even liquidation values.

  1. What might happen at the macro level to help the dust settle?

One silver lining here of late is the weakening dollar.  If the narrative was China being forced to weaken its currency and emerging markets hurting due to the strengthening dollar, then these problems start to get alleviated as the dollar backs off its recent strength.  Oil prices go up, along with other commodities.  Of course while all this helps, there are some big market-neutral platforms that saw drawdowns of as much as 5%-20% for January, and this on top of at least one immediately prior weak year.  So many allocators will pull back from these platforms and try to high-grade their books even as some of the dust is settling.

  1. If the contrarian in you has you tempted to do something, what is it?

The contrarian in me is wanting to back up the truck and buy equities of companies without debt maturities in the near term or significant debt at all, where there is a sustainable business model, positive free cash flow, no need for access to capital markets.  These are stocks that represent babies being thrown out with the bath water.  The next trick is whether a fund has something closer to permanent capital – or at least locked-up capital, because that allows a fund to wait for and survive the bottom and benefit from the inevitable reversion to the mean.

  1. If a fund does not have permanent capital and is somewhat levered and net long exposed, what is the most appropriate advice?

One has to de-gross or take down overall exposure.  Of course, this exacerbates the downside among crowded names, where everyone is selling the same things at once.  But the problem with a levered fund is that risk becomes existential, there is actual business risk for a fund, and my job as a CRO is to prevent that above all.

  1. Should funds reduce net long exposure or should they focus on gross?

Over the past couple of months gross represents risk.  When the multi-strategy funds are unwinding or their longs have sold off well more than their shorts, they can't just cover their shorts and let themselves get off-balance on that score.  So they are just de-grossing; they cannot organically take one side of the book down without doing so on the other side.  What this means is that even with an up tape, funds can still get hurt badly if they own some crowded names.

  1. Are there some over-arching themes to be aware of in addition to all this?

It’s been talked about for some time now that liquidity has become extremely limited among some stocks in the market, especially as you go down the market cap ladder.  This causes exits to be very narrow and even if only one fund has to unwind its book, there are simply not enough incremental buyers to take on the stock for sale if the selling has to be fast and sizeable in nature.

There can be secondary and even tertiary effects where the fund exiting a set of names has overlap with names of a second fund that cause the second fund to become stressed, which in turn could impact a third fund holding only a couple of the second fund’s assets.

For a fund with longer-dated capital, this kind of forced or artificial liquidation can represent an opportunity.  Going through the big holdings of what are known to be highly stressed funds and correlating to market volumes and trader scuttlebutt to build a sense for completions to unwinds, name-by-name, is not an uncommon practice.

  1. For funds that lack a CRO (or the means to pay for one), what advice would you give?

There are available tools that can help a fund measure its exposure to any number of factors; even Bloomberg can be of help.  One should be aware of some common mistakes and pitfalls from a risk standpoint: beta, thematic or industry mismatch.  Being long software and short semis – even if beta matched – can prove to be a sector mismatch; one can be too heavily long growth and short cyclicals without being completely aware of such nuances and potential consequences.  There are things that can go wrong on the earnings front that can shift how the fund appears to a risk model, even overnight.

  1. What other risk variables are commonly overlooked?

The Barra model now counts so many factors (I think there are something like 64 industry groups alone).  There is country exposure, basis risk, and even temporal variables to be aware of.  For instance, the short energy/long consumer trades people put on may appear appropriate at first, but the consumer names only see their up move resulting from low energy prices with an extreme lag (up to6 months to a year).  A fund can have balance sheet mismatches, where one is long companies that need access to capital and short some less levered ones.  Or, a fund could be long a few European banks and short some U.S. financials, where the market’s treatment of those different categories cause downside from a geographic mismatch.

  1. What aspects of Alpha Theory can a PM make use of to be more risk-aware and mindful of exposures?

The Confidence Checklist is certainly one element of the application that can allow users to score or grade for risk factors of any kind.  Re-underwriting one’s price targets – even to incorporate recessionary earnings scenarios or market multiples – can also be a potential help.  Just focusing on individual company outlooks and prospects for their own sake and ignoring some of the noise can be a constructive exercise that Alpha Theory forces.  Analysts should be making these company-by-company assessments regularly.

February 22, 2016

How Do Hedge Funds Become Better Forecasters? - A collaborative study between Novus and Alpha Theory.

We believe that one of the few untapped frontiers in Alpha Generation is measuring and putting process around forecasting.  Alpha Theory co-authored “How Do Hedge Funds Become Better Forecasters?” with our friends at Novus to explore a few ways investors can improve their process and forecasting acumen.




Selected Quotes from the Article:

“Many investors chafe at price targets because they smack of “false precision". Those investors are missing the point because the key to price targets is not their absolute validity but their explicit nature which allows for objective conversation about the assumptions that went into them.”

“Unlike real life, investors can track every investment choice they have ever made. Being able to analyze statistically significant trends on a complex and numerate datasets is a huge advantage and is a crucial tool in avoiding the confirmation biases that anecdotal thinkers lean on when rationalizing decisions.”

“Developing a process orientation isn’t about stifling fluidity or gut feel. It is about recognizing that intuition is actually an informal process. By being able to document and empirically study past behaviors, all investors can understand flaws in their internal process.”

January 26, 2016

The Performance of Process

We would like to welcome our first blog post from our very own Justin Harris, Data Analyst here at Alpha Theory:

With markets more volatile than we’ve seen in years, it’s easy to give into emotional biases that affect good decision making. It may be gut wrenching to see positions lose value but avoiding mental pitfalls and following a disciplined investment process can improve returns by 4x based on our research. Here’s how we measured it.

Alpha Theory provides software and consulting to help firms build a repeatable decision process. The process driven software suggests position sizes based on an asset’s expected value and set of constraints set by the client. Essentially, users create their own rules-engine for efficiently sizing positions by determining what factors matter most. Users input their price and probability forecasts which are translated into a suggested position size by comparing it against the user’s constraints.

Across our client base, users can log into the app to see expected value changes, can update their forecasts, and can add new forecasts. We rate engagement with the app based on login count, how many of their current holdings have a price target forecasts, the freshness of those forecasts, and the correlation of the current book’s position sizes to optimal position sizes.

All these metrics would show how disciplined an investor really is. For example, users who run a book closer to the optimal position size, given the process they’ve outlined, would be more disciplined than those that ignore the position sizing recommendation. Those who log in frequently to measure deviations in position sizing can make regular adjustments to optimize the portfolios expected return.

We did a study to see how users with higher app engagement scores ranked against those with lower engagement scores. The sample size was 48 clients. The results of the study show that adherence to process and monitoring valuations more closely, as measured by forecast freshness, coverage and logging into the app, was rewarded. The table below shows that each quartile outperformed all lower quartiles.


Discipline is difficult when our natural instincts are working against us and that is why adherence to a methodology that optimizes a funds expected value using a process based, disciplined approach is key to differentiation as market turmoil attempts to shake out those less disciplined.

December 29, 2015


The end of each year is a time for reflection, a time to evaluate the things we’ve done well and places we can improve. Here’s a list of questions you can ask yourself as you look towards the new year:
1. GOLDEN RULE: How often do we follow the investment Golden Rule? (Golden Rule: If I were investing in this asset for the first time today, would I be at this position size?)
2. SMALL POSITIONS: Do small positions have a positive or negative impact on our performance and process?
3. MENTAL CAPITAL: How effectively do we allocate our mental capital?
4. NEW IDEA PROCESS: How can we improve our process for getting new ideas in our portfolio?
5. POSITION SIZING: Is there a formal process for sizing positions?
6. EXISTING IDEA PROCESS: How can we improve our process for managing existing positions in our portfolio?
7. PRE-RESEARCH CHECKLIST: Do we have a checklist of things that preclude us from investing?
8. RESEARCH CHECKLIST: Do we have a checklist for things we deem important for every investment?
9. SELF-IMPROVEMENT: Do we encourage our team to improve their mental models and outside view?
10. MEASURE & FEEDBACK: Do we have a continuous process of measurement and feedback that allows our team to see how they’re doing?


1. Self explanatory
2. Small positions that consume time and energy (mental capital) but do not have a material impact on performance are a drag on efficiency.
3. In a given year, your team has a limited number of hours to devote to new and existing investments. It is important to figure out the total number of hours per analyst (usually around 2000 hours) and divide that by the number of hours you expect each analyst to spend on an investment. This will approximate the total number of investments an analyst can research.
4. Is there a formal process for pitching an idea that allows for the devil’s advocate, evaluation of the investment checklist, etc.?
5. Position sizing is almost as important as asset selection. What are the inputs that determine how we size positions and what is our process to make sure we’re appropriately sized?
6. What is our process for monitoring existing positions? Do we have a process for taking new information and determining how it should change position size? As prices change, how do we ensure that we maintain the appropriate position size? When prices go against us, what is our process for re-underwriting the position?
7. There are certain factors (cost to borrow, liquidity, initial valuation, crowdedness, etc.) that may preclude an investment and, if discovered early, would save mental capital.
8. There is a set of requirements for every investment decision (model, conversation with management team, conversation with at least three contrary opinions, assumptions stress test, etc.). Are these documented so that important items aren’t missed?
9. Google gives its employees “20% Time” to allow them to work on projects of their own choosing. The idea is that inspiration comes from many sources and employees can be bogged down in a “thought routine” and lose perspective. How can you help your team break out of those ruts with encouraged self-improvement? Forced day out of the office? Reading list? Classes? Creative “in-the-field” analysis (see the film “The Big Short” for examples).
10. Improvement requires feedback. Feedback requires measurement.

Clearly there are dozens more great questions (and if you have any, please send them our way), but this is a good start to frame the conversations for how you can become even better investors in 2016.

November 30, 2015

Superforecasting: The Future of Analyst Performance Measurement

Anyone that buys a stock is a forecaster. The only reason to buy a stock is because you believe the value of the stock will be greater in the future than it is today. That is the definition of a forecast. But how do you know if you’re any good? How do you measure your stock forecasts? And how can you use those measurements to improve your forecasts in the future? Answering these questions is the next mission of Alpha Theory. A vision that was crystalized by the illuminating book “Superforecasting”.

P&L is the primary measure of forecasting acumen used today. But making money on an investment is only a crude measure of forecasting skill. It can be corrupted by trading decisions, sizing decisions, liquidity constraints, etc. Positive P&L can also mask bad process and outlier P&L observations can overwhelm the data and convince you that luck is actually skill. Forecasting skill should be measured by actually measuring price forecasts. That seems obvious, but there are very few managers actually doing it.

If it isn’t obvious, the primary reason for a better forecasting measure is so that you can improve. There are hundreds of psychology, intelligence, and physical training studies that show how important good feedback is to making improvements (Feedback | Mundanity of Excellence). Right now, the investing community operates like a golfer trying to improve his swing by practicing at night with no lights. He can feel if he made good contact and has a general sense of direction, but doesn’t know how far his ball landed from the target. Turn the lights on and the golfer gets immediate feedback that allows him to improve rapidly. I believe that will happen in investing for those willing to “turn the lights on” by making price target forecasts a required part of their process.

What is required? The first step is getting explicit price forecasts from your analysts. The ideal way is to ask for a range of potential outcomes with probabilities. This allows your analysts to describe the range of possibilities that come from their research. Step two is to measure the forecast in a way that allows you to compare analysts. Step three involves creating feedback that helps refine and improve your process.

Step One: Capture Price Forecasts

A system that stores a history of all price targets and allows analysts to make frequent updates is required (this statement is clearly self-serving given what Alpha Theory does, but it’s a requirement none the less). The required inputs are price targets and probabilities that add to 100%. Additional nice to haves are a time horizon for price targets and rationale for how the price targets were derived.

Step Two: Measure the Forecast

The plan is to use a modified Brier Score.


f = probability of outcome

o = outcome (1 if occurs, 0 if it doesn’t)

Brier Scores were designed to measure probability-based binary outcomes. For example, assume I forecast an 80% chance that Donald Trump is the GOP Presidential Candidate in 2016. The math works like this.

BS if not nominated = (0.8 – 0)^2 = 0.64

BS if nominated = (0.8 – 1)^2 = 0.16

Brier Scores are like golf scores, lower is better. The worst score is a 1 and the best is a 0 (100% probability forecast and you get it right = 0). The issue with simple Brier Score is that it assumes binary outcomes (i.e. Trump is the candidate or he is not). It also doesn’t account for “close” to right (it doesn’t matter if you were close to right forecasting Trump was the candidate but it is important if you were close to your price target forecast). The Alpha Theory team is actively working to develop an adapted Brier Score to measure forecasting acumen. We welcome any and all feedback as we tackle this challenge.

Step Three: Use Feedback to Improve

Over the past five years, Alpha Theory has accumulated over 10,000 price target forecasts from over 500 buyside analysts. With this data, we will test our measurement techniques and give our clients’ forecast assessments and feedback loops required for improvement. Feedback comes in many different forms. Immediate feedback will be notifications when prices meet or exceed price forecasts. The notifications will include the date of the original forecast and the rationale for the target. Intermediate feedback will be the measured difference between the actual and forecast price at the peak, trough, and terminal price. Finally, after an analyst has made a statistically significant number of forecasts, a modified Brier Score will incorporate both price target and probability forecasting acumen.

The future of forecasting involves better measurement. Political pundits, economic analysts, meteorologist, and every other profession that gets paid to make forecasts should have their forecasting score right beside their forecast. Imagine you’re watching CNBC and a floor trader comes on and says, “I believe there is a 70% chance the S&P breaks 2,000 in the next couple of weeks.” And right beside their name you see a Forecasting Score of 0.61 and realize that the forecast is slightly worse than a coin flip. It is much easier for pundits to operate in a world where statements are rarely judged. I think we’d all love that world. If you’re a portfolio manager, you have the power to create a forecasting world where track records and performance are measured and matter. Capture price targets with probabilities, keep a history, and measure accuracy.