spx S&P 500 Historical Correlations, 1945- February, 2001

S&P 500 Historical Correlations, 1945- February, 2001:

Any historical correlations should be taken with a large grain of salt, but nevertheless, they help put things into perspective. For example, it is meaningless to look only at nominal levels of the S&P 500 when attempting to answer the question, "How high is high?" Instead, we have to put the index in relationship to earnings or dividends, or some other variable, since over the very long run, stocks rise because earnings rise.

Stocks fluctuate much more than earnings do. Why is this? Because investors are willing to pay more for a dollar of earnings during some historical periods, less during others. If anyone can figure out how or why they change their minds en masse, they would be fabulously wealthy, but since no one seems to be able to with any precision, we are forced to make a few assumptions:

1. Over the long run, stocks will rise in rough proportion to earnings;
2. In the short run, stocks can rise much faster than earnings rise and fall much further than earnings fall because investors will bid stocks up to reflect not just earnings today but anticipated earnings tomorrow;
3. Investors tend to overreact in the short-term, bidding stocks up to very high multiples of earnings when they are optimistic, and selling them down to very low multiples of earnings when they are gloomy;
4. This overreaction by investor should create opportunity, allowing some investors to step back and recognize that stocks are historically overvalued or undervalued and act accordingly.

The fourth assumption is the most tenuous. It is hard to disagree with assumptions 1-3 since stocks do tend to overshoot on both the upside and downside. The question as to whether one can systematically exploit this (buying stocks when they are relatively undervalued and selling them when they are relatively overvalued) is open for debate. Several obstacles prevent such a strategy from succeeding:

1. The investing mood of the time is contagious. At the market peak in March, 2000, the fear was not that one could lose money but that one could miss the next 100% move up; technology was changing the landscape, the Fed and worker productivity were believed to have abolished the business cycle, and we were in a New Paradigm. Conversely, in the early 1980s, stocks had just had a miserable decade of performance, inflation was high, and real estate and gold were the rage. It is very difficult to divorce yourself from the predominant investment mood of the age. The investor is both observer and participant.
2. Taxes and commissions can drastically reduce the advantages of moving into or out of the stock market;
3. Past historical correlations sometimes degrade or disappear altogether, leaving one on the sidelines prematurely or permanently while the market advances;
4. Even the strongest correlation indicates that other factors, or random factors, influence price movements in the short-term, so that a perfect model could never be developed;
5. Markets may overreact to the upside and downside with much more than anyone could anticipate;
6. Even during periods of relative underperformance by stocks, other investment classes often perform even worse. Over the very long run, stocks have offered the best appreciation potential, so avoiding them altogether with your investable assets is risky.

Perhaps a balanced approach would be to maintain a certain core position of stocks, say no less than 25%-50% of your portfolio, then to increase this proportion when times are more propitious. Periodic rebalancing (such as selling after a strong market advance to maintain prior proportions) may also decrease volatility.

So with all these caveats, let's look at over half a century of monthly stock data, using the S&P 500.

I first constructed an index that reflected the impact of both reinvested dividends and inflation. The resultant index could then be used to calculate rolling 10-year real (after-inflation) total returns. I then asked a series of simple questions: If you knew that the market was trading at a level of x in a given year in the past, what would your total return have been over the next 10 years even after taking inflation into account?

Now, if investors on balance do not overreact (if assumption #3 is incorrect), but appropriately revalue stocks based on new information (a tenet of the efficient market hypothesis), then the correlation between any variable and future returns of the market should be zero. Why is this? Because if investors are on balance right, then stocks should always be at the "right" value and low price-earnings ratios should not be followed by higher than average returns, nor should low price-earnings ratios be followed by higher than average returns.

As it turns out, investors do get ahead of themselves. Markets are not appropriately valued at all times. Instead, markets at times get relatively overvalued, and over the next decade tend to revert to their former valuations, leading to lower than average returns. This would not occur if investors were able to anticipate the future more accurately, bidding stocks up to higher valuation levels in anticipation of much higher future earnings.

The correlation between each of the variables and the subsequent 10-year real total return of the S&P 500 follows, sorted by the strength of the correlation. 1 is a perfect correlation, meaning that as one variable rises, the other rises also. 0 means no correlation at all, and -1 means a perfect inverse relationship - as one rises, the other falls, and v.v.:

Variable:	Correlation With Next 10 Year's Average Annual Real (Inflation-Adjusted) S&P 500 Return:
Price-Earnings Ratio:	(0.711)
Dividend Yield:	0.684
Earnings Yield:	0.673
10 Year Yield:	0.580
Federal Funds Rate:	0.527
Earnings Yield - 90 Day Yield:	0.387
Earnings Yield Stochastic:	0.340
Yield Curve (30 year / 90 day yield):	0.324
Earnings yield - 30 year yield:	0.308
Change in CPI (year-over-year):	0.300
Change in PPI (year-over-year):	0.237
Real Fed funds rate:	0.167
Earnings yield - Inflation rate:	0.162
30 Year Yield / 6 Month Moving Average:	(0.095)
12 month % change in the 30 year yield:	(0.079)
Yield on 30 Year Treasury - 6 Month Moving Average:	(0.052)
Ratio of S&P 500 to its 6-month moving average:	(0.032)
12 month % change in the 90 day yield:	(0.006)
Last 36 month % change in the S&P 500:	(0.002)
Data: monthly closing values from January, 1945 - February, 2001:

What is fascinating here is that several strong relationships do exist (or at least have over the past 56 years). The strongest one is between the price-earnings ratio and the return you would get over the next 10 years in the market after inflation. This correlation coefficient is -.711, meaning that this is a strong inverse relationship. The lower the price-earnings ratio, the higher your subsequent 10-year return.

Dividend yield, disparaged as an antiquated measure of value, is surprisingly the second strongest predictor of future stock returns. High dividend yields have been followed more often than not by high total stock returns. This would not happen in an efficient market (if for no other reason than investors who know that high dividend yields are associated with future subsequent abnormal returns would buy stocks when dividend yields are high and sell them when they are low, making this relationship disappear).

The earnings yield, or 1 divided by the price-earnings ratio, is the next most powerful relationship, with a correlation coefficient of .673. Since this is the inverse of the price-earnings ratio, this should not be surprising. Buying stocks when they are cheap relative to earnings on balance leads to higher total returns over the next decade than buying them when they are expensive.

The next few variables are not quite as strong in their relationship to the next 10 year returns. Some are surprising. For example, the yield on the 10-year Treasury bond is positively correlated (.580) with future returns in the stock market. Why would this be? One reason is perhaps regression to the mean; when interest rates are very high historically, perhaps they tend to fall, and there is a strong correlation between falling interest rates and rising stock prices. However, since one cannot anticipate the future trend of interest rates, this relationship should be taken with a grain of salt.

The same reason probably applies to the Federal funds rate, a key interest set by the Fed. When it is high, it has room to fall, and no doubt over time does. The relationship between Fed interest rate cuts and subsequent rising stock prices is well-documented.

An interesting, although weaker correlation, exists between a variable I created, the difference between the earnings yield and the 90 day Treasury bill yield. The .387 correlation coefficient is low no doubt because interest rates fluctuate quite a bit over the next decade, and so knowing the earnings yield relative to interest rates is probably less useful than knowing the earnings yield alone. (Note, however, that in the shorter term, the earnings yield relative to interest rates is more useful than the earnings yield alone.)

The earnings yield stochastic is another indicator I developed which simply puts the earnings yield in relationship to its past 3-year history. In other words, if the earnings yield is much higher (stocks are much cheaper) than it was at any time over the past 3 years, stocks tend to perform better than average, over the next 10 years. However, since the correlation between earnings yield alone is much stronger, for long-term stock market anticipated returns, earnings yield alone is probably superior.

The earnings yield minus the 30 year Treasury bond yield is another variable I developed to put earnings yield in relationship to interest rates. It is interestingly less strong than the difference between the earnings yield and 90 day Treasury bill yields.

The year-over-year change in the CPI, the most widely used measure of inflation, had a surprising POSITIVE relationship with subsequent stock market gains. This was surprising and counterintuitive. In other words, during periods of high inflation, you can expect to get higher than average subsequent stock market returns. Two factors may account for this:

1. The tendency of stocks to sell off during periods of high inflation, which in turn creates low price-earnings ratios and high earnings and dividend yields (which we know are correlated with subsequent stock returns); and
2. The tendency of periods of high inflation not to last forever, but to be followed by periods of lower than average inflation (at least historically), and subsequent revaluation of stocks (upwards).

The PPI (producer price index) showed a similar relationship, perhaps for the same reasons.

The Real Fed funds rate (the Fed funds rate minus the year-over-year change in the CPI) was very weakly (.167 correlation coefficient) correlated with future 10-year returns.

Note that normalizing earnings yield for inflation (.162 correlation coefficient) had much less correlation with stock market returns than normalizing them for interest rates (.387 correlation coefficient with earnings yield minus 90 day Treasury bill yield, and .308 correlation coefficient with the earnings yield minus the 30-year Treasury yield). This is probably because the year-over-year change in the CPI only tells us where inflation is today; the bond market, on the other hand, which is exquisitely sensitive not just to current but to future anticipate inflation, gives us some indication about future inflation. The bond market tends to rise only if market participants are confident that the inflation will be low and stable in the future. (It is notable that bonds are currently (March, 2001) behaving very well despite some signs of rising inflation.)

The other variables are next to useless for long-term predictions, as evidenced by their very low correlation coefficients. The ratio of the 30-year Treasury bond yield to its 6 month moving average (-.095), the year-over-year change in the 30-year Treasury bond yield (-.079) and the difference between the 30-year Treasury bond yield and its 6-month moving average (-.052) were all weak predictors of future stock prices. Knowing how much the market rose over the past 3 years was even more useless (-.002).

Bottom Line: Stocks have historically returned the most when earnings yield and dividend yield were high, when price-earnings ratio was low, as you would expect. Regardless of the rationalizations given about why "this time it's different," the stock market when richly valued tends to perform poorly over the next decade, and performs much better over the next 10 years when it is relatively cheap relative to earnings.

The correlation coefficients of other variables was too low to be of much significance, but are more important for what they don't show you then for what they do. Don't be scared away from stocks because interest rates are high or because stocks have risen strongly over the last 3 years, for example. High interest rates today are associated with strong stock market gains over the next 10 years, and there is virtually no correlation between the stock market's returns over the past 3 years and their returns over the next decade.

More Detailed Analysis

So now that we know that "low" values of one variable are associated with "high" or "low" subsequent stock market returns, it might be helpful to know the historical range of these variables. This gets back to the question, "How high is high?" Let's look at each in turn:

			Total S&P 500 Return Per:		Total S&P 500 Real Return Next 10 Years:
	Earnings Yield:	PE Ratio:	Month:	Year:	Total:	Average Annual:
Maximum value:	16.5%	35.4	16.7%	61.0%	474%	19.1%
75th percentile value:	9.4%	18.3	3.9%	25.5%	239%	13.0%
Median value:	6.7%	15.0	1.2%	14.0%	155%	9.8%
25th percentile value:	5.5%	10.6	-1.5%	3.0%	46.4%	3.9%
Minimum value:	2.8%	6.1	-21.5%	-38.8%	-35.4%	-4.3%

The earnings yield, the inverse of the more popular price-earnings ratio, has ranged from a high of 16.5% to a low of 2.8% over the past 55 years. (It currently (March, 2001) stands at about 3.4%.) The median (middle) value was 6.7%, meaning that 50% of the time the earnings yield was greater than this and 50% of the time it was less. The 25^th percentile - the value above which the earnings yield fell 3/4's of the time - was 5.5%. It is significant to note that the 4.3% earnings yield of the S&P 500 is below this value, meaning the market has been this richly valued less than 1 out of 4 times in the past.

The price-earnings ratio tells the same story of course. The median price-earnings ratio was 15.0 (it's currently 23 and change). Half the time the price-earnings ratio ranged between 10.6 and 18.3. It got as high as 35.4 (in April, 1999) and as low as 6.1.

So we know that the market is highly valued even after the recent declines and that high valuations have been highly correlated with low subsequent 10 year returns in the S&P 500.

OK, so how low is low?

Look at the last two columns, showing the total decade return (after inflation and including dividends) as well as the average annual return over those decades. A couple of things become evident:

• The median after-inflation return in the stock market was a phenomenal 155% total return per decade; this means that your money would have more than doubled over the median 10-year period after inflation; this underscores the tremendous wealth-building power of stocks;
• However, stocks did better 50% of the time and worse 50% of the time. Half the time, they returned between 46.4% on the low end and 239% on the high end, which works out to 3.9% - 13.0% per annum real return for each decade;
• Note also that stocks did have at least one losing 10-year period, losing as much as 35.4% of the starting capital (after inflation); this is why any method that helps us avoid or underweight stocks during historically poor periods should be of some utility (caveat: bonds and cash performed even worse during this high-inflation decade);
• Note that the best decade, for all its worth, returned a phenomenal 474% total real return, or 19.1% per year after inflation.

So how can we use this information? Let's assume that since we are in the lowest quartile of earnings yield we will achieve roughly what we would have achieved in the lowest quartile of S&P 500 decade-long average returns. This would work out to anywhere between -4.3% on the downside and 3.9% on the upside after inflation. Put another way: history would indicate that we should expect less than 4% a year from stocks after inflation going forward. If inflation runs at 3.5% a year, we would expect 7.5% per year return, best case. Not catastrophic, but certainly much worse than what we have experienced during the 90s.

Is there a more precise way to anticipate how much we should make in the market? Not really because there is a large portion of market return that is unaccounted for by any single variable. But let's try anyway.

The correlation coefficient tells us how much one variable's change is associated with another's. It does not give us a means of determining, predicting, or anticipating (whichever verb you prefer) the change in y given a value of x. To do this, we must perform linear regression.

	Earnings Yield:	PE Ratio:	Earnings Yield - 90 Day Treasury Yield:
Correlation with 1 year return:	0.260	(0.215)	0.215
For linear regression line: 10 year return = mx + b
m (10 year):	1.28	-0.95%	0.755
b (10 year):	-2.5%	20.8%	5.5%
R2 (10 year):	0.360	0.404	0.236

This is a busy table, but gives in a compact form all the information you need to hazard a guess as to the next decades average annual rate of return. The objective here should not be to make a precise prediction, but to give you some idea of what you can realistically respect from stocks going forward. If this return is significantly below what you could make by paying off a debt, for example, you might want to do that with any new money before investing in the stock market.

At any rate, if you look at the earnings yield, you see that it has a low .260 correlation with the next year's return; this underscores how long-term historical relationships can get out of whack in the short-term. One cannot say with much precision where the market will be in a year but ironically one can anticipate with some degree of accuracy what the average rate of return might be over the next decade. Why? Because markets will oscillate in the short-term, but settle down over the long-term, finding a certain equilibrium in their relationship to earnings.

Nevertheless, that relationship is far from perfect. The R2, a measure of how well the data points fit a line, is low for each of the variables above (1 would indicate a perfect fit with a line). This means that many other variables must explain returns or that a straight line is not the best curve to describe the relationship between the variables.

Nevertheless, let's look at the equations to discern what we can about the future:

Average annual 10 year return (after inflation) =

1.28 x earnings yield - 2.5%

-.0095 x PE + 20.8%

.755 x (earnings yield - 90 day Treasury bill yield) + 5.5%

The first two equations should be given greater weight.

So, for example, the S&P 500 currently has an earnings yield of 4.3%. This means we should anticipate a return over the next 10 years of :

1.28 x 4.3% - 2.5% = 3.0% per year after inflation;

This would grow a $100,000 portfolio to about $134,000 in a decade (after inflation, meaning that the dollar value would be greater, but the purchasing power in today's dollars would be about 34% more).

The earnings yield minus the 90 day Treasury bill yield gives a more optimistic picture: about 8% real, only slightly below average. However, the correlation coefficient and R2 are much lower for this variable.

Bottom line: we should temper our expectations for stock market returns going forward, and anticipate below average stock market returns. However, we should understand there is a large error component and that stocks may return much more or much less than these numbers. But all things being equal, returns should be lower than average.