Month: June 2017

The Misuse of Correlation Part 2 – the results

In this post, I want to talk about an insidious error that creeps in with the usage of correlation in finance.

FT lexicon supports the idea that:

“a correlation is said to be positive if movements between the two variables are in the same direction and negative if it moves in the opposite direction.”

This definition is not unusual, commonly seen in finance textbooks.

Occasionally the formula may be presented:

But caveats in using the formula will likely be absent or at the least hidden from view.
By that, I mean the terms and are critically important but this importance is rarely appreciated. [1]

In the previous post, I asked you about the correlation of the changes of the two assets in the chart below:

a. Positive correlation

b. Negative correlation

c. They are uncorrelated

d. Not sure (be honest!)


The most obvious answer is of course b)
One line goes up and the other goes down so this means they have a negative correlation. This is unfortunately strictly incorrect if you paid attention to the instruction to consider the “changes in the two assets”.

A good answer is d)
Given the amount of information I had supplied, it’s a perfectly reasonable one.

Because another answer is a)
The correlation of the changes in the variables is +1, PERFECT POSITIVE correlation
and the lines are going in the OPPOSITE direction!!

(If you doubt this result please look at the data and calculations in the sheet attached (download) and use the CORREL function in excel.)

b) is an intuitive answer but a) is the answer that a financial analyst would calculate. If you imagine of situations where you are being given financial advice, it is clear there could be an immediate conflict!


First insidious confusion – the importance of the mean

If you have never seen this before, you may think I am lying or this is a convoluted trick. But it rests upon one key part of the calculation of correlation that is missing from virtually every definition I see, and is certainly missing from the vast bulk of work done by analysts in the finance industry.

The key is that correlation is calculated by looking at the relationship in deviations from the means (the terms and in the complicated mathematical equation).

In our example, the changes in the two variables in the chart have equal and opposite means, and so trend in different directions. However, the day to day volatility (deviation from the mean of the changes) is identical for both variables, and it is this term that drives the correlation whilst having no impact on the trend.

Here is a scatterplot of the % changes for each variable. Observe all the dots are distributed along the line – a perfect POSITIVE correlation.

This has a clear relationship to the way we think about the change in market prices of any asset:

In financial markets, the daily noise is usually much greater than the daily trend, and so forms the focus of most market commentary.

The key result is that if the noise term correlates for two assets, then they will correlate irrespective of their underlying trend, given the way correlation is calculated.
i.e. they could end up in very different places even if they are positively correlated!

Second insidious confusion – levels vs changes

The second insidious confusion can arise from a reference to correlation of the CHANGES or a correlation in the LEVELS of the two variables.

In financial markets, the method invariably used is to look at the changes in variables. In our example, we get the answer of positive 1 i.e. perfect positive correlation.

If we calculate the correlation using the levels or prices, we get an answer of -0.97
i.e. strong negative correlation

The intuitive result is the opposite of the result most likely to be calcuated by financial analysts.

Why does finance prefer the use the correlation of changes?

It is done for good reason. When you are looking at data with strong trends, as a lot of asset prices do, the correlation of levels can yield very strange results. Let’s take an example.

Let’s look at the US equity market (S&P 500 price – white line) and its PE ratio (orange line) over the last 30 years.

If we first look at the correlation of levels, we get a correlation of virtually zero.
This suggests a rather unintuitive result that there is no meaningful correlation between PE ratio and equity prices!

If we instead look at the correlation of changes, we get that there is a meaningful positive correlation of 0.78 which makes a lot more sense.

Conclusion

If these differences in the correlation results is were just some statistical fluke, from a couple of silly examples, then it would not matter.
But it is not an unusual result and it occurs when looking at the biggest and most commonly traded financial markets. It is therefore critical to avoid confusions such as these when thinking about what type of correlation to use or, more often, what someone else has used in the analysis you are reading.

[1] I very much enjoyed this paper by Francois-Serge Lhabitant which explains this issue very well. http://www.edhec-risk.com/edhec_publications/all_publications/RISKReview.2011-09-07.3757/attachments/EDHEC_Working_Paper_Correlation_vs_Trends_F.pdf

The misuse of Correlation Part 1 – Quick Refresher and Quiz

First, let’s refresh our memories of what correlation means.
This may seem very basic right now, but I would like to make sure the meaning is clear before we move on to its use.

I have included a question at the end, once you have read and thought about the definition:

  • A definition from the FT Lexicon:
    “a correlation is said to be positive if movements between the two variables are in the same direction and negative if it moves in the opposite direction.”
  • You can read examples in a number of sources such as

https://www.mathsisfun.com/data/correlation.html

and
http://www.bbc.co.uk/schools/gcsebitesize/maths/statistics/scatterdiagramsrev2.shtml

Here is a range of correlations, shown via a scatterplot:

Some important concepts

  • A positive correlation is “when the values increase together”
    An example would be temperature and ice cream sales as “warmer weather and higher sales go together”.
  • A negative correlation is “when one value increases and the other decreases”
    Note this is sometimes called an “inverse correlation”.
    An example would be weight of a car and its fuel efficiency as “cars that are heavier tend to get less miles per gallon.”
  • No correlation is when “there is no connection”. An example would be IQ and house number.”
  • For those of you with a more formal approach the mathematical formula for correlation is:
  • In practice, most of us find it much easier to use the function CORREL() in Excel!

Question time

Here is an example with two asset prices A and B. When we represent the data in a chart it can often be done in one of two ways.

This chart has two lines, showing how both the prices of asset A and B moved over time.

The other way to chart this is to put the prices of A and B on the two axes instead. It looks like this.

To make sure you have understood the basic concept of correlation, I would appreciate it if you could vote on an answer to the following question. (all anonymous of course!)

Career Tips

I was asked recently to speak at an undergraduate event. Part of it was to give some career advice in the form of 3 tips. Here is what I came up with:

Many people after leaving university find adjusting to the world of work difficult and become very unhappy. Focusing on a lack of “meaning” in their job while searching for a “mentor” to guide them, they can quickly come to resent their firm and co-workers.

It does not have to be this way.

The most important thing to realise is that the workplace is not going to feel like an extension of education – it is completely and fundamentally different. For at least the first two decades of your life, focusing on your knowledge and your skills is the key and the whole environment around you is geared to helping you develop. However, the ability of a student to successfully transition into a happy and productive career has remarkably little to do with the knowledge and skills they start with.

What really matters is how well they can change their mindset.

Here are 3 things to focus on:

  1. It’s not about you any more

This is the piece of advice students generally find the most upsetting. A big change in mindset is required to succeed in a work environment compared to the one needed for education.

In education, the student is the product. The ultimate aim for a student, with the help of teachers, is to gain the skills and knowledge required to pass exams. This does not mean that students have complete free rein to do what they want. There will be various restrictions on behaviour, such as a requirement to go to lectures, prepare for tutorials, do reading, problem sets and essays – however these are all designed with the success of the student in mind. The best attitude for the student is to be focused on themselves and their own needs.

In the workplace, the business is the product. The ultimate aim for a new employee is to become useful. Many graduates find this transition to the workplace a shock. Senior members of staff may not think that a key part of their role is to educate you and make you more productive or happy. In a few years’ time, you will also be more senior too and it will be obvious to you that this is not a priority either. You will want to be productive at work, impress your boss, get promoted, get a bonus etc.

Adjusting to this new reality, the best attitude is to be focused, not on yourself, but on the needs of the people around you and of the firm – Be useful! You will then find good things will start to happen to you. Given reciprocity (see “Influence: The Psychology of Persuasion” by Robert Cialdini), people you help will also help you. Senior people will start to spend time helping you learn and improve. You will have signalled to the firm that you have the right mentality to succeed and so will be promoted more quickly, paid more and given more training.

Having a real job is extremely helpful in preparing you for work and choosing a career path. I spent my Gap year working full time as an economist, but working at McDonalds may have perhaps been even better. You need to understand what it is like to be the other side of the counter.

  1. Be flexible.

In education, a targeted focus and narrow determination are extremely helpful for excelling with high results. The world of academia is fragmented and siloed, with status derived from expertise in ever more specialised areas.

The world of work is very different. A modern and successful career will come with many parallel and some orthogonal leaps into new areas, combined with an ability to master a broad range of cross-disciplinary problems.

I could easily have become a consultant or economist and I think I would have really enjoyed it and been successful. In banking and hedge funds, my career could have gone in lots of different directions. The only way to take opportunities is by being open minded.

  1. Work with people you would like to become.

This piece of advice was given to me as an undergraduate, and it has repeatedly proven itself true as my career developed.

Don’t think that you can join an Investment bank for the money and not become like them. Either you will change to fit in, or you will not and you will hate it and leave.

You must judge it from meeting real employees, not from impressions from TV shows. Being a lawyer is not the way it is on Suits just as being a Hedge Fund manager is not like Billions (well mostly anyway). That is why internships are so useful.

Conclusion

The world of work is can be a stimulating and fulfilling experience. For that to happen you need to be able to have the right mindset to take advantage of the opportunities on offer.

Money 4 – Why does it matter?

The elimination of money from economics theory and teaching leads to major practical problems.

  1. Why did we have the financial crisis and the prolonged recession?

The Queen famously asked why economists failed to see the crisis and ensuing recession coming. What is less talked about is how they subsequently also failed to understand a) what was happening as it was occurring and b) the nature of the recovery. Once you appreciate that money and credit are central to a modern economy, and academic macroeconomists were using models without money or credit, this failure is much easier to understand.

Some policy makers did a better job of learning and adapting to the crisis. Ben Bernanke, at the US Fed, with his study of the 1930s depression years, was well placed to support the economy once the crisis was underway. The Bank of England was not so well led. Mervyn King appeared to believe in a banking model in which the lender of last resort need not exist. When this model failed to have any correspondence to reality, he acted as though reality was at fault, not his personal model.

The financial crisis and its aftermath was predicted and understood by some people however.
But they were likely to be eclectic economists, on the fringes of the mainstream, who did not exclude the views of Keynes and Minsky for their lack of “microfoundations”.

  1. Why did the enormous monetary stimulus not lead to a stronger recovery?

The answer is that the monetary stimulus was not so enormous. The numbers were large, but the transmission mechanism was very weak, and therefore the recovery has been slower than most predicted.

Another misunderstanding follows, since the recovery has been slower than expected, new ideas have been sought to explain it away, such as secular stagnation. But once you accept the idea that QE is eye-catching, but not very powerful for the economy (it may be more powerful for asset prices but that is a different matter) then the slow recovery is not so surprising.

  1. Why do we ever have unemployment at all?

The academic models we have been looking at, theoretically make the existence of unemployment impossible. Given that this is evidently not the case, the models must be augmented with ad-hoc frictions, to make them have some connection to observed reality.

If money is allowed in the model at the start then you do not run into such issues, and surely this is evidence that the theories with don’t include it, don’t make much sense.

Why do economists believe these myths?

If an economist is typical pressed on this, responses vary from claiming that the representation is broadly accurate (it is not!) or more likely that it does not matter (it does!). If the assumption does not matter, why choose such a strange one?

A more recent defence has been that the latest batch of sophisticated new Keynesian models incorporate money and credit and a banking sector. But if that is the case why not change all the teaching? Why is money tacked onto the end of a model rather than incorporated as a critical building block?

I think that they attempt to tack money onto the end of their model building because it is not possible to incorporate it at the start. The assumptions which exclude money are critically important to the complex mathematical models that the current breed of academic economists revel in building. The worry for me is that armed with them, they go on to lead to key policy and market implications. It would also be fair to say that pretty much everything I do in studying the macroeconomy would not be classed as macroeconomics by a current mainstream academic.

Modern academic economists believe that conversations about macroeconomics should be based upon General Equilibrium (GE) and rational expectations and have “microfoundations”. The most recent iteration is the Dynamic Stochastic General Equilibrium (DSGE) model. GE is a truly majestic piece of mathematics which describes an economic system based upon essentially perfect barter.

The concept of money is added as purely commodity money. Any asset can be arbitrarily chosen as the denominator in which to price all others, it is just the numeraire. This helps with the solution as it reduces the number of independent variables by one when solving a set of simultaneous equations.

The advantage of building models in this way is that you can translate many concepts used in micro economics and apply them to macroeconomic questions. This is known as “microfoundations” and many Noble Prizes have been won, tying the neat General Equilibrium theory up with clever mathematics.

After the financial crisis, it is obvious that money and credit had to be included, and so the most recent batch of Neo-Keynesian models attempt to do so. But this is an ad hoc tacking on of a couple of new variables that do not connect to the central mechanism of the model. I see these models as sophisticated in the same vein as the geocentric models used to argue against Galileo.

If we use Kuhn’s model of paradigms, then this looks like economists trying to bury “anomalies” during a period of “model drift” when their models are increasingly unable to answer the questions people think matter. The next stage is “model crisis”. Or perhaps we are already there.

Relationship to Politics and Free-market thinking

This model creates the illusion of a perfect economy in which everything works, with the practicalities of reality being termed “imperfections” such as imperfect competition or sticky wages. This links strongly to the ideology of free markets being the answer to all questions i.e. the idea is to make reality behave more like the model.

Economists of a more interventionist or left-wing persuasion can exist within this paradigm. But ad hoc elements such as asymmetric information have to be added, combined with some pretty inventive and tortuous modelling, eventually producing models which suggest intervention is the correct policy response.

Conclusion

Recent mathematical models cannot be held responsible for the birth of the myths of money and banking. In Classical economics the concept of value is separate from money and logically prior to it and so JS Mill told us that “There cannot, in short, be intrinsically a more insignificant thing, in the economy of society, than money”.

We have recently seen stirrings from eminent economists that all is not well with the profession, https://piie.com/system/files/documents/pb16-11.pdf, but it is not yet filtering through to how the subject is being taught at grass roots.

Where we are left is a deeply divided set of disciplines. Practitioners, both in financial markets and many Central Bankers have a different approach to pure academics. But even academia is split between macroeconomists who study an economy without money and Finance professors who study a monetary system without an economy.

Both can be seen, to borrow a phrase from Keynes, as “an extraordinary example of how, starting with a mistake, a remorseless logician can end in bedlam”.