Exams and Goodhart’s Law

I blogged recently about the difficulty of finding the right assessment system. Afterwards, in a throwaway remark on twitter, I said that finding the right assessment format was a bit like finding the right system for exchange rates – fiendishly difficult because you want one system to serve a multitude of purposes.

There are other ways in which I think economics and assessment face similar problems. Goodhart’s Law is a famous law that was invented by Charles Goodhart, a former Bank of England adviser. He observed that ‘as soon as the government attempts to regulate any particular set of financial assets, these become unreliable as indicators of economic trends.’ Extrapolating from that, you get the more general Goodhart’s Law: when a measure becomes a target it loses all value as a measure.

Exactly the same has happened with the 5 A*-C including English and Maths measure. Once upon a time it probably was a fairly reliable measure of how well a school was doing. Since it has become the government’s favourite target of how well a school is doing, it is now much less reliable as a measure. Schools have attempted to game the target using GCSE equivalencies, different exam boards and focussing their attention on C/D borderline pupils.

Campbell’s Law expresses the same point but in the field of social science. It was elaborated by the US social scientist Donald T Campbell and has been used in the US to explain the problems there with high-stakes testing. Campbell’s formulation is: ‘The more any quantitative social indicator (or even some qualitative indicator) is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.’

It’s for this reason that I think accountability measures should use more than one measure. I suggested a possible model for this a while back in this post and in this one. In the comments on the first post, Mr Chas made the following wonderful point:

Once you get to 4 or five measures, as the writer here says, you can ‘cheat’ on maybe one, but then as soon as you do it kicks the others way out of kilter. Like trying to squeeze into jeans a size too small. The fat has to go SOMEWHERE, it just pops up in a different place!

Of course, since we wrote that the government have adopted something similar – four key measures of accountability rather than one, plus an unspecified ‘destination measure’ to look at what pupils do after they leave school. I think this has to be an improvement on the previous system.

Advertisements

6 thoughts on “Exams and Goodhart’s Law

  1. chemistrypoet

    This effect can be termed ‘unintended consequences’, and applies anywhere there are targets and measures. Often targets are well meant, being a reduction of a very complicated situation and designed to ensure something bigger is achieved. But, there are always unintended consequences that mitigate against the desired outcome. In a business setting this is also prevalent, but there is usually a clear overall objective (making a sustained profit) that is very sensitive to unintended consequences (although this doesn’t prevent the target/measure/failure cycle). In education the overall objective is really not very clear (and very difficult to capture succinctly) and the whole process can notch up very big unintended consequences before they are spotted. There is inherent uncertainty.

    Reply
  2. logicalincrementalism

    It’s called sub-system optimization at the expense of systems optimization and is a well-known phenomenon in systems design. Unfortunately, the people designing public systems don’t appear familiar with it.

    Reply
  3. Broadcast Belle

    I remember watching and loving this very TED talk, then blogging about it afterwards too. I wish I had a bit more grit as a child especially when it came to maths, as I believed I couldn’t do anything when it came to numbers so always gave up. I love the idea of letting a child know that if they can’t do something, it’s only that they can’t do it ‘yet’.

    Reply

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s