Different types of evidence

Stephen Twigg’s announcement of his plans for an Office of Educational Improvement has got a lot of people talking, including me, here. On Lib Dem Voice, Stephen Tall  welcomed the proposal, arguing that it would reduce the influence of unevidenced dogma and politicians’ pet whims. But I think that, on the contrary, the body as it is imagined by Twigg would do very little to reduce the amount of dogma in the education system.

This is because there are two types of evidence that are important in education. The first are econometric analyses – the sort of randomised control trials where you attempt to isolate a certain factor and measure its impact. This is the sort of evidence which we are fairly teeming with in the UK. I would include in this category research that tries to measure the impact of structural or systemic changes such as uniform, class sizes, teaching assistants and school structures. I would also include research that tries to measure the impact of teaching strategies – direct instruction, synthetic phonics, discovery learning, etc. These types of analyses can be very valuable, but they do have their flaws. One flaw is that of the Hawthorne effect – almost any change will have some sort of an impact. Another flaw is that in the social sciences, as opposed to the hard sciences, it’s much harder to isolate a particular factor. For example, if we run a trial showing that pupils who receive private tuition tend to do better in school, can we be sure that it is the impact of the private tuition that is leading them to do better? Or is the fact that these pupils are part of families who care very much about education and create a home environment conducive to study? Another flaw is that many of these studies will measure the success or otherwise of a factor based on whether it has improved GCSE grades. Now, given that GCSE grades have been increasing nationally at 1-2% a year for the past two decades, there are obviously problems with using this as a measurement.  I would also include in this category of evidence statistical analyses of different countries’ education systems, such as PISA.  And of course, the frequently noted flaw with such studies is that people tend to look at a high performing system and cherry pick certain aspects they were keen on already.

None of this is to say that this evidence isn’t valuable. For example, the bible of teaching strategy research is John Hattie’s Visible Learning, which is full of valuable insights about what does and doesn’t work in the classroom. This book is a meta-analysis of hundreds of pieces of educational research, and the large sample size obviously reduces some of the problems I noted above. Some of the techniques it recommends I used already. Some I didn’t. The advice it gives on feedback and how that can improve student performance is something that has really changed my practice for the better. Likewise, it’s extremely interesting and insightful to read about how things are done in other countries.

However, as useful as the above evidence is – and I do think it is useful – it is something I find much less useful than the other type of evidence I wish to consider. This is scientific evidence about how the brain works. Over the last thirty years, scientists have discovered more about how the brain works than ever before. We now have a fairly reliable working model of how the brain learns. This research has immense implications for classroom practice. Unfortunately, very little of it is known in this country. Indeed, so little of it is known that many schools up and down the country persist with teaching techniques such as VAK that are based on completely faulty scientific premises. More subtly, the entire design and structure of the curriculum rests on implicit assumptions that are scientifically dubious. One of these dubious assumptions is the idea that we can teach transferable skills. This assumption has had a very real impact in the classroom – it has led to programmes such as ‘learning to learn’, entire curriculums such as the Royal Society of Art’s Opening Minds, and wide-ranging cross-curricular project-based learning activities.

I would argue that this latter category of evidence is of a higher category than the former – or at least, individual examples of the former. What I mean by this is that this latter category of evidence is itself based on huge amounts of observation and research.  It is a scientific theory. If we find a piece of educational research that does a randomised trial of a learning to learn programme, for example, and it concludes that such a technique is extremely useful, then I think we are entitled to question that piece of research. Steven Weinberg puts it more eloquently:

“Medical research deals with problems that are so urgent and difficult that proposals of new cures often must be based on medical statistics without understanding how the cure works, but even if a new cure were suggested by experience with many patients, it would probably be met with scepticism if one could not see how it could possibly be explained reductively, in terms of sciences like biochemistry and cell biology. Suppose that a medical journal carried two articles reporting two different cures for scrofula: one by ingestion of chicken soup and the other by a king’s touch. Even if the statistical evidence presented for these two cures had equal weight, I think the medical community (and everyone else) would have very different reactions to the two articles. Regarding chicken soup, I think that most people would keep an open mind, reserving judgment until the cure could be confirmed by independent tests. Chicken soup is a complicated mixture of good things, and who knows what effect its contents might have on the mycobacteria that cause scrofula? On the other hand, whatever statistical evidence were offered to show that a king’s touch helps to cure scrofula, readers would tend to be very sceptical because they would see no way that such a cure could ever be explained reductively…How could it matter to a mycobacterium whether the person touching its host was properly crowned and anointed or the eldest son of the previous monarch?”

The other advantage of scientific evidence is that it allows us to see why something is successful, not just that it is successful. For example, Hattie’s meta-analyses show us that synthetic phonics are a very powerful method of teaching reading. But if that were the only evidence we had, then we would be entitled to wonder if that were just because the people using other methods hadn’t been trained very well in them, or if those other methods for some reason tended to be used disproportionately by less innately able pupils, or whatever. But the scientific evidence about how the brain works shows us that the success of synthetic phonics is not a fluke. Synthetic phonics is a successful method of teaching reading because it recognises the limitations of working memory and the importance of committing facts to long-term memory. It isn’t just a statistically plausible method of teaching reading – it is scientifically plausible too.

As well as being of a higher category, I would also argue that this latter type of evidence is more important for UK schools because it is the least-known category of evidence. There is a great deal of statistical and econometric evidence that is known and cited by policymakers and even teachers. Very little of the scientific evidence is known by policymakers and teachers. Twigg revealed this himself. In the very same speech where he was promoting evidence in education, he cited transferable skills as a policy which had a strong evidence base. In actual fact, the scientific evidence is extremely sceptical about the possibility of teaching transferable skills. My fear – and I think it is a well-founded fear – is that any Office of Educational Improvement would make Twigg’s mistake, except on a much grander scale.  It would inevitably be staffed by the sort of people who produce many of the econometric and statistical analyses at the moment, and the sort of people who have historically had quite a lot of impact within the education establishment. It would far more difficult, if not impossible, to staff a UK/English Office of this sort with experts in the latter category of scientific evidence. That is because, as I have said, there are very few experts on this in this country. Therefore, rather than institute an expensive central agency staffed with people who are already influential, I would suggest that advocates of evidence-based education policy should rally round this policy: the government should buy a copy of Dan Willingham’s ‘Why Don’t Students Like School?’ for every teacher in the country.


7 thoughts on “Different types of evidence

  1. teachingbattleground

    I’ve been thinking about this a lot lately.

    Partly, it is that there are questions over the reliability of the two types of evidence you cite (actually I would consider controlled experiments and large-scale, statistical studies to be different types of evidence, so make that three). Empirical studies in teaching are of notoriously low quality making it very difficult to know how much credence should be given to any of them and, in particular, what merit there is in unrandomised, unpublished or qualitative research. Results in cognitive psychology are far more likely to be reliable but there is still some reason to fear systematic bias in the discipline, particularly because experiments are so often performed on undergraduates at Western universities and because of some of the statistical techniques used.

    Then there are a couple of other kinds of evidence to be considered. The first is, I guess, possible to describe as “philosophical” evidence, basically the analysis of whether claims are coherent and concepts well-defined. I often suspect that a lot of claims made in education fall at this hurdle, and empirical research is pointless because the claims are not even good enough to be wrong. The second is the historical record. Has an idea been tried before, and if so did it produce the sort of results its current advocates claim it will produce? This is probably the hardest to be objective and rigorous about, yet it is often the first indicator that a supposedly new idea will not work.

  2. Pingback: Evidence and Learning Styles « The Wing to Heaven

  3. The Wing to Heaven Post author

    I agree with your point about ‘philosophical’ evidence. This is actually what I find to be the most persuasive argument against the whole word method of reading – it just doesn’t make sense logically. But that is why I worry so much about the econometric types of evidence. An econometric report with lots of statistics in it can give a faulty veneer of respectability to utter illogicality. I have blogged further about this here. http://wp.me/p24WgD-2a

  4. teachingbattleground

    I have similar worries. With enough variables, or enough restrictions on which part of the population you look at, it is always possible to find some correlation, a point made rather well here: http://xkcd.com/882/

    I have seen examples of evidence for the effectiveness of Charter Schools that use exactly this approach.

  5. Jonathan Whellan (@NottsAST)

    I think that the culture wars are starting to hit education in a big way. There has been a shift in recent years to use the language of maths and science to attempt to legitimise opinions and ideaology. The word evidence has become interchangeable with the term ‘someone writing something somewhere says…’ Throw some numbers into the mix (regardless of their validity or provenance) and people become a lot more cautious about querying the claim.

    It becomes a very real problem when people begin to mistake these suggestions and ideas for concrete truth. I get to observe a lot of teachers across a range of abilities, and to be honest if I watched someone using any techniques (even Brain Gym) that was able to get results (in an objective, non-survey way) then my response would be ‘Keep doing it.’ I know from talking to some people in the LEA, and HMI that they would take a similar approach. I also know many wouldn’t, and I would say it becomes a particular problem when viewing lessons outside of one’s own specialism, where the need to rely on a structured list becomes more important.

    This approach to judgements of teaching effectiveness would rely on long term data (exam results in other words) with all of the problems that would bring. It also relies heavily on the ability of individual teachers to make reasonable evaluations of their own work. I have heard many (and I’m sure I do the same thing) teachers make claims that a certain style works, and yet it is very difficult to strip out the many factors involved and make an empirical judgement on its effectiveness. I suspect that a decent (no thoughts on specifics yet) base of experience based tactics, pressure-tested in classrooms, would give a more reliable account than an empirical study that lacked the complicating variables of a classroom. For example, one paper I read (admittedly from the 70s) used an argument for ‘active learning’ that seemed to based on the vocabulary of an 8 year old; the paper suggested applying meaning to words helped understanding but neatly sidestepped the issue of acquiring the knowledge required to understand the words in the first place.

    I am more concerned currently with the move to a hazy relativism in teaching which places all subjects into the same melting pot and assumes that each subject can be approached in the same way. This is obvious in interpreting Bloom’s across different subjects; evaluate means many things to different people, and they bring their own ideas into classroom observations with them. Given that not only pupil’s education but also judgements about teacher’s capabilities and pay are being based on semantics I’m surprised more teachers aren’t getting angry about this.

  6. Pingback: Policy Based Evidence Making « Scenes From The Battleground

  7. Pingback: Some more thoughts on educational research – razorbladeinthecandyfloss

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s