Skip to content

Seven Myths: The evidence base, part I

February 7, 2014

In Seven Myths about Education, I make two claims: first, that in English education, a certain set of ideas about education are predominant; second, that these ideas are misguided. Finding the evidence to prove the second point was relatively straightforward. It is scientifically well-established that working memory is limited and that long-term memory plays a significant role in the human intellect. This has clear implications for classroom practice, implications which others have made and which I was happy to recap.

However, there was no one evidence base which could prove (or indeed, disprove) the first claim. Instead, I identified a range of different evidence bases to prove my point that in English education, the seven ideas I discuss are predominant. (In doing so, I also defined clearly what these ideas meant in theory and practice.) Here’s a recap.

• The writings of prominent theorists, and the proof that such theorists are indeed prominent (eg book sales, presence on government committees, the judgment of their peers, etc.)
• The advice given in popular teacher training textbooks, and the proof that such textbooks are popular (eg book sales, presence on reading lists)
• The National Curriculum
• Ofsted reports – I created an appendix of 228 exemplar lessons that were described in Ofsted subject reports from the last three years. These exemplar lessons were in turn drawn from the thousands of lesson observations done by Ofsted inspectors over the previous 3-5 years. See here for more information about this and a link to the appendix, which is available for free online.
• Popular non-governmental curriculums (and the evidence of such popularity, eg by number of schools applying it)
• Examples of lessons from popular resource sharing websites

I could have included a lot, lot more examples from the latter category which would have proved my point, and then some. I deliberately chose not to because it is hard to tell how popular or influential such lessons are. Had I used a lot of these, it would have been easier to accuse me of simply cherry picking the worst examples I could find on the entire internet. So I went easy on these, and only used a few examples from such websites, and then only from websites I could demonstrate were popular.

Given this range of evidence, I found it odd that some people criticised the book as being reliant on anecdotal evidence. As I say in the introduction, I do add in the odd anecdote to try and liven up the text, but only when I have clearly established that my anecdotal experience is in line with the evidence. In his review of my book, Tom Sherrington refers to my book being based on ‘personal anecdotal experience…from a very specific teaching situation’. That is absolutely not the case. It’s also particularly baffling that Tom Sherrington would say this given that the only counter-evidence he brings to bear is his own anecdotal experience. His strongest counter-argument is that ‘for me, the myths just don’t ring true’.

Interestingly, when I first started writing the book, my instinct was that my own personal anecdotal experience would bear little relation to the wider system, simply because I couldn’t believe that an entire system would have endorsed beliefs that were so completely at odds with all the available evidence. I actually set out planning to write a book that critiqued some parts and aspects of the education system. It was a genuine surprise to me to see that clear examples of bad practice were being endorsed as good practice on a system-wide level.

In a largely positive review, Michael Fordham did criticise my use of Ofsted reports as ‘a reliable and unproblematic account of pedagogy’. I think he is right to say that Ofsted reports are problematic. In the book I do in fact discuss the strengths and weaknesses of Ofsted reports, conceding that,

The one flaw with these reports is that Ofsted inspections are pre-announced. This means that teachers can, and do, put on a show for Ofsted. It has long been a complaint of many teachers that the kind of lesson Ofsted grade as outstanding is simply not possible to repeat consistently. So, when I give an example of an outstanding lesson from an Ofsted report, I do not mean to suggest that lessons exactly like this are going on all the time in every lesson. However, given the power we have seen Ofsted have, given the cottage industry of Ofsted preparation and given the fact that most schools will organise their own internal observation systems around Ofsted criteria, it is still fair to say that if Ofsted demand a certain type of lesson, that matters.

Also, I am not just using Ofsted reports as evidence for how teachers teach. I am also using them as evidence for how teachers are told to teach.

So, Ofsted’s inspection reports and subject reports are a fairly reliable guide to what actually happens in schools and a very reliable guide to what teachers are told to do.

Given that I am trying to prove what the dominant views are in the English education system, finding out how teachers are told to teach is just as important as finding out how they actually do teach.

Finding completely reliable and valid evidence of how teachers in England teach is always going to be difficult. Even a well-funded research study would run into methodological problems. Any large-scale observation programme, however low-stakes, would face the Hawthorne effect. However, whilst finding completely reliable and valid evidence is hard, I still think the Ofsted reports, backed up by the other sources listed above, offer a fairly reliable and valid picture.

Also, finding reliable and valid evidence of how teachers in England are told to teach is nothing like as methodologically difficult. Essentially, it is as straightforward as looking at what Ofsted and the government tell teachers. I would argue that even if teachers completely and utterly ignored this advice, the advice would still tell us something important about the education establishment. And in any case, there is good evidence that teachers and schools do not and can not ignore Ofsted. Since I first published the book, I think events have reconfirmed the central importance of Ofsted and their judgments in the education system.

I am always interested in discovering new sources of evidence about classroom practice. If people think there are other important sources I’ve missed, I would really like to know. In Tom Sherrington’s case, he seems to be suggesting that his own personal experience is a more valid and reliable source of evidence than the list I’ve outlined above. In Michael Fordham’s case, he doesn’t put forward any alternative. If anyone wants to suggest any others in the comments thread, I’d be glad. (Two very interesting sources which in the end I didn’t use were the PISA and TIMSS teacher, school and student surveys. These are really fascinating and have a lot of interesting data in them. Liz Truss and Laura McInerney, amongst others, have used these surveys to make interesting points. But they didn’t have the level of detail about classroom practice that I was looking for. TIMSS have done video surveys of some classrooms, but unfortunately not, so far as I know, of English classrooms.)

In any case, as I will go on to discuss in my next post, a lot of the arguments about the evidence base for my first claim are actually just a rather illogical smokescreen for people who are really interested in challenging my second claim.

7 Comments
  1. Fair point that I did not suggest alternatives and all of the methodological issues that you raise are good ones. I suspect that pupil work is probably one of the most valid measures we have of everyday pedagogy – the types of questions they answer, the problems they are set, the feedback given and so on. The IHR’s History in Education project (examining how history was taught over 20th century) made considerable use of pupil workbooks alongside interviews and inspection reports. See http://www.history.ac.uk/history-in-education/about.html.

    I think the Ofsted problem goes even further. We’ve all ‘put on’ Ofsted-friendly lessons for performance management observations and inspections, and, when I wrote those comments last year, I had in mind numerous conversations with colleagues about lessons were were planning that ‘we’d definitely not use in an inspection!’ The bigger problem, though, is the way the reports are constructed. I think the range of things inspectors comment on is actually quite narrow and they might miss out things which don’t fit in with their inspection framework. This is a link to a ‘good practice’ visit to my department in 2012 – http://bit.ly/1f78IRi. I rate the author (Mike Maddison) but even he uses very Ofsted-y language – I would have written the report in a completely different way and emphasised different things. I guess this is at heart a problem of rhetoric and reality – and how each affects the other.

  2. Hi Daisy. I was wondering if you had read my review post: I sent you a link at the time. I’m quite open about basing arguments on my personal experience. That’s all I have the resources for. I just think you fall into the classic research spiral of looking for things you want to find and characterising our system’s failing as being due to these myths is massively overblown. I’m suggesting that lots of weak practice I’ve witnessed has nothing to do with your myths and after teaching for 27 years, I’d expected them to, well, ‘ring true’ a bit more..,,but they don’t. Ultimately this is about values; you could never prove your point to me because we probably see education differently at a fundamental level.

    What disappoints me is that this could have been a persuasive book that urged teachers to consider thinking differently. For example, the power of memorising facts and using more drills. There is a really good case for that but it doesn’t need you to prove that the system is enveloped in myths…actually that’s more of a barrier. For me..at least.

    I’m as biased as anyone..I know that,,,but I haven’t written a book making a massive claim about the whole system. I don’t think I need to prove my point: I don’t recognise the myths as a characterisation of our system; that’s just an assertion based on working in several schools of different types for quite a long time.

  3. Hi Tom,
    Thanks for taking the time to respond. I’m happy for you to use anecdotal evidence, of course – I just want to make it clear that I don’t rely on this and I think your blog implied that I did. I think you are right to say I make a pretty big claim, and that is why I am so keen to restate the evidence base and make it clear that I don’t rely on anecdotal evidence. I’m sorry the book disappoints you and that you don’t find it persuasive, but I hope you did find at least some things about it useful, perhaps on memory and drills.

Trackbacks & Pingbacks

  1. Seven Myths – the evidence base, part III | The Wing to Heaven
  2. Daisy debunks myths: or does she? | logicalincrementalism
  3. Seven Myths: The Debate I Bunked | headguruteacher

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 2,854 other followers

%d bloggers like this: