Closed questions and higher order thinking

I know that Andrew Old often writes about the way that open questions are often, wrongly, seen as superior to closed questions – ie, it’s seen as being better to ask pupils questions that have lengthy answers and many possible answers rather than those that only have one straightforward right answer. I think one of the reasons for this is that open questions are seen as being effective promoters of higher-order thinking. That is, asking a question like ‘Why was there a war in Europe in 1914?’ is likely to lead to higher quality thinking than ‘When did Britain and France sign the Entente Cordiale?’

I would make two points here. First, it is true that closed questions are very effective at testing facts and knowledge. However, ‘higher order thinking’ such as analysing, evaluating and synthesising is in fact made up of ‘lower order’ facts and knowledge. To know why the First World War happened you have to have a very good knowledge of chronology and knowing when the Entente Cordiale was signed is an important, if small, part of that wider question. Indeed, the question ‘Why was there a war in Europe in 1914?’ requires a lot of facts to be able to answer successfully, and  one of the things that allows you to be able to analyse a deep, open question like this successfully is how many facts you have and how well you know them. This is as true of the academic level as the classroom level. I am currently reading Christopher Clark’s book on just this question. In the introduction, he notes that one of the problems with this topic is just how many facts have to be understood and marshalled, and that they are all in so many languages.  One of the things which makes Clark’s book so brilliant is that he has spent several decades mastering very many of these facts.  There is no way that a 20-year-old could write a book like this, however innately smart and innately good at analysing they were. Clark’s superb and revisionist analysis of the causes of the First World War springs from his detailed mastery of the facts. His excellent analysis is a function of his wide and deep knowledge.

Second, closed questions can in fact be extremely good at testing analysis and evaluation – in fact, they can often be more efficient and effective than open questions. For proof of this, take a look at the British Columbia leaving exam. (The link is to an absolutely excellent report by Common Core which features extracts from curriculums and exams from around the world.)

The BC leaving exam in history asks pupils to write essays, yes. But it also asks them to answer multiple choice questions. Here’s a typical example.

15. How did the Soviet totalitarian system under Stalin differ from that of Hitler and Mussolini?
A. It built up armed forces.
B. It took away human rights.
C. It made trade unions illegal.
D. It abolished private land ownership.

I think this is an excellent question which very definitely asks for higher order thinking. The reason why I think this question is so good is that it tests a finer gradation of understanding. Everyone knows the Nazis and Soviets were evil, and because they were evil, it is easy for pupils to just think that their regimes were the same. And of course the regimes were very similar. But they were different in interesting ways too, and this question probes that. A pupil who got this question right would have understood something important. A pupil who didn’t would have misunderstood something quite important.

And it isn’t just this question. I went through all of the literature and history multiple choice questions on this paper and all of them made me think hard about deep issues. I couldn’t have rote learned or memorised the answers to any of them. So I would say that the closed questions can be very effective at promoting higher order thinking. Of course, I wouldn’t suggest only having these types of questions – but as I said, the BC exam has a judicious mix of these and essay questions.

I know another technical criticism of MC questions is that it is too easy to guess them, but I think this criticism can be simply countered by having a penalty for wrong answers and/or a very high pass rate.

Update: I’ve since written two more posts on this topic, here and here.


13 thoughts on “Closed questions and higher order thinking

  1. Barry Naylor


    Fascinating topic, making me think. Having taught AP in US system I do believe it is possible to write valid and reliable test items testing higher order skills.

    “I think this is an excellent question which very definitely asks for higher order thinking.”

    I will have a guess having never studied either political system but having seen and read quite a bit about WW2, Cold War etc. My guess would be D but no great confidence.

    What makes you confident that answering the question quoted requires higher order thinking. Any chance you could explain briefly.


  2. Barry Naylor

    Didn’t mean that to seem rude. Sorry.

    I appreciate that you are very busy and time management etc. may dictate that you do not provide an individual reply.

    Have a great weekend.

  3. Pingback: Multiple choice questions, part two | The Wing to Heaven

  4. Pingback: Research on multiple choice questions | The Wing to Heaven

  5. Pingback: How could the curriculum be improved? | Ariadne's Thread

  6. Pingback: Closed questions and higher order thinking | The Wing to Heaven | Learning Curve

  7. Pingback: On Mitra’s “no holes in the wall” and the DT Willingham meme: weak research needs clever discourse to survive | Education: the sacred and the profane

  8. Andrew Lowery (@AndrewDLowery)

    Interesting. Re: The risk of ‘guessing’. You mention penalising wrong answers. Another interesting idea is linking with ‘confidence questioning’ (students can gain extra marks if they feel ‘confident’, but lose more marks if they are wrong on those Qs).


Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s