Par for the course
Glenys Stacey's speech to the HMC conference.
Good afternoon, I would like to thank you so much for inviting me to speak today.
I would like to start by winding back the clock twenty years to the mid-1990s when a rather crafty history student from Harvard convinced his department chairman that a year studying at St Andrews would help his academic career. It was apparently vital for him to travel 3,000 miles, to the charming town we find ourselves in today, in order to effectively complete his thesis. The subject of the thesis? The History of Golf.
The student, one Darren Kilfara, worked out that he could earn a year’s worth of university credit and – as a resident of St Andrew’s – become eligible for a year of unlimited play at the Royal and Ancient Golf Club of St Andrews, the birthplace of the game.
Mr Kilfara, who I probably don’t need to mention is an avid golf enthusiast, wrote a book on his adventures entitled “A Golfer’s Education”. Presumably this was separate to his thesis. The blurb describes it as the story of a man ‘who began his year at St Andrews as an intense, uptight golfer…and ended it a changed and wiser man’, who ‘overcame his obsession for scores and handicaps and learned to love the simplicity of the game’.
Sounds wonderful, doesn’t it? Being free of measurement and getting back to the reason you fell in love with a game, a subject or a purpose in the first place. I’m sure we can all identify with that concept. But while we could all take a leaf out of Mr Kilfara’s book every now and again, I suspect even he remained interested in whether or not he improved his game. In fact he did admit in an interview after the 2001 publication of his book: “There are a number of ways to enjoy the game, and as a single-digit handicapper who likes to tangibly measure his abilities, scorekeeping is one of those ways that suits me.”
There can be huge complexity in golf and education, yet the underlying objectives are also simple to understand. Such is often the way with many areas of expertise that those outside the field cannot easily perceive the level of thought, planning and care that goes into the outcome. I’ve not played golf, but I can appreciate the myriad influences that affect a player’s final score. Despite all the training and preparation there are factors that can influence the outcome that are outside their control – you can end up in the rough because of the way the wind is blowing. There are external factors that can affect exam results too, but those involved in assessment have a responsibility to identify and address them where they can. Exam boards have to deliver qualifications in which we all can have confidence and trust, and which are not dependent on the equivalent of the vagaries of the British weather.
Your chairman identified some factors concerning him earlier this week. In particular he was concerned about the quality of marking and suggested paying markers more. But we know from extensive research that most markers don’t do it for the money, much like I’m sure most of you did not choose a career in education for the money. Instead, most teachers become markers to get close to exam board specifications and mark schemes. It is not clear to me that paying markers more will of its own deliver improvement, although of course we don’t argue against this. What we do say, and argue for, is for exam boards to consider afresh how to secure a sufficient number of good quality markers, and to look particularly at whether employment arrangements can evolve to provide more certainty, and so that markers commit themselves to exam boards ahead of the marking season.
Your chairman was also concerned about item-based marking. But we know it has its part to play in good-quality marking. We have published strong evidence that it is at least as effective, if not more effective, than whole papers being marked by one individual. I realise this may not chime with a more intuitive view, but it is well-researched and evidenced.
Those are two issues on which we may not see eye-to-eye, but let me talk now of the relationship we at Ofqual wish to enjoy with HMC. It has developed in recent times: put simply the combination of our technical and regulatory expertise and the practice-based expertise and feedback HMC has allowed us, with exam boards, to improve qualifications and their delivery. To give you some examples in four areas:
- Our work on the characteristics of the marker workforce (published in February 2014).
- Our evaluation of the strengths and weaknesses of the current marking arrangements (published in February 2014).
- Our review of the shortcomings in the design of modern languages examination papers (published in September 2014).
- Our first annual review of summer marking and awarding, summer 2014 (published in December 2014).
We have already seen much progress in these areas. For example, we conducted a review into the design of modern language examination papers and implemented changes this summer such that those students who should get the higher grades in A level languages were more likely to do so. Exam boards were expected to reconsider their question and mark scheme design to differentiate better between students. We also required exam boards to ensure a wider spread of raw marks for the summer 2015 exams to better identify the most able candidates. I was encouraged that both HMC and ASCL welcomed these changes, but there is still more to be done.
Beyond those issues I have previously mentioned, your chairman also listed three things earlier this week that you would like to see addressed within the exams system.
The first was consistency across subjects and exam boards. Maths is an area where we have paid particular attention to comparability between exam boards. Earlier this year we conducted four strands of research looking at exam boards’ sample GCSE maths papers, their approaches to problem-solving and whether any differences could undermine the teaching, learning and assessment of the subject. The research confirmed that the new GCSE maths papers will be of greater challenge and exam boards were asked to refine their sample exam papers. These new papers are now exceptionally similar in terms of expected level of difficulty.
Secondly, Mr King called for a larger marker workforce and we welcome all efforts on your part to encourage good teachers to play their part in marking and awarding. Marker capacity and planning are being actively considered by exam boards and we will continue to engage with them to assure us of their ability to get the job done.
Thirdly, he talked about the need for greater confidence in, and transparency around, exam paper review and the appeals process. In contrast to golf where the score a player shoots over their round is absolute, for some exam questions there are valid differences of professional opinion and a judgement to be made. Of course there may be issues beyond differences of opinion. We know marking can never be completely free from error and we share a common belief that it needs to be as good as it can. I am sure you would also agree that where candidates believe they have been marked incorrectly, there needs to be access to a system to review and, if necessary, remedy any clear mistake, and the system should be swift, accessible and confidence inspiring.
With that firmly in mind, we conducted research earlier this year to compare the current process with three alternative methods of marking review. By way of background, exam boards received over 400,000 review requests last year. More than four out of five were within the original marking tolerance and less than 1% of all grades awarded overall were altered though this process. The alternatives we considered included single-blind and double-blind review. Intriguingly, none of them performed noticeably better than the current system. Further there would be significant additional costs and burden imposed by double marking – up to £10m we estimate. We found that in all four review processes, the average mark change across a number of different units was less than +1. In other words, the current review process performed as well, and in some cases better, than the others. We need to balance of course the prospect of incremental improvements to the system against the cost of introducing changes. I’ll come back to that in a moment.
As part of the same research, we also looked at whether reviewing marks sometime after the completion of exams, rather than immediately after results, changed outcomes. We found that markers were a little less likely to increase, and a lot more likely to decrease a mark after three months. The differences were very slight, but they do show that markers may be affected by knowledge of students’ likely proximity to a grade boundary and the immediate implications for the student.
So to return to my golfing metaphor, the challenges we face in the review and appeals process do not appear to be equivalent to a Scottish gale blowing in over the links, but to those who find themselves affected, the result is as frustrating. We know that the ‘human factor’ if you want to call it that, makes a difference, and is likely to affect all review processes in similar ways.
So where next? Given the evidence I have outlined, we have been working with the exam boards to look at enhancements that can be made to the current review process to counter unacceptable differences in marker judgement, while continuing to accommodate legitimate and expected differences in professional opinion. This may not seem radical – it is more evolution than revolution – but I hope you can see that it is being developed on the basis of strong evidence. We have already discussed our outline proposals with your leaders and we will be launching a consultation soon on how the appeals system might be significantly improved.
Marking and appeals are critically important - of course - but they are not our only interest. I hope you will be pleased to hear that we are poised to publish definitive work on another longstanding and seemingly intractable issue – inter-subject comparability. Your chairman has called for greater comparability: we want the discussion and we hope it is informed by our research and other work in this area that we will publish this autumn. There will be a debate to be had and we look forward to your engagement in it.
I’ll conclude by saying this is the last time I will have the pleasure of speaking at your annual conference, as I will be leaving Ofqual early next year having spent five years at the helm. And while I may have been inspired by our surroundings today, I do not anticipate taking up golf to fill my hours in the years to come. That said, I do know one aspect of the game that many of us all enjoy - the 19th hole - and I wish you a pleasant drinks reception this evening.
Last updated 13 November 2015 + show all updates
-
Typo correction
-
First published.