Reform themes
Glenys Stacey's speech to the Westminster Education Forum.
Good afternoon.
The theme of your conference today is ‘England’s exam system – next steps for reform’. Given what has already been achieved and what is occurring at present, it is a little daunting to think about what else might make it on to the reform agenda in the years to come. Indeed, once the current programme of GCSE, AS and A levels reform is completed I know that most involved in the sector would appreciate a period of stability. To that end I hope you will allow me to leave the future to whoever succeeds me as Chief Regulator, while I take this opportunity to bring you up to speed with what has happened and what is underway. I understand my fellow presenters have covered a number of the key developments during this morning’s sessions, but I would like to add my thoughts in some regards. First, the quality of marking and the appeals process, to which I spoke at the HMC annual conference in St Andrews just last week.
They key points I made were threefold. That paying markers more does not necessarily guarantee improved quality of marking. We know from extensive research that most markers don’t do it for the money, much like most teachers did not choose a career in education for the money. Instead, most become markers to get close to exam board specifications and mark schemes. It is not clear to me that paying markers more would of its own deliver improvement, although of course we don’t argue against this. What we do say, and argue for, is for exam boards to consider afresh how to secure a sufficient number of good quality markers, and to look particularly at whether employment arrangements can evolve to provide more certainty, and so that markers commit themselves to exam boards ahead of the marking season.
There have also been calls for a larger marker workforce and we welcome all efforts to encourage good teachers to play their part in marking and awarding. Marker capacity and planning are being actively considered by exam boards and we will continue to engage with them to assure us of their ability to get the job done.
Further, there have been calls for greater confidence in, and transparency around, exam paper review and the appeals process. For some exam questions there are valid differences of professional opinion and a judgement to be made. Of course there may be issues beyond differences of opinion. We know marking can never be completely free from error and we share a common belief that it needs to be as good as it can. I am sure you would also agree that where candidates believe they have been marked incorrectly, there needs to be access to a system to review and, if necessary, remedy any clear mistake, and the system should be swift, accessible and confidence inspiring.
With that firmly in mind, we conducted research earlier this year to compare the current process with three alternative methods of marking review. By way of background, exam boards received over 400,000 review requests last year. More than four out of five were within the original marking tolerance and less than 1% of all grades awarded overall were altered through this process. The alternatives we considered included single-blind and double-blind review. Intriguingly, none of them performed noticeably better than the current system. Further there would be significant additional costs and burden imposed by double marking – up to £10m we estimate. We found that in all four review processes, the average mark change across a number of different units was less than +1. In other words, the current review process performed as well, and in some cases better, than the others. We need to balance of course the prospect of incremental improvements to the system against the cost of introducing changes. I’ll come back to that in a moment.
As part of the same research, we also looked at whether reviewing marks sometime after the completion of exams, rather than immediately after results, changed outcomes. We found that markers were a little less likely to increase, and a lot more likely to decrease a mark after three months. The differences were very slight, but they do show that markers may be affected by knowledge of students’ likely proximity to a grade boundary and the immediate implications for the student.
So if we know the ‘human factor’ if you want to call it that, makes a difference, and is likely to affect all review processes in similar ways, where next? Given the evidence I have outlined, we have been working with the exam boards to look at enhancements that can be made to the current review process to counter unacceptable differences in marker judgement, while continuing to accommodate legitimate and expected differences in professional opinion. This may not seem radical – it is more evolution than revolution – but I hope you can see that it is being developed on the basis of strong evidence. We have already discussed our outline proposals with the exam boards and other key stakeholders and we will be launching a consultation soon on how the appeals system might be significantly improved.
Your second session today was on the developing role of e-assessment and e-marking in schools. There is strong evidence to suggest that technology can enhance assessment. For example, it can help facilitate item-based marking in particular, which we know can be at least as effective, if not more effective, than whole papers being marked by one individual. We also know that exam boards are using technology to monitor marking quality. Seed questions can be inserted into materials marked online to help assess whether standards are being maintained. And exam boards can monitor when markers are actually marking and how long they are taking to do so. That might sound like ‘Big Brother is Watching You’, but it’s 2015 and it is exactly the sort of data that provides reassurance to us as the regulator.
Developments in technology typically go hand-in-hand with innovation. A risk for any regulator is that we get in the way of innovation, perhaps by setting rules that assume particular ways of doing things or prevent risk-taking. That’s why we have a legal duty to have regard to the desirability of innovation. And we take that duty seriously. Our aim is to allow and to enable awarding organisations to innovate, including in the use of technology. Our rules are therefore outcome-focused, looking at what qualifications are aiming to do and we do not make assumptions about how they might do it. And we encourage awarding organisations to tell us where we might be getting this wrong.
On the flip side, we cannot and should not try to force technology on awarding organisations: if there are well established approaches to assessment that remain valid and appropriate, it is not our job to try to stop that. The leadership for and the demand for innovation needs to come from elsewhere: from the market, from employers, from policy-makers, from funders. From those of you in this room.
Another of your sessions today was about tackling exam malpractice. An ongoing concern of ours relates to the pressure points in the exams system that can lead to malpractice, and maladministration for that matter. Over the past year or so we have been focussing our attention on areas that are within our control and seeking to influence others who can also have an impact.
Earlier this year we held a joint symposium with Oxford University on Teacher Ethics in Assessment. This brought together teaching leaders, teachers, academics and exam boards to discuss pressures faced by teachers, and what could be done to better support those facing ethical dilemmas. The symposium followed a survey we conducted in 2014 into practices known to be used by teachers to maximise exam results. The seminar involved and was welcomed by the sector. ASCL in particular has noted its desire to develop an ethical framework for teachers, with the symposium securing support for this work.
The survey and our symposium are two elements of a broader strategic approach we have adopted to drive improvement in this area. So as well as providing support to the teaching profession in addressing malpractice, we have been looking at what we can do to help reduce opportunities for and maximise deterrents against it. Our work has been progressing in a number of ways, encompassing prevention, investigation and action.
For example, in early 2014 we obtained commitments from the exam boards to help prevent malpractice occurring at source. These included implementing changes to the examination inspection service, and internal and external guidance and procedures. The boards also agreed to adopt a more consistent approach to decision-making about any sanctions where malpractice is proven.
Separately, in late 2014 we conducted an initial audit of other actions exam boards are taking to counter malpractice, including looking at the way in which they investigate and manage conflicts of interest when allegations of malpractice are made. In light of our work the exam boards have committed to improve their risk analysis and management of conflicts during investigations and improve the documentation used to support and record their decision-making.
These changes, along with the teaching profession’s own commitments, point to progress, and we’ll provide a formal report on all our work this December. I would like finally to update you on the progress that has been made in relation to GCSE, AS and A level reform in England. Last month saw the first wave of new qualifications being studied in schools and colleges, with further subjects being introduced next autumn and in 2017. In total, 38 new GCSEs are planned and 38 A levels - I should say that it is no more than coincidence that these two figures are the same. As commentators have pointed out, the number of available qualifications in future will be lower than in the past. Some 27 GCSEs will not be reformed, nor will another 27 AS and A levels - again, there is no science behind the similarity in these numbers. So why will there be fewer qualifications on offer in future?
In the past, exam boards were free to develop GCSE, AS and A levels in any subject. This led to a wide range of qualifications with a degree of overlap in some places. Following consultation, we adopted a principled approach, allowing reformed GCSE, AS and A levels in subjects that:
- are consistent with the primary purpose of a GCSE, AS or A level as defined in our ‘General Conditions of Recognition’
- are capable of securing a comparable level of challenge to other reformed subjects
- can secure valid assessment, including reliability and manageability
- can be assessed in a way that differentiates across the full range of ability
- are based on content that is sufficiently distinct from other subjects
This last factor accounts for 21 of the 54 qualifications I mentioned previously to not be reformed. Exams boards chose not to propose the development of new content in 26 cases. Of the remainder, we rejected an exam board’s proposal in 4 cases as we were not confident content could be developed. And in 3 cases the Department for Education determined that it would cease development of new content.
So the reduction is as much a result of decisions made by the exam boards themselves as it is by us as the regulator.
We are currently in the process of looking at exam board proposals for the development of 20 GCSEs and 11 AS and A levels for first teaching in September 2016, to complement accredited specifications in the 3 GCSEs and 13 AS and A levels that began to be taught last month. It is fair to say that accreditation of this year’s exam board specifications has been slower than we had hoped. We keep a running tally of those specifications that have been accredited on our website and it makes for scant reading at present. However, I am optimistic about progress over the next few months. We provide feedback to exam boards when their submissions are not accredited and they have strong incentives to revise them appropriately. Of course, we will continue to maintain our robust accreditation procedures. If you’re interested in learning more, I would direct you to a recent blog and postcard we produced on the subject
However, I would like to touch briefly on our accreditation process for GCSE science, as it has, like GCSE maths generated some media headlines. You may be aware that following our supplementary work on maths earlier this year we decided to undertake a similar, though not identical, research programme to support the accreditation of GCSE science subjects. As it turned out, the correlation between judged question difficulty and actual student performance on boards’ sample exam papers was lower than in our earlier maths study. We were always aware of that possibility, given that science papers typically have a much wider range of question types than maths papers. Nevertheless, the research was valuable: the results were provided to our accreditation panel and have prompted further discussion about the holistic judgements made in accreditation. We are now analysing the research findings further, to help us hone the approach for future studies.
Examples like this help to emphasise that reform has been, is, and will likely continue to be a challenging process. But I hope I have also left you with the clear impression that the rich package of changes currently being undertaken will leave our exams system in a stronger place than today.
Reform is very much a team effort. My colleagues and I play our part, but you all have your role to play too. So whether it’s developments in the quality of marking or appeals process, innovations in the use of technology, or finding new ways to tackle malpractice we are open to suggestion and receptive to innovation. To that end my colleagues who have been in attendance today have been taking notes of what you have had to say. We will use that feedback to inform the development of any future consultations that we may have on these topics, and we look forward to receiving your contributions as and when. As for my contribution to your conference today, I just have one more thing to say.
Thank you.