Research and analysis

Responding to Criticisms of the CASLO Approach (Report C)

Published 18 November 2024

Applies to England

Authors

Latoya Clarke, Paul Newton, Astera Brylka and Milja Curcin from Ofqual’s Standards, Research and Analysis directorate.

With thanks to

All of the teachers, students, employers and higher education professionals for their willingness and enthusiasm to participate in this research.

The CASLO Research Programme

This report is part of a series that arose from Ofqual’s 2020 to 2024 programme of research into the CASLO approach:

1. The CASLO Research Programme: Overview of research projects conducted between 2020 and 2024.
2. The CASLO Approach: A design template for vocational and technical qualifications.
3. How ‘CASLO’ Qualifications Work. (This was published in February 2022.)
4. Origins and Evolution of the CASLO Approach in England: The importance of outcomes and mastery when designing vocational and technical qualifications.
5. Responding to Criticisms of the CASLO Approach (Report A): A taxonomy of potential problems.
6. Responding to Criticisms of the CASLO Approach (Report B): Views from awarding organisations.
7. Responding to Criticisms of the CASLO Approach (Report C): Views from qualification stakeholders.
8. Responding to Criticisms of the CASLO Approach (Report D): Properties of qualifications from the CASLO research programme.
9. Understanding Qualification Design: Insights from the 2020 to 2024 CASLO qualification research programme.

Introduction

Vocational and technical qualifications in England have been subject to several reforms over the past 3 decades to increase their relevance, rigour and value. Despite significant changes across the landscape, there is a group of outcomes-based qualifications which have continued to exist. These qualifications adopt a specific approach to qualification design which we call the CASLO approach. Qualifications designed using this approach are recognised by 3 key characteristics (Newton and Lockyer, 2022):

1. the domain of learning is specified as a comprehensive set of learning outcomes, which tend to refer to elements of knowledge and skill
2. a standard is specified for each learning outcome, via a set of assessment criteria, which are used to judge student performances directly
3. a pass indicates that a student has acquired the full set of learning outcomes specified for the domain (mastery)

Since the emergence of the first CASLO qualification of national prominence, the National Vocational Qualification, the academic literature has been critical of qualifications which adopt the CASLO approach. The criticisms raised throughout the late 1980s through to the 2000s have been diverse in their nature and have targeted different aspects of CASLO qualifications, with suggestions that specific CASLO characteristics are responsible for a range of hard to resolve teaching, assessment and delivery problems which threaten the validity of these qualifications and their educational value.

To understand the relevance of the criticisms raised in the academic literature to CASLO qualifications today, Ofqual researchers developed a taxonomy of potential problems.^{[footnote 1]} We then invited awarding organisations in England to nominate a CASLO qualification which exemplified an important feature of the CASLO approach. A total of 14 awarding organisations nominated a single ‘exemplar’ CASLO qualification, with one awarding organisation nominating 2 qualifications. These awarding organisations participated in an interview with Ofqual researchers to describe the benefits of their CASLO qualification in addition to exploring whether they recognised criticisms described in the academic literature, and whether they put in place any mitigations to reduce the risks of potential problems occurring.^{[footnote 2]}

Discussions with awarding organisations revealed that some of the problems described in the literature are still relevant to CASLO qualifications today. However, for each potential problem, awarding organisations described mitigations and protective factors put in place to support validity, enhance impact and reduce the risks of potential problems arising in their qualifications. Mitigations and protective factors described by awarding organisations were not exclusive to specific problems. Instead, each mitigation often prevented the occurrence of multiple potential problems. Furthermore, mitigations were varied and included implementing support, guidance and quality assurance processes for centres in addition to promoting holistic contextualised learning and assessment within qualifications. Protective factors described by awarding organisations included the occupational and professional expertise of teachers, in addition to positive attitudes of students.

The findings from our study involving awarding organisations offer new insights into current CASLO qualifications, revealing that potential problems described over the past 2 decades in the academic literature have different degrees of relevancy to CASLO qualifications today. Whilst the qualifications nominated by awarding organisations are within the same family of qualifications (for instance, they all adopt the CASLO approach), the risks of potential problems arising and the approaches to reducing the risks of these problems occurring vary significantly across qualifications explored in this research. 

To further understand and build on insights gained from the awarding organisations who participated in our main study (report B), the present study triangulates their views on mitigations and protective factors for CASLO qualifications with those of 3 key stakeholder groups. Focusing on a subsample of 4 qualifications, this was intended to help to validate (or challenge) the views expressed by our awarding organisations. While the design of this study would not permit us to reach strong conclusions concerning the effectiveness of the proposed mitigations and protective factors, it represented a step in this direction, and avoided relying solely on awarding organisation views of their own qualifications.

Methods

Sample

Qualifications

This study focused on a sample of 4 (out of 15) qualifications nominated by awarding organisations in our main study. Two of these qualifications were classified as vocationally related qualifications at Level 3 (equivalent to Level 4 on the European Qualifications Framework) and 2 were occupational qualifications at Level 2 (equivalent to Level 3 on the European Qualifications Framework). The qualifications included in this study were:

1. NCFE CACHE Level 2 Certificate in Supporting Teaching and Learning (referred to as Teaching_Support_L2)
2. Pearson BTEC Level 3 National Extended Certificate in Business (referred to as Business_L3)
3. UAL Level 3 Diploma in Creative Practice: Art, Design & Communication (referred to as Creative_Practice_L3)
4. VTCT Level 2 Diploma in Women’s Hairdressing (referred to as Hairdressing_L2)

These qualifications are well-established with high certification levels relative to other vocational qualifications on offer. Other qualifications were considered, however, the lower-certificating ones would have had fewer participants available to sample from. In addition to this, views from the higher-certificating qualifications may produce findings that are more generalisable due to the scale of their use. Throughout this report, the qualifications are referred to using abbreviations rather than their full qualification names. These abbreviations are noted in brackets next to each qualification in the list above. 

The 4 qualifications in this study have been classified by the Ofqual research team according to their purpose. These purposes are:

1. ‘confirming competence’ qualifications which are largely delivered in the workplace and/or can lead directly to employment or to certify competence.
2. ‘dual purpose’ qualifications which are largely delivered in college settings and/or prepare students for, usually, entry-level employment roles or progression to higher levels of education.

The Business_L3 qualification and Creative_Practice_L3 qualification are classified as dual purpose qualifications and the Hairdressing_L2 and Teaching Support_L2 qualifications are ‘confirm competence’ qualifications. These classifications are not official Ofqual categorisations but have been established for our main study and this validation study to explore key differences between qualifications in our analysis. Other differences between qualifications in this sample included the inclusion of synoptic assessments in 2 out of 4 of the qualifications (Business_L3 and Creative_Practice_L3 qualifications). One qualification in this sample included non-CASLO externally assessed units (Business_L3 qualification).^{[footnote 3]}

Participants

This study included 3 stakeholder groups: 

1. Teachers delivering or assessing one of the 4 CASLO qualifications sampled for this project.
2. Students studying one of these CASLO qualifications.
3. Employers or Higher Education (HE) recruiters with direct experience of a student who had completed one of these CASLO qualifications.

To identify stakeholders with direct experience of 1 of the 4 CASLO qualifications sampled for this project, we adopted a bespoke approach to recruitment, as outlined below.

For the recruitment of teachers and students, we asked awarding organisations to share a brief overview of information about the study with all centres delivering the selected qualification. Centres then expressed their interest to participate in the study directly with the Ofqual research team. At the point of centres expressing their interest to take part, they nominated 1 teacher and where possible 1 student to participate in a focus group. The study information was shared by awarding organisations on 2 separate occasions (in December 2022 and January 2023).

For the recruitment of employers and HE recruiters, the Ofqual research team asked awarding organisations to share study information with employers who had experience of the qualification through working closely with the awarding organisation. The research team also asked participating teachers to share study information with employers and/or HE recruiters who had recruited students following the completion of their qualification, or to provide Ofqual researchers with employer details to contact them directly. Ofqual researchers also contacted employers and HE groups who through their website had explicitly stated that they recruit students who had completed this qualification and invited them to participate in the study.

Once participants had expressed their interest, their details were recorded, focus group dates were scheduled and reminder emails were sent to participants at 2 time points (one week before and the day before the focus groups). A waiting list was also operated when more than 8 participants expressed an interest in participating in focus groups on a specified date.

We aimed to recruit a total of 96 participants, or 24 participants for each qualification, consisting of 8 teachers, 8 students and 8 employers or HE recruiters. However, despite awarding organisations distributing study information to all of their centres and employer or HE contacts at 2 separate timepoints, it was challenging to recruit our target of 96 participants. Consequently, this study included a total of 57 participants across the 4 qualifications. This included 34 teachers, 17 students, 3 employers and 3 HE recruiters across the 4 qualifications.

Chart 1. Stakeholders across qualifications

Data collection

We conducted 12 focus groups and 4 semi-structured interviews between 8 March 2023 and 31 March 2023. Each focus group or interview was held online (via Microsoft Teams) with participants from across the country. To facilitate focused discussions, each focus group included a single stakeholder group (for example, student, teacher, employer or HE recruiter) and a single qualification of interest.

Typically, each discussion included 2 Ofqual staff and between 2 and 6 stakeholders and lasted for between 33 minutes and 1 hour 23 minutes in duration (the mean duration was 96 minutes). 

All participants received an information sheet and informed consent form prior to participating in their focus group or interview. All students were aged over 16 and so no parental consent was required for this stakeholder group. Focus groups and interviews were all audio recorded (with participant consent) and transcribed verbatim for analysis. Participants did not receive any financial incentives for taking part in this study.

Focus group questions were developed to broadly reflect questions asked in the main study. Participants were firstly asked more generally about what works well or not so well in teaching, assessment and delivery of their CASLO qualifications, followed by qualification specific prompts. The prompts were designed to gather stakeholder perspectives on mitigations and protective factors described by awarding organisations. The approach to designing focus group questions was carefully considered by the research team to ensure that overarching questions were sufficiently broad to encourage participants to describe important aspects of teaching, learning and assessment. This was balanced with prompts which were highly focused for each focus group to elicit different user perspectives of specific mitigations and protective factors described by awarding organisations. It would not have been feasible to have asked stakeholders to respond directly to all of the mitigations described by awarding organisations without large amounts of resource or a common understanding of the problems raised in the academic literature. As a result of this, the questions asked were sometimes indirect rather than referring directly to all mitigations described by awarding organisations (see the appendix for focus group schedules, which include focus group questions but not mitigation specific prompts which were bespoke to each focus group).

Analysis approach

This study adopted a thematic approach to analysis with all data analysed using the qualitative analysis software package NVivo. The analysis was conducted in line with guidance published by Braun and Clark (2006) and Nowell, Norris, White and Moules (2017) in 3 separate phases:

1. An initial coding framework was developed based on focus groups and interviews across stakeholder groups. This framework was tested by researchers on one transcript from each of the 4 stakeholder groups.
2. Once the coding framework had been piloted, there were discussions between researchers to assess agreement and consistency during the coding process. The approach to coding passages of text, including the double coding of text and coding passages of text rather than single sentences was agreed on. Researchers further refined the coding framework to ensure it captured themes from across the various stakeholder groups.
3. The final stage involved coding all transcripts from across the 3 stakeholder groups.

Phases 1 and 2 were particularly important for ensuring that stakeholder views were represented in the framework and were not concealed through overlaying the data onto the existing framework, developed as part of our main study involving awarding organisations. 

The decision was made to analyse and report on the collective views and experiences of all stakeholders (teachers, students, employers and HE recruiters) in a single report. This allowed for the comparing and contrasting of perspectives across qualifications and groups, in addition to building a broader picture of user perceptions of mitigations and protective factors described by awarding organisations. This approach also facilitated the recognition of subtle differences across the sample, as compared to exploring the qualifications and groups in isolation. The themes highlighted in this report are explicitly linked to themes identified in our report of research with awarding organisations (report B) to serve the purpose of triangulating awarding organisation views with those of stakeholders.

This report includes direct quotes taken from focus group and interview discussions. Longer quotes are presented separately from the main text (and marked with a line to the right) and are attributed to a specific stakeholder group and qualification, for example, Student_Business_L3. Shorter quotes are placed in quotation marks within the text and are not always attributed to specific stakeholders or qualifications. They are included within the main body of text to illustrate key points made by participants.

Perceived benefits of the CASLO model

It was important to hear about the aspects of the CASLO model which were most important to stakeholders as users of the 4 qualifications sampled in this study. We explored their experiences through asking broad questions about teaching, assessment and delivery alongside specific questions related to mitigations and protective factors described by awarding organisations. Through exploring stakeholder experiences in this way, we aimed to further understand user perspectives of mitigations and protective factors described by awarding organisations in our main study.

In this section of the report, we will share reflections generated from discussions with stakeholders, which are specific to the CASLO approach rather than qualifications more generally. For example, in this report, we did not include comments related to centre level grades issued by teachers to boost engagement and motivation. Whilst there were many broader discussions explored within focus groups, the purpose of this report was to validate awarding organisation views on CASLO specific issues. 

Most of the views related to CASLO specific features came from discussions with teachers and students rather than those with employers and higher education recruiters. This might be a consequence of the proximity of teachers and students to teaching, learning and assessment within the qualifications sampled for this project, but will also reflect the smaller sample size of the employer and HE recruiter group involved in this study. Therefore, the views of teachers and students are most prominent throughout the results presented in this report. 

We will start this section by exploring the perceived benefits of the CASLO model, with a focus on 3 distinct mechanisms. These are transparency, flexibility and mastery learning and assessment. The views shared by participants in relation to these mechanisms largely align with the benefits described by awarding organisations in our main study. Whilst there is some overlap in the views shared, there are also nuanced perspectives. These are explored in this section.

Transparency

In our main study, awarding organisations emphasised the importance of transparency within their qualification. They explained that transparency played a key role in building student engagement, instilling confidence and supporting the development of knowledge and skills. 

When exploring what works well in teaching, learning and assessment in stakeholder interviews and focus groups, transparency also emerged as a recurring theme. Several teachers and students explained that learning outcomes and assessment criteria provided a level of clarity surrounding requirements for teaching and assessment. Furthermore, several participants commented that continuous support and feedback throughout the programme of study builds in transparency for students. This subsection explores this theme of transparency.

Transparency of the learning journey through support and feedback

The CASLO approach has been characterised as an integrated teaching-learning-assessment system (Newton & Lockyer, 2022). This integrated system, supported by learning outcomes which are designed to be transparent, is perceived to provide users with clarity in relation to the learning journey. For many students in this study, continuous feedback within classroom-based activities and formative and summative assessments was critical to understanding gaps in knowledge and how to demonstrate assessment criteria. Feedback and support based on learning outcomes came directly from teachers, both verbally and in written form. It was often described as being specific, detailed and personalised enough to be meaningful. Participants also described valuable peer to peer feedback and support related to qualification content, as well as support in developing study skills, such as referencing.

But then also talking about feedback, I love how specific it is, you know exactly which areas you need to improve on and you know where you’re doing well, so it gives you that all-round picture of how you’re doing, compared to an A-level where you’ll just sit an exam paper and you’ll just get a grade in a red circle. It’s a lot more, it gives you much more to improve on and I’ve mentioned before how I love the course and how it’s good for moving you into the workplace in a practical sense and I like how the feedback is set out and I think that’s similar to how you’d be given feedback at work on projects you’re completing and through appraisals, I feel like it’s all very similar, which is a good thing. Student_Business_L3

Transparency was not only about the quality of feedback or the person providing the feedback, but its timing was also an important factor recognised by many participants. Many Hairdressing_L2 students received feedback which was “pretty instant” and they could, therefore, use it to immediately set necessary improvements in motion. In contrast to feedback that would be received at later timepoints and, thus, be potentially less effective, instant feedback enabled students to perform better in assessments that followed. Continuous assessments and accompanying feedback also allowed progress to be tracked, giving students a clear sense of where they were in their learning journey. 

I think something else that’s really valuable is that it’s pretty instant that these students get the feedback and because it’s so instant, it then means that they can go away and work on it, whether it’s what they need to do well or what they didn’t do as well and I think it’s far more valuable than waiting three months or two  months. You need to know quickly how to improve and I think that’s a really, really, really positive attribute about this qualification. Teacher_Hairdressing_L2

Teachers in this study also spoke about how, in their views, transparency in their qualification increases when clear feedback from awarding organisations is received. Some teachers candidly described the benefits of requesting support from nominated awarding organisation leads on specific assessment related queries, whilst others praised routine feedback resulting from the quality assurance processes. In both instances, feedback provided teachers with direction, recommendations or actions which could be used to support teaching and assessment practices. It is important to note that the extent of transparency from awarding organisations was found to be mixed within and across qualifications, with some teachers indicating that they would value additional feedback. 

Transparency of assessment requirements through assessment criteria and grading thresholds

A large number of teachers and students in this study explained that assessment criteria provided a level of transparency through listing all knowledge and skills assessed in their qualification. For teachers, the criteria often served as an important reference point for guiding students through the assessment process. In addition to this, teachers explained that assessment criteria supported them in devising assessment materials and making judgements. For this reason, many teachers considered assessment criteria to be both central to their qualification and a useful and supportive tool. 

Interestingly, teachers spoke explicitly about the transparency of assessment criteria being a direct result of the way that the criteria are written. For instance, Creative_Practice_L3 teachers shared that the “broadness” of assessment criteria, combined with a small number of units and simple language used in the specification, made these criteria easier to digest and use across different pathways within the qualification. Whilst for Hairdressing_L2 teachers, the prescriptiveness of the performance criteria resulted in enhanced transparency.  

Yeah it’s really simple. Yeah and because there’s not loads of units, there’s only a few units, you know, if I go and assess the final project that’s one unit so it’s easy to go through and the language is simple to understand. So it’s really easy to assess. Teacher_Creative_Practice_L3

I think it’s really, really useful for the students […] having the performance criteria absolutely mapped out, because, one, they can understand what they need to do, but, two, they can also understand where they went wrong. So, I think that’s useful, not just to us as assessors, but I think that’s a really useful one for students to be able to reflect on, whereas maybe other courses or other qualifications aren’t maybe quite as prescriptive, but I think they can really see what they need that way, which is good. Teacher_Hairdressing_L2 

Across focus group discussions, students and teachers explained that assessment criteria often needed decoding before the requirements could be fully understood by students. However, once this process of explanation and familiarisation with the content, structure and language of the criteria had taken place, students valued having a single point of reference for completed, current and upcoming assessments. Students commented that, from their perspective, assessment criteria introduced more fairness and transparency, and removed any ‘guesswork’ from the assessment process. It also promoted student independence and agency. 

Ours was all broken down into criteria. We was given the sheet of what was going to be expected from us for each unit so we knew exactly what we needed to do for ours, so ours was explained really clearly and it stopped you panicking and going off topic as well, we could stick to it. Student_Teaching_Support_L2

An additional aspect of transparency raised across several focus group discussions related to the performance standards set out in assessment criteria. Hairdressing_L2 teachers explained that when training as an assessor and when assessing student performance against assessment criteria, standards explicitly described in criteria enabled them to be cognisant of the difference between their own personal standard and the qualification standard. 

I think you have to be careful not to assess them on what your standard is, you have to be careful. You have to look at the criteria and sometimes think, oh, if that was me, I don’t think that’s good enough, but actually [the student is] meeting the standard. Teacher_Hairdressing_L2 

For some teachers, the awareness of the discrepancies between one’s personal standard and the qualification standard seems also to have influenced teaching practices. This was reflected in one teacher’s comment when they spoke about the importance of ensuring that the same tutor taught across all relevant units, to support consistency in the technical skills instilled in students and, therefore, the standards demonstrated during assessments. 

Flexibility

In the main study, awarding organisations described prioritising flexibility when designing the 4 qualifications sampled in this study, primarily through ensuring learning outcomes are written in a sufficiently broad way to facilitate tailoring to specific student interests and contexts. This flexibility was viewed by awarding organisations to be essential to the effectiveness of qualifications due to the diverse contexts that qualifications operate within and the broad range of students studying them. A theme of flexibility also emerged strongly from the focus groups and interviews conducted with participants in this study. Despite not being asked directly about flexibility, stakeholders described flexibility in teaching, learning and assessment.

It is important to mention that these flexibilities were described to differing extents across the 4 qualifications. This subsection will explore the themes emerging from discussions surrounding flexibility in this sample.

Flexible teaching and learning through flexible qualification delivery and design

All CASLO qualifications are characterised by a comprehensive set of learning outcomes, which tend to refer to elements of knowledge and skill that need to be learned on the course. In this study, the broadness of learning outcomes facilitated a range of different approaches to delivery. Teachers described delivery programmes which operated on a full time and part time basis, some described rolling rather than fixed entry point programmes, whilst others referred to delivering accelerated programmes based on learner needs and circumstances.

One of the selling points that we have over one of the other colleges locally in the area is we offer the courses in the evening online, we’re not asking them to come out of their working day and we have a roll on, roll off provision and we can do one-to-one learning as well if people can’t come to those, you know, assessors. We adapt the learning to meet their individual needs really. Teacher_Teaching_Support_L2

More generally, there was some recognition that broadly defined learning outcomes supported the tailoring of teaching programmes to a range of contexts and learner interests. Several participants from the Creative_Practice_L3 qualification commented that broad learning outcomes could be tailored to the pathway that students were studying (for example, photography, fine art, fashion or textiles), making the content of the qualification more malleable at centre level. There was “diversity” and “freedom” in what could be delivered, as long as centres had the required facilities available and providing teachers could “tie [learning] to the criteria”. In addition to this, human resources in the form of teaching staff and relationships with industry experts also influenced the degree of tailoring in the 4 qualifications. 

Obviously you look at the criteria first. You’ve got to hit that. It’s just the area that we’re working in. […]  What skills do we need, what facilities have we got, what kind of resources have we got, and try to make it innovative. And try to make it kind of relevant in terms of the ambitions of the students in terms of university and in terms of employment or apprenticeship, that kind of stuff, but you’ve got to make it fun as well haven’t you. It’s got to be fun. Teacher_Creative_Practice_L3

Several teachers explained that it was a strength of CASLO qualifications to have the “freedom” to construct teaching programmes which could be contextualised to their local situation. Teachers described a focus on ensuring that teaching and learning included contemporary issues and local industry professionals as reference points. To support this level of contextualisation and personalisation in teaching and learning, teachers recognised underlying themes in the qualification which were often broad enough to be interpreted in different ways and explored at different points in the course. This point is highlighted in the quote below:

I think there’s some underlying themes within the UAL curriculum that have got to be explored. So, for example, equality and diversity, sustainability, preparation for progression, they’re some of the key things. But again there’s flexibility to move that around where it fits best within your own curriculum and scheme of work, which I really like. For example […] we do a lot of work on sustainability around the marine environment, it just suits us really well, but we have the flexibility to interpret that as we want to. This year I’m collaborating with [a local organisation] and [a national charity] so it gives us the ability to go out there and look at what’s going on in the real world, find partners to do external live briefs with us. And I just love the fact that, you know, if an idea like that pops up we’ve got the freedom to be able to pursue that and be able to bring that into the curriculum. Teacher_Creative_Practice_L3

A further flexibility supported by broadly written learning outcomes and the unitised nature of CASLO qualifications was the scope for teachers to sequence units according to the needs of their students and their local setting. The unitised nature of qualifications allowed teachers to consider a range of factors when deciding on the order of unit delivery. For instance, teachers explained that completing non-CASLO externally assessed units earlier rather than later in the Business_L3 course allowed students to access resubmission opportunities, if needed. In the Teaching_Support_L2 qualification, teachers prioritised in their delivery these units that involved knowledge and skills students would need during placements. Hairdressing_L2 teachers described taking a front-loaded approach to teaching and learning, to allow for more time later on in the course for practising and embedding key skills. Other teachers considered how easy it would be to engage with various units at different stages during the course, based on the level of challenge within the unit. Importantly, across students there was a level of recognition that teachers delivered units in an order that ensured a cohesive teaching and learning experience. 

Interestingly, our tutor didn’t do all the units in order, so they didn’t do them 1 to 9, or 1 to 10 I think we’re on, so I think she found a better way to link them, like a jigsaw and maybe you’d want to do the corners first, for example, and I think it’s worked so far. Student_Teaching_Support_L2

Flexible assessment

The theme of flexibility was recognisable in focus group discussions surrounding assessments in all the 4 qualifications sampled in this study. Flexibility in assessments appeared to be facilitated by assessment criteria written in a sufficiently broad way to allow assessments to be tailored to student needs and interests. Students from across the 4 qualifications shared their experiences of shaping “rather than having really set briefs”. For instance, one student provided an example of a brief requiring the demonstration of “ICT numeracy, writing and maths” as part of an observation in the Teaching_ Support_L2 qualification. The student was able to conduct a lesson involving times tables, integrate a song video, play a game and set a worksheet to meet this brief. These individual elements were decided by the student, highlighting the flexibility aspect of the assessment, whilst also meeting the requirements of the brief. There were several comments suggesting that the broadness of assessment briefs promoted engagement in the assessment process, as the more granular aspects of the brief could be decided by each student. 

Flexibility in assessments also appeared to be facilitated by assessment criteria written in a sufficiently broad way to allow teachers to assess student work with a method they felt was most appropriate. Across the 4 qualifications, teachers described using an assortment of assessment methods, such as portfolio-based assessments, observations, written assignments, oral questioning, and examinations. Again, the broadness of assessment criteria often enabled teachers to use their professional expertise to assess students using the most engaging and valid methods of assessment for their setting. This was reflected by Teaching_Support_L2 teachers who recognised that written assessments were not found to be the most engaging for many students. As a result, knowledge and skills were often assessed through a variety of practical assessment tasks over time. 

I do lots of practical activities with them and the way I get round, we do the Play-Doh and the cornflour and so to ease them in so that they know it’s not all weeks and weeks of writing, we’re going to go out and do risk assessments, we do displays in the classroom, we’ve just done a lovely Easter display, spring display. So, there’s lots of other ways you can get your evidence, rather than just all the formal writing, so I think once they know that they’ve got options, add pictures, add colour, add music, it’s a lot softer, so that works with my learners. Teacher_Teaching_Support_L2

One of the things I really, really adore is, particularly for my SEN students, the support, it’s much more flexible, in that I can assess in lots of different ways. To put some of those students under a final exam, it really isn’t particularly robust or particularly valid, I don’t feel, because you can literally, they can just go to pieces on the day, or things can be open to interpretation. Teacher_Hairdressing_L2

Flexibility not only emerged in relation to how students are assessed but was also recognised in relation to when students are assessed. Students described the benefits of being assessed when ready within their qualification. For instance, in the Hairdressing_L2 qualification, some participants described a student led process whereby students indicate their readiness for assessment (for instance, through a traffic light system). This was particularly useful for assessments taking place in real-world settings, as students could ensure that they could demonstrate the skills needed for the assessment whilst having the appropriate conditions to complete the required performance. In these instances, each student was aware of the qualification assessment criteria and was, therefore, able to take ownership in the assessment process. Similar positive sentiments were shared in the Teaching_Support_L2 qualification, as students explained that they had some input in relation to the timing of their observations.  

Yeah we have like some, I’ve forgotten what they’re called, like labels if you’ll, they’ve got green, amber and red. So when you’re happy and feel confident and you want to be assessed you put it up on your board, and you say to your teacher that, she’ll see on the board which one you want to do, if you want to do a formative or a summative, and it’s basically down to your own confidence when you feel ready. Student_Hairdressing_L2

The option to complete assessments over several days or weeks was also raised as a beneficial aspect of assessment within this sample. This was evident in the Business_L3 qualification, as participants highlighted that one non-CASLO externally assessed synoptic unit was carried out over 2 days.^{[footnote 4]} Similarly, teachers from the Creative_Practice_L3 qualification explained that students have the opportunity to complete their final synoptic assessment over several weeks:

The final major project, is an examination situation really with flexibility built into it. So it is, you know, it is run on the similar lines to coursework and the students are given a problem that they’ve got to solve over a period of weeks, but there is that feeling that there is, you know, it is going to be submitted to the board to be moderated. And so there is still that feeling that there is, it’s just a different form of examination, not sitting down and writing about something. I think that’s important that students are able to experience that. Teacher_Creative_Practice_L3

Completing assessments over days or weeks, rather than in a single sitting, which is typical of a general qualification, was viewed as important for adding validity to the assessment process. Teachers explained that the “stress and pressure” to perform in a single examination is often removed when students have the “time and space to develop and build and create at their own pace”. This was perceived to be especially important in the Creative_Practice_L3 qualification, as it mirrored the creative industry where “thinking and reflecting” was described as fundamental to the creativity process.

Contextualisation of learning and assessment

Learning outcomes written at a sufficiently high level of generality to support contextualisation of learning activities to local contexts and industry emerged as a consistent and important theme in this study. Teachers and students described benefiting from a range of scenario-based learning activities and industry facing activities such as placements, day trips to relevant local businesses and workshops involving external guest speakers. Students expressed that contextualised learning was often introduced at different points in their course and was critical to bringing subject content to life and connecting the qualification teaching to the industries that they were preparing to enter. Many students emphasised the benefits of linking theoretical or hypothetical concepts to concrete and local sector specific examples. For instance, participants described live workshops involving public facing exhibitions in the Creative_Practice_L3 qualification and marketing campaigns responding to the Covid-19 pandemic in the Business_L3 qualification. 

Whilst assessments needed to align with learning outcomes and assessment criteria, the autonomy and scope to shape assessment briefs was viewed as engaging for teachers and students alike. For some teachers, developing assessment tasks and events for students as part of a teaching team introduced an element of professional development and collaboration. Discussions amongst teachers surrounding the process of devising assessment briefs highlighted their careful considerations of learning outcomes (and thus skills and content that students should learn), cross-referencing with qualification specifications, and integrating student interests and pathways. 

So I really enjoyed being able to write my briefs myself about stuff that I thought would be really engaging for students, and also include contemporary practitioners, and bringing all that together I think is really empowering for you as a teacher and for your team. Everybody’s contributing to that as well. So I think the challenge and the working as a team to deliver this practice is really a strength of this course. So I think you do need to have a team of people working to bring all their specialisms to students as well. I think that was one of my main things really. Teacher_Creative_L3

Within this theme of contextualised learning, some teachers described the benefits of having the flexibility to embed certain elements of other subject areas into their qualification. For instance, participants explained that there is scope to integrate Maths and English into teaching programmes. Some aspects of these subjects, prerequisite to achieving the qualification, were integrated into the curriculum in a contextualised way to ensure that learning remained relevant and engaging for students. 

Mastery learning and assessment

In the main study, awarding organisations described mastery learning and assessment in CASLO qualifications as a key mechanism for ensuring competence and supporting student engagement. When exploring what works well in teaching, learning and assessment in stakeholder interviews and focus groups, the mastery requirement was also recognised by some as supporting engagement. Because of this requirement, stakeholders were prone to consider every aspect of the qualification to be relevant, with no parts being seen as “superfluous” or a “tick box exercise”. This point came through clearly in discussions with Creative_Practice_L3 participants. It was evident that the assessment briefs mirrored the fundamental aspects of industry closely, from both teacher and student perspectives.  

And everything from the proposal writing, we do that kind of stretching challenge, we do pitches. We do pitches in front of panels. We bring in industry and they pitch in front of industry. So we kind of stretch throughout every part, nothing feels superfluous, you know, everything feels like it’s done for a very specific reason. It follows that process. So the students, they don’t have the tick box exercise. Everything they see the point of, they see the relevance of it, and there’s nothing in the qualification where we have to say oh look just do it because it’s the criteria. You know, it all fits very holistically into any project. Teacher_Creative_L3

For many students in this sample, the mastery requirement was also beneficial as it tended to act as a driver for developing confidence, knowledge and skills which would be necessary for assessment. This was highlighted in discussions with Hairdressing_L2 participants who described the positive reinforcement that resulted from practicing technical skills and seeking support where needed, to ensure that they were well equipped for each assessment. The mastery requirement ensured that students became sufficiently proficient in technical skills to conduct treatments with fee paying customers, which is the goal that students were cognisant of throughout the course. Several teachers also explained that students recognised the importance of meeting each assessment criterion – primarily because each assessment would count towards the completion of their qualification. Thus, to support students in keeping track of which assessment criteria they have already met, teachers often used e-platforms that visualise progress against assessment criteria. This was reported to be highly engaging and motivating for students. 

Mastery learning and assessment was also considered a driver for preparing students for a range of routes post-qualification, including further or higher education courses, employment and apprenticeships. Preparedness for a range of routes was in part attributed to the transferable skills developed throughout each qualification, in addition to the “confidence and resilience” fostered through continuous assessment and learning. Furthermore, for students moving onto the next level of study within the same pathway, there was some awareness that the current qualification “paved”’ the way to the next one. They expected the next level of study to be an ‘extension’ of what they had encountered already in terms of content. There was also the expectation that the structure and format of the next level of study, including the mastery requirement for some, would be familiar which would also be beneficial.

yes I feel confident going onto the level 3, I know it’s an extension of what we’re already doing, more observations, yeah it’s paved the way, level 2 has definitely paved the way. Student_Teaching_Support_L2

Stakeholder views of potential assessment related problems in CASLO qualifications

As part of our main study, awarding organisations were asked whether or not they recognised 5 assessment related problems identified in the academic literature by Ofqual researchers. These potential problems were:

• unclear assessment criteria and grading thresholds
• atomistic assessor judgements
• poorly designed assessment tasks and events
• lenience and malpractice
• inappropriate support

When asked whether they recognised these potential problems in the context of their exemplar qualifications, most problems were recognised by most awarding organisations (their responses were classified as recognised, not entirely recognised, or not recognised). In response to problems that were recognised, awarding organisations described a range of mitigations put in place to reduce the risk of the problems occurring. Mitigations for assessment related problems included contextualisation of qualifications, real-life task setting and holistic assessments, in addition to offering support, guidance and implementing quality assurance processes. They also recognised the role of protective attitudes of students, and occupational and professional expertise of teachers. Awarding organisations explained that these kinds of mitigations and protective factors were important for ensuring the risks identified in the academic literature were reduced or, in some cases, nullified in their qualifications.

In this stakeholder study, participants discussed what worked well and not so well in relation to assessment, and responded to qualification specific questions about mitigations described by awarding organisations. The findings presented in this section reflect stakeholder views grouped into the 5 assessment problems identified above. The findings are structured in this way to link the findings from this validation study to our main study.    

Inaccurate judgements against unclear assessment criteria and grading thresholds

The academic literature identifies problems related to assessors making inaccurate or inconsistent judgements on whether students meet the relevant assessment criteria in CASLO qualifications, which results in some students passing when they should not, and some students not passing when they should. According to the literature, that happens because assessment criteria are very hard to write and interpret precisely. Therefore, assessment criteria alone cannot communicate the threshold between passing and not passing, or between different grades, precisely enough. This can potentially present significant problems for the CASLO approach because assessors need to make heavy use of these criteria when judging (and grading) student performances directly.  

Most awarding organisations in our main study recognised the potential problem of inconsistent or inaccurate interpretation of assessment criteria that stems from their inherent limited precision. However, they described several mitigations to reduce the risk of this problem occurring, including conducting quality assurance processes, incorporating a range of assessment design processes and promoting contextualised, holistic assessment practices. Furthermore, awarding organisations referred to the importance of ensuring the occupational and professional expertise of assessors to support with interpreting and applying criteria.

In this stakeholder study, a large number of teachers described assessment criteria to be sufficiently clear and transparent to support them in making judgements. On the whole, teachers explained that they understood the requirements of assessment criteria and found value in using these criteria in contextualised and holistic assessment events. Repeatedly across discussions, teachers emphasised that they were able to conduct assessments which were “valid” and “robust” in real-life contexts. Furthermore, student knowledge and skills were often continuously observed or assessed by teachers to ensure standards were consistently met throughout the qualification. This was particularly evident in the Hairdressing_L2 qualification and the Teaching_Support_L2 qualification which involved observational assessments. 

I like the fact that you can keep observing over multiple periods of time, because you can really ensure that you’ve got proper, robust validity for these assessments and I think that’s so important. Just one day, one exam, I don’t think is good, you only needed to have one thing go wrong on the morning, so I think this is what’s so good with it. Teacher_Hairdressing_L2 

When discussing the process of judging student performances, many teachers described the importance of quality assurance processes. Teachers particularly emphasised various benefits of specific external quality assurance processes which supported them in interpreting assessment criteria accurately and consistently. Cross centre external quality assurance events organised by awarding organisations, including standardisation and moderation activities functioned as important anchors for collectively understanding, and applying criteria and gaining insights into different approaches to delivery. These quality assurance activities were in some cases extensive, with “groups of teachers” spending a full day in other centres, exploring student work and approaches to delivering the qualification. Teachers from the Creative_Practice_L3 qualification suggested that such events, particularly those conducted face to face, provided opportunities for “networking” and “sharing of good practice” across institutions, which was important for understanding and applying assessment criteria. 

In contrast, some other quality assurance activities were viewed less favourably by teachers in this sample. For example, teachers across qualifications described that, at times, advice which they got on important aspects of assessment and delivery from external quality assurers could be conflicting. Some teachers referred to contrasting advice received on the amount and types of evidence that students needed to demonstrate to fulfil criteria. Other teachers described differing advice being given for the same qualification by previous and new external quality assurers. 

It’s the discrepancy with the EVs each time […] we’ve got someone different and they’ve picked up something totally different to last year. They said we did really well, and we haven’t changed anything, so it is the interpretation that does really differ and it’s quite frustrating. Teacher_Business_L3

We are all working under our EQAs as well and I’m sure, because I’m in [a specific county in England], we’ve all got different EQAs who have got different expectations from us. Our EQA is quite happy with what we’re doing, but your EQA might not be happy with what we’re doing, is what I’m trying to say, so it’s about standardising EQA as well, isn’t it? Teacher_Teaching_Support_L2

So, we just standardise, we pick up bits through our EV visits, but it depends because they change, so you get someone coming and say one thing and then the next one will say something completely different. Teacher_Hairdressing_L2

The importance of internal quality assurance processes also came through as a key aspect of the process of making judgements. A small number of teachers described delivering or completing internal quality assurer qualifications alongside or prior to teaching their CASLO qualifications. For these teachers, understanding and applying the qualification standard, rather than their own standard, during the judgement process relied heavily on referring back to the criteria. It was suggested that even for experienced assessors, it takes time to learn where the standard is. In addition to this, many teachers emphasised the important role that internal quality assurers play in supporting consistent and accurate use of assessment criteria within their centres. Some teachers raised that they valued having an internal quality assurer to support them with making valid judgements where there was a degree of uncertainty.

I also teach assessing and IQA qualifications, and those that want to go out and assess in hairdressing, they come in to me on my salon sessions so I’m actually teaching their assessing qualification alongside my other learners and I think that they do, when they’re first learning the assessing, they do look at it as their standards of what they would be doing and you have to take them back down to, no, you need to read the performance criteria, this is the range that we’re covering and we’ve got to actually put it back to they are either a level 1, level 2 or a level 3 student and shopfloor’s way up here, so you do still do a lot of learning. We’re all still learning day by day, even with all our experience in hairdressing, that’s why we do CPD. Teacher_Hairdressing_L2

An important element of the judgement process involved grading student performance or evidence. Teachers that delivered the graded qualifications in this sample, described their varied experiences surrounding clarity when using grading criteria. A subsample of teachers explained that understanding what was required at pass and merit level was often straightforward. However, understanding the threshold between a merit and distinction grade was not always clear from the assessment grading criteria. In one qualification, teachers highlighted that despite the availability of exemplar materials, there were very few instances of students achieving a distinction for one non-CASLO externally assessed and marked unit. While in this case the perceived lack of clarity between the merit and distinction requirements in the grading criteria did not impact assessor judgements directly, it still affected how well they could prepare their students for this unit’s assessment. Within this qualification, teachers also raised a potential training need, particularly for new teachers, to enable them to support students with exam techniques and judging mock assessments.

Atomistic assessor judgements    

Another criticism from the literature related to the nature of the assessment criteria in the CASLO model is that assessors are often reduced to ticking off assessment criteria, criterion by criterion, when assessing. This can lead to the potential problem of atomistic judgements, as there may be more to having met a learning outcome than having satisfied each individual assessment criterion.  

Less than half of awarding organisations in our main study recognised this potential problem in their qualification. Those who did recognise the problem, described a range of mitigations to reduce the risk of this problem arising, including holistic delivery and assessment, contextualisation and real-life task setting, building in qualification and assessment design processes and offering support, guidance and quality assurance activities within centres. Ensuring the occupational and professional expertise of assessors was also seen to reduce the chances of this problem arising.

In this study, stakeholders described a range of holistic rather than atomistic approaches to assessing student performance. The most commonly cited approach involved contextualised assessment conducted in live workplace environments. Teachers in the occupational qualifications (for instance, the Teaching_Support_L2 and Hairdressing_L2 qualifications) carried out assessments in live rather than simulated environments, which required students to utilise skills and knowledge for the purpose of the assessment task and to fulfil their designated role within the workplace setting. In the Teaching_Support_L2 qualification, students during their placements were expected to carry out classroom-based teaching activities as a teaching assistant would normally do, which enabled them to both demonstrate relevant assessment criteria and meet the needs of students. Consequently, when teachers judged student performance, they often did so in a broader context, rather than criterion-by-criterion. In addition, teachers highlighted that assessment tasks were often designed to be holistic through assessing more than one criterion at a time. This resulted in assessing several naturally interconnected criteria during each assessment event.

Further to this, several teachers explained that judgements of student performances were rarely based on a single assessment event. In some cases, judgements surrounding the acquisition of competencies could only be made after repeatedly witnessing a skill or evidence of knowledge. There appeared to be variation across teachers in the number of observations they required for different skills, with teachers taking into account advice and guidance from external quality assurers and the strength of evidence observed. Furthermore, teachers found it useful to use a range of assessment methods when repeatedly assessing skills.

I will set time at the end of the second observation, or third if I absolutely have to, that’s when we’ll do a little Q&A or a professional discussion at the end and then we can mop up everything that I haven’t seen and go into detail for those things that I’ve partially seen and to cover any of those describes that I couldn’t see. So, we do it that way, so we have at least two  assessment methods for the skills, but yeah, it works pretty well because I’ve been teaching it for a while now so I know what I can see and what I need to discuss. Teacher_Teaching_Support_L2

Teachers in this study also described the importance of professional and occupational expertise when making assessor judgements. Professional judgements were used to interpret and apply the criteria. Teachers suggested that professional judgement is used when:

• considering when further evidence was required to fulfil the criteria
• differentiating between grades
• recognising performance that was beyond the assessment criteria

There was some recognition that when assessment opportunities were limited (or judgements needed to be made on very limited evidence), professional judgements were used to decide whether assessment criteria had been met. This was particularly important as the nature of assessments was often ”personal to each individual learner”. Although teachers recognised the value of professional judgements, a few suggested that additional range statements would be useful and supportive when it comes to judging the quality of evidence collected. 

Quality assurance processes also emerged as fundamental aspects of ensuring discrete judgements were made in the context of holistic tasks. Teachers described internal quality assurance activities such as internal verification processes, blind marking, multidisciplinary marking, assessor discussions and examining performance across formative and summative assessments as important for standardising judgements. These varied and staggered quality assurance processes ensured authenticity of the judgement process, even when judgements were made discretely.

Although the number of employers involved in this study was limited, some employers shared their views on the risk of atomistic judgements. Employers suggested that often teachers have limited windows of time to ensure that students meet every assessment criterion to pass the course. This, according to employers, leads teachers to become atomistic in their approach to assessment. This perception limited employer confidence in allowing students to begin carrying out treatments when they entered the workplace, as employers were not always confident in the skills students acquired. Hairdressing_L2 tutors were aware of this, but felt that it might be a misperception, and suggested that employers were not always aware of the learning journey of each student. According to teachers, students often start the qualification with differing levels of knowledge and skill, and they work with each student in a holistic way to ensure that they meet all assessed criteria. Teachers recognised that employers focused on the competencies of students at the end of the course without recognising the progress made throughout the qualification. These contrasting perspectives are illustrated in the quotes below. The first quote reflects the views of a salon owner and former teacher-trainer, with 30 years of industry experience. The second quote reflects the views of a current college-based Hairdressing_L2 teacher, who had recently participated in a panel discussion exploring industry needs and expectations.

Assessments, again, they work as best as they can, but they have to still have a correlation to the units. And, you know, I feel sometimes unfortunately as a lecturer it becomes a tick box exercise, and it’s just you go in and it’s just your head down and you’re ticking all the boxes, and then you’re forgetting to look up and really make that assimilation with the student, the client and the textbook, and it just feels sometimes that they all don’t mesh very well. It becomes a bit of a cacophony where it’s just like just loud noises and it’s like I’ve got to get this done because this is what the book says, and oh my gosh we’re now in January, and oh my gosh we’ve only got X amount of months. And in that anxiety and anxiousness, it’s from both sides, it’s from the lecturer and the student, and then at the end of it, you’ve got this logbook, you’ve just got all these ticks and these consultation cards, but what do you do with it, because then you’ve gone into an employment environment and then, to them, it’s just like well yeah, fine, great, look, we’ve got, can you just, can you sweep the floor, can you make the tea? Then it’s like oh my god, is this really what I signed up for? You know, it’s difficult, it really, really is. Employer_Hairdressing_L2

The ones who are in industry aren’t teaching, so they have to sometimes understand where we’re coming from and what we start with and what they get at the end result. Teacher_Hairdressing_L2

Poorly conceived assessment tasks or events

Some critics say that having detailed assessment criteria as standards to assess against makes it look like the assessment process is extremely straightforward. As such, they claim, assessors often fail to appreciate how hard it can be to elicit construct-relevant assessment evidence, even where the standard to assess it against may be clear. For this reason, CASLO qualifications are vulnerable to being based on poorly designed assessment tasks or poorly conceived assessment events that do not elicit the right kind of evidence against the assessment criteria. 

Most awarding organisations in our main study recognised this as a potential problem for their qualifications and described putting in place packages of mitigations that included support and guidance for centres, explained and facilitated quality assurance processes, emphasised real-life task setting, recommended contextualised and holistic assessment and qualification and assessment design processes and features. Similar to the potential problems described earlier, the occupational and professional expertise of assessors was viewed as also protecting against this potential issue.

In this stakeholder study, assessment events conducted in real-life settings were viewed favourably by teachers and students alike. Participants in the Hairdressing_L2 and the Teaching_Support_L2 qualifications agreed that it is live rather than simulated assessment events that elicit the highest quality evidence. The students studying these qualifications explained that whilst the process of assessment was “daunting for some people”, many were put at ease by the familiarity of both their assessor and the assessment environment. Familiarity with the setting and required performance standards often developed over time, through placement activities and teaching and learning conducted in the assessed environment. Hairdressing_L2 teachers described allowing students to invite non-fee-paying clients into the salon environment during the teaching and learning phase of the course, to support students in developing the skills and confidence needed for assessment events. This highlights the interconnected nature of teaching and assessment, as the effectiveness of the live assessment tasks was in some cases contingent on familiarity with the real-life setting.

It is important to highlight that both students and teachers were aware of the challenges of ensuring that real-life settings presented natural opportunities to meet assessment criteria. For example, teachers in the Hairdressing_L2 qualification explained that some elements of the criteria, such as “infection” or “infestation”, appeared less frequently in the salon environment and, thus, were often more difficult to observe. To ensure that such criteria could still be met by students, it was useful for teachers to have other methods of assessment, such as “oral questions”, available to explore the skills and knowledge of their students.

In addition to utilising assessments that take place in real world settings, teachers “enjoyed” opportunities to write assessment briefs which would elicit appropriate evidence. The process of writing assessment briefs was often well thought through by teachers as they described considering cohort needs (such as level of “challenge” for students) and learning outcomes, whilst also bringing together “contemporary practitioners” locally and nationally. The process of writing briefs allowed for high levels of collaboration within a community of practitioners and promoted sharing of knowledge to create high quality briefs which would be engaging for students. Within the Creative_Practice_L3 qualification, highly contextualised assessment briefs were also appreciated by students. They expressed that the broad assessment criteria often enabled them to engage directly with industry as part of their assessment tasks, making the assessment process “inclusive for everyone on the course”. Students explained that “live briefs” promoted use of a wide range of skills and facilitated production of high quality evidence.

Although only one qualification in this sample included the non-CASLO feature of externally set and assessed exams, it is important to recognise the positive sentiments shared about the evidence elicited from these assessment events. The requirements of external assessment tasks in the Business_L3 qualification were viewed positively as “opportunities to get students exploring their own brands, their own businesses they’re interested in”. Personalising assessment tasks allowed students to demonstrate their knowledge and understanding within areas of Business that interested them most. Many teachers and students explained that the externally set exams introduced “rigour” and an “accurate measure on how [students were] actually doing”. However, there were also concerns that exams required high levels of preparation to ensure that students knew how to perform in the externally assessed event. This preparation was often exam specific or related to teaching students how to complete the exam, and therefore was felt by some to have little value beyond the assessment event.

Participants in this study also discussed support and guidance from awarding organisations in the form of exemplar assessment tasks and assignment briefs. Teachers valued awarding organisation resources and indicated that the flexibility to adopt or adapt assessment tasks to suit the context and cohorts was highly useful, as opposed to imposing on teachers a more prescriptive approach to task design. In fact, some teachers expressed a compelling need for awarding organisations to provide additional exemplar assessment briefs and tasks.

I don’t believe the awarding body really should, in my opinion, be responsible for creating all your resources. I do think that they should give you something to relate to […] I feel like they’ve given, VTCT have given a lot of information for us to be able to use that information to create what works best for our learners and to be able to meet the criteria for assessment. Teacher_Hairdressing_L2

In contrast to this, a small number of teachers highlighted perceived misalignment between awarding organisation-set assessment tasks and briefs and other qualification documentation. For instance, one teacher explained that the language used in assignment briefs and grade descriptors was not always well aligned. Such mismatches, resulting from the assignment task outlining requirements which were not reflected in the marking criteria, in some cases increased the chances of students not achieving higher grades. When instances of such misalignment occurred, gaining further support from the awarding organisation or from the centre lead for the qualification was found to be helpful. In another qualification, one teacher described a lack of clarity in the language used to convey the assessment criteria and an accompanying awarding organisation assessment task, which annually raises concerns about whether students can understand the requirements appropriately. However, the participant stated that once teachers explained the language to students, they were able to understand the project and achieve the grades that they aimed to achieve. These kinds of discussions highlight the importance of assessment tasks aligning with other qualification documentation and resources.

Lenience and malpractice

Some critics say that the imprecision of assessment criteria can act as a smokescreen for assessors, allowing them to be intentionally lenient towards students who have not quite reached the qualification standards, giving undue benefit of the doubt. This can be exacerbated towards the end of sessional courses, for students who are just about to leave, but who still have not quite achieved all their learning outcomes. Assessors may try to pass students who are a long way from meeting the qualification standards, resulting in malpractice. They can get away with this – according to some critics – because it is extremely hard to detect and correct inaccurate assessor judgements under the CASLO approach. 

Most awarding organisations in our main study recognised both of these potential problems but described mitigations put in place to reduce the risks. Mitigations included quality assurance processes and support and guidance for centres, supporting students and learning through flexible delivery and using specific qualification and assessment design features. Awarding organisations also recognised the protective role of practitioner attitudes in guarding against this risk.  

On the whole, stakeholder discussions did not reveal any explicit descriptions of lenience or malpractice within centres. Instead, there were discussions related to the protective attitudes of practitioners in seeking to maintain standards in qualifications. This was apparent in discussions surrounding attrition rates of students, as teachers recognised that not all students would be able to meet the demands of the course. Indirectly, these kinds of discussions highlighted an awareness of the standards required and the need for the course requirements to be met without exception. 

[a number] of our first year students don’t survive the course. It’s not what we want, we are trying to really hard to support the students and this year we’ve been on a right old mission to make sure that doesn’t happen. Teacher_Business_L3

Stakeholders held similar views emphasising the importance of standards being met and attained, as students will be working within industries such as school settings, with potentially vulnerable users. Therefore, exceeding rather than undermining standards was often encouraged. Teachers also described their focus on implementing high quality teaching and assessment to ensure that students were well-prepared for opportunities beyond their qualification. Further to this, centres often built longstanding relationships with placement providers, some of whom went on to employ students post qualification. Thus, to maintain the relationships and reputation of the centre, it was imperative that the quality of students completing the qualification was high.   

Qualification design features such as having the scope to extend the amount of time available to complete the course were described favourably by teachers. For example, Hairdressing_L2 teachers explained that when justified, it was possible to apply for extenuating circumstances for students who were unable to achieve the qualification in the allocated time. Teachers explained that applying for an extension of completion time required them to indicate to awarding organisations early in the process that certain students would not meet the qualification requirements in the time available. Being able to extend the qualification time from 12 to 18 months for those students was described as highly beneficial.

Teachers from across the 4 qualifications also described diverse delivery approaches, such as fast-tracked courses or online courses, which due to their condensed or low contact nature could be vulnerable to students being given undue benefit of the doubt. However, teachers agreed that quality assurance processes, including external quality assurance checks, played a crucial mitigating role in ensuring that competency had been demonstrated and, therefore, the risks of undetected lenience occurring in such courses were minimal. 

When discussing potential lenience and malpractice, teachers from a range of centre types, such as adult learning centres and independent training providers, occasionally mentioned funding arrangements. Interestingly, whilst teachers described reductions in funding available for specific qualifications, this was not cited as a potential risk for teachers giving students the undue benefit of the doubt or passing students who did not meet the standard to secure or attract funding. Instead, the availability of funding was recognised as a barrier for some learners accessing qualifications, particularly adult learners who had previously qualified for funding programmes that are no longer available. 

Inappropriate support

The literature identifies problems related to the blurring of the lines between formative and summative assessment in the CASLO model, which typically uses internal, and often continuous, assessment in both workplace and college settings. This can lead to students being given too much support by tutors or assessors (intentionally or unintentionally) and then being assessed as having achieved a higher standard than they would have achieved independently. 

Most awarding organisations in our main study recognised inappropriate support and blurring of the lines between formative and summative assessment as potential problems for CASLO qualifications. As with other problems, multiple interrelated mitigations were suggested, including quality assurance processes, support and guidance for centres, hybrid aspects such as limiting resits or use of external assessors and contextualised holistic assessment.

In this stakeholder study, focus group discussions related to formative and summative assessments did not reveal any explicit descriptions of the blurring of the lines between these 2 types of assessments. In fact, many teachers and students explained that assessment events taking place in real world settings were often clearly delineated from formative assessment activities. This was because students often explicitly indicated their readiness for summative assessment events, and observations were scheduled and conducted with the intention of recording those events as evidence of competence.

Yeah, so for your assessment, it depends on what criteria you need for an assessment. So say if I needed a short grad assessment doing. So when you come into the salon and you look at what clients are booked in for a cut and a blow, you don’t really know what you’re getting until that client’s in and they tell you about I want a short hairstyle I want this. So you look at what assessments can I gain from this client and then you’ll think right I can pass an assessment. I’m confident to do a short grad. You put your card on the board to say that you’re ready for an assessment, and your teacher will assess you on your style and your cut and your finish. And you’ve got to hit that criteria what you individually need. Student_Hairdressing_L2

Teachers in this study also recognised the value of awarding organisation support and guidance on how to provide appropriate levels of feedback to students. This support offered useful direction on how to provide both “manageable” and “meaningful” feedback. This guidance reduced the workload of teachers, as they were advised, for instance, that they did not need to write feedback at criterion level. 

In addition to discussing significant amounts of feedback available post assessment events, teachers and students also commented on high levels of support prior to assessments. For instance, a small number of students described regular feedback through monitoring points or mid-point assessments, which provided opportunities to “reflect on [how] to get a better grade”, with some teachers offering provisional grades against each assessment criterion. Similarly, students from another qualification described completing “a lot of mocks” prior to exams to support them in knowing how to apply their knowledge to the information presented in live assessments. In another qualification, students described receiving tailored one-to-one feedback prior to observations, with some explaining that teachers indicated where there was sufficient evidence or where additional evidence could be added to portfolios. For some, this feedback was available “after every session” when assessment events were close in proximity. Whilst the insight from employers and HE recruiters on the topic of inappropriate support was limited, one HE recruiter raised the concern that continuous feedback and support in CASLO qualifications sometimes results in overreliance on tutor support, which would not be available to students to the same extent in a university setting. This participant explained that some students request high levels of support when preparing for assessments and can be low in independence because of the “scaffolding” received when completing their CASLO qualification.

As a final point, several participants described limited resubmission opportunities as a strength of the Business_L3 qualification. One HE recruiter highlighted that it ensured assessments were not repeatedly “modified” based on feedback received from teachers, which could result in the assessment losing its “authenticity” or reflecting feedback rather than the student’s ability level. This was a concern in earlier versions of the qualification, particularly as students would not have had the same resubmission opportunities in university settings.

Potential teaching, learning and delivery problems in CASLO qualifications

As part of our main study, awarding organisations were asked whether or not they recognised several teaching, learning and delivery problems identified by Ofqual researchers in the academic literature. These problems were:

• local or personal irrelevance and a lack of qualification currency
• content which is hard to pin down being missed
• downward pressure on standards
• incoherent teaching programmes
• lack of holistic learning
• superficial learning
• student demotivation and disengagement
• assessment burden

Similar to the assessment problems discussed in the previous section, awarding organisations discussed these problems to varying degrees, with most problems being either fully or partly recognised, and others not being recognised at all. In response to problems that were recognised, awarding organisations described a range of mitigations put in place to reduce the risk of these potential problems occurring. Mitigations for teaching, learning and delivery problems included various quality assurance processes, recommendations related to holistic approaches to teaching and learning, and providing centres with support and feedback. The occupational and professional expertise of teachers and assessors also emerged as a protective factor against some teaching and learning problems. 

In stakeholder focus groups and interviews, participants discussed what worked well and not so well in relation to teaching, learning and delivery, and replied to qualification specific questions relating to mitigations described by awarding organisations. The findings presented in this section reflect stakeholder discussions surrounding potential problems. These experiences have been grouped underneath the problems identified at the beginning of this section. The findings are structured in this way to align this validation study with the main study (report B). 

Local or personal irrelevance and lack of currency

Because CASLO qualifications are often highly specific about the learning outcomes that need to be acquired, this has led some critics to claim that they are too inflexible to respond to local economic needs, the bespoke needs of small employers, or needs of students with particular interests or aspirations. This lack of flexibility may then lead to content being taught that is of little local or personal relevance to users. Relatedly, the level of detail included in learning outcomes and assessment criteria in CASLO specifications is seen to inevitably tie them to existing work functions, or to contemporary concerns. This may limit their currency and mean that CASLO qualifications provide poor preparation for the future. 

Only a small number of awarding organisations in our main study recognised these as potential problems for their qualifications, with 2 recognising local or personal irrelevance and 3 lack of currency as potentially relevant. The main mitigation types proposed by awarding organisations were holistic delivery and assessment, contextualisation and real-life task setting, qualification and assessment design processes, features and support, guidance and quality assurance processes. The occupational/professional expertise of assessors was also recognised as a protective factor against this problem.

In this stakeholder study, highly contextualised and tailored teaching and assessments emerged as an important feature for ensuring relevancy and currency in each qualification. The scope to contextualise and tailor qualifications appeared to be linked to the broadness of learning outcomes, which were adapted to student interests and the local contexts within which qualifications operated. This was particularly evident in the Creative_Practice_L3 qualification, as teachers and students described the individualised nature of student learning journeys. Furthermore, many students described high levels of engagement through pursuing topics and pathways of most interest to them.

It’s very broad based. It’s very, I think it’s relevant to the students because they’re coming in with different ambitions, and I think from what I see with my students they want to do different stuff. So the course is, because it’s so broad based and it allows students to explore their ideas or explore their ambitions in different areas I think that’s one of the strengths of it. Teacher_ Creative_Practice_L3

In other qualifications in this sample, it was possible to contextualise qualifications through engaging with local and national employers, and industry professionals. Again, the scope to contextualise was provided by relatively broad learning outcomes, which could be tailored to each centre’s local situation. However, several teachers recognised that the scope to contextualise qualifications in this way was heavily dependent on the availability of centre resources, such as time and budget. Thus, the degree of contextualisation appeared to differ within qualifications. 

Contextualisation of qualifications through placement requirements was also recognised as a positive aspect of one qualification in this sample. The broad learning outcomes in the Teaching_Support_L2 qualification often enabled students to complete placements in settings of interest to them, with some participants completing primary school placements and others completing their placements in colleges. This was highly beneficial and increased the qualification relevancy for students, as they could tailor learning outcomes to a range of settings. However, securing placements in a setting which enabled students to collect evidence to meet qualification requirements was a challenge for some. Teachers and students explained that workplaces offering placements could, in some instances, require students to support their operational needs, rather than allow students to focus on the qualification requirements that needed to be fulfilled. Consequently, for teachers, there was a delicate balance to be struck at times to ensure the interests of their students were being met, but also to maintain relationships with employers that would host future student placements.

It is interesting to note that the level of contextualisation and personalisation differed between qualifications in this sample. For instance, teachers from the Creative_Practice_L3 qualification described several pathways through which students could personalise and contextualise their learning as they moved through the qualification (for example, architecture, fine art, graphic design and photography). This approach clearly differs from the one adopted by the Hairdressing_L2 qualification, in which the focus is, as explained by teachers, on students first and foremost developing specific technical skills. In this qualification, there are fewer opportunities to personalise teaching and assessment events beyond what naturally presents itself in observations, which largely depends on customer requests. 

In the Hairdressing_L2 qualification, the influence of National Occupational Standards^{[footnote 5]} appeared to add value to the qualification, as students were developing technical skills needed for industry to a required standard. But there were several discussions related to the fast-paced changes that occur in industry and concerns of how quickly these translate into qualifications. This challenge was particularly evident amongst Hairdressing_L2 teachers who mentioned changes to treatments involving hair colouring and hair straightening techniques as examples. Teachers highlighted that these kinds of treatments have evolved at a quick pace and client demand for specific treatments have also changed over time. As a result, a risk arises as to the relevance and currency of this qualification if its content is not updated at the same speed as industry trends and practices. 

The currency of the qualifications in this sample was also considered when the alignment between qualification requirements and industry requirements was discussed. Teachers explained that, at times, they had to put considerable effort into delivering their qualification in a way that would assure both meeting the requirements of the assessments and preparing students for industry. It was not clear from discussions whether this issue arose as a result of misalignment between assessment and curriculum, or whether it reflected other, more complex factors. Nevertheless, there was some recognition amongst teachers that they needed to teach specific content for the purpose of the assessment rather than for industry.

For many participants in this sample, the broad nature of learning outcomes enabled teachers to prepare students for a range of progression routes post qualification. Often, teachers described supporting students to develop broad skills which would be useful to both employment settings and university settings post qualification. For instance, in the Business_L3 qualification, teachers shared that around 60% of their students progress into university either locally or nationally. These students went on to study a broad range of courses including business, engineering, architecture and law. Teachers highlighted that following the completion of their course, a proportion of students also moves into apprenticeships and employment. Students and HE recruiters also recognised the value of universally relevant skills developed by students which could be generalised to a range of contexts. 

I feel like the course prepares you for your next steps and because of how practical of a course it is, it ensures that you’re in the best position possible to move into university next year because, as I’ve said, I feel like the set-up and the way the course is taught and assessed is like parallel to how further education is, but also if you go into an apprenticeship, the fact that you’ve got to meet deadlines, work in teams, all the core employability skills are portrayed through the course, so I feel like it excellently prepares you for your next steps. Student_Business_L3

Content hard to pin down gets missed

Some critics say that learning outcomes that may be essential to a qualification, but that may be very complex and difficult to put into writing in the commonly used format of relatively brief statements, get left out of CASLO specifications. This might mean that students miss out on essential learning. This kind of content might include, but is not limited to, relatively esoteric outcomes such as “independence”, “autonomy”, “problem solving” or “professional judgement”.

Only 4 awarding organisations in our main study acknowledged the relevance of this potential problem to their own qualifications, whereas most awarding organisations did not recognise this potential problem either partly or entirely. Those who did, described mitigations related to qualification and assessment design processes and features, the inclusion of implicit content, contextualisation and holistic assessment and the occupational or professional expertise of assessors, which were presumed to reduce the chances of this problem arising. 

In this stakeholder study, several participants described the inclusion of esoteric outcomes in assessment criteria and in implicit content within their qualification. Teachers and other users recognised and referred to criteria requiring students to demonstrate “imaginative”, “perceptive”, “individualised” skills or knowledge. This was particularly evident in the Creative_Practice_L3 qualification. Teachers described making heavy use of assessment criteria that included such abstract outcomes prior to assessment events to ensure that students were aware of the meaning of these terms and how they would be assessed. Interestingly, if the qualification was graded, these more esoteric constructs were often reflected in the grading criteria for higher grades, such as distinction. Teachers explained that these criteria supported them in understanding and providing students with clarity on the meaning of such outcomes.

And those words are the words that are embedded in the synoptic assessment criteria: imaginative and perceptive and individualised. And that helps you to be able to talk to students about how their work is progressing. I think that synoptic assessment is really helpful to have those conversations with students ongoing through their practice and to talk to them about where they might feel they sit within the assessment criteria. I think that’s really helpful. Teacher_Creative_Practice_L3

It is also interesting to note that one HE recruiter recognised the presence of hard to define skills such as “initiative” and “resourcefulness” in Business_L3 students following the completion of their qualification. These skills were discussed as being, in some ways, byproducts of skills assessed explicitly within the qualification. 

I think that they’re used to doing group presentations. The creativity is certainly there in terms of using their initiative. I think that some of the better students, they are very resourceful, and that’s good to see. So yeah creativity, using their initiative, doing group presentations, they’re very used to doing group work, and they’re pretty good at managing people in groups as well, which is obviously a really important skill, the development of those teamwork skills and those interpersonal skills, and how they’re very happy to call out non-contributors, which I think is very important. HE Recruiter_Business_L3

A further point raised about content which is hard to pin down centred on content that is not taught or assessed but is seen to be valuable to employers. Some teachers recognised that broader, more esoteric skills were often desirable to employers and, therefore, teachers sought to embed them within qualifications through contextualised teaching and assessment. For example, in the Hairdressing_L2 qualification, teachers often encouraged students, if possible, to develop skills beyond the assessment criteria. This was done through assigning students to different roles to develop soft skills when in the salon environment (such as being on reception), integrating ‘pressure tests’ (involving preparing their workstations) or introducing students to other trade practices to foster development of broader skills important for future salon roles. 

We teach separately behaviour and attitudes of what’s expected, which I know everybody is doing to some extent, but I think behaviour and attitudes is such an underlying or an underpinning part of being a hairdresser, that maybe that’s an area that needs developing more. Teacher_Hairdressing_L2

Similarly, whilst not necessarily part of the formal teaching or assessment for the qualifications, broader enrichment activities were actively encouraged to develop student knowledge and skills. Teachers described various competitions or voluntary roles as providing insights into and experiences of industry, which helped students to put teaching and learning into practice, to integrate their learning and to develop broader transferable skills. However, some participants explained that there is limited time available to engage in such experiences because of the continuous assessment requirements. For instance, in the Hairdressing_L2 qualification, many participants described the value of entering industry competitions to support skill development and build industry knowledge and experience. However, it was not always possible to prepare for and enter into certain competitions as students needed to prioritise the final assessment. This challenge of competing demands did not appear to be limited to the more occupational qualifications or a specific level of study, as similar experiences were also reported in the Business_L3 and, to some extent, in the Creative_Practice_L3 qualification.  

The only thing I would say is with having the last assignment being the colouring one, the big like exam that we have on the colouring, we also had the competitions running alongside it, and I felt to be given that opportunity to go and do a competition, which is good, and it’s good for keeping your skills up and getting a good experience behind you. But also having the exam in the background as well I felt like it was too much to pull together. Student_Hairdressing_L2

Downward pressure on standards

Because awarding organisations have to specify standards that should be achievable by all students in a target cohort, critics say that this puts a downward pressure on standards. This means that, potentially, no single learning outcome can be pitched at a level that is beyond the reach of the lowest attaining student within the targeted cohort, especially where, with a strong mastery model, failing even one learning outcome would mean not achieving the entire qualification. 

Only 2 awarding organisations in our main study recognised this problem as potentially relevant to their qualification. The main mitigation types proposed included qualification or assessment design features and processes such as getting the demand right for the qualification level, providing a range of materials and supporting learning. Recognising the role of protective attitudes was also mentioned as another possible mitigation against this problem. 

In this stakeholder study, there were very few explicit discussions surrounding the level or standard at which qualifications are pitched. Indirectly, teachers expressed their preference towards those approaches to delivery which ensured that qualification standards are maintained, as opposed to approaches that may contribute to lowering standards. For instance, in the Business_L3 qualification, some teachers opted to deliver units one by one rather than simultaneously, to enable effective learning and to support students in demonstrating assessment criteria at the required standard. Teachers explained that diluting teaching and assessment through focusing on too many units at any one time, presented challenges in relation to students achieving the unit or qualification (rather than lowering of standards).

Focus group discussions related to qualification entry requirements also provided insights into stakeholder views on qualification standards. Teachers across the qualifications expressed a strong awareness of the importance of meeting qualification standards. Whilst many teachers acknowledged the role that many CASLO qualifications play in offering students with a low prior attainment access to further study, the accessibility of the qualification content to those who are offered a place on the course was an important consideration. For this reason, a large number of teachers described having some entry requirements imposed at centre level, such as minimal acceptable GCSE (or comparable qualification) grades in subjects such as Maths or English. This was to ensure that the content of the course could be engaged with and that required standards could be maintained. 

Our entry requirements for BTECs are, like everyone else, a little bit lower, but we look at the maths. They’ve got to have only just failed maths by like one grade very closely otherwise we don’t accept them because they can’t access the work. Teacher_Business_L3

Standards were also described indirectly in discussions related to the triangulation of assessment evidence. Teachers from the Teaching_Support_L2 qualification explained that in addition to conducting holistic observations which enabled them to assess student performance in real time, they also corroborated this information with witness testimonies and reflective accounts. In other qualifications, standards were ensured through other means. For instance, in the Business_L3 qualification, external assessments (in the non-CASLO unit) were viewed favourably as they were seen to “add a lot more rigour” to the qualification. Such assessments also allowed for “differentiations from pass to distinction” to be applied to grades across units, which was not seen in this qualification historically “when it was all coursework”. It is interesting to note that externally set assessments were not viewed as favourably in the Hairdressing_L2 qualification. Some teachers explained that written assessments were sometimes challenging for some students due to potential language barriers of their cohorts, which is expressed in the quote below:  

Yes, I agree, and I’ve got two Ukrainian students in my cohort this year and their technical skills are really, really excellent, but I’ve had to drop them down to a level 1 because they’re not able to read and write [in English] and understand exams, so there is that issue, but I think in terms of their skills, which is what we’re trying to at the end of the day develop, I think the qualification does work well really. Teacher_Hairdressing_L2

Incoherent teaching programmes

According to criticisms in the literature, because CASLO qualifications pay so much attention to learning outcomes – which can downplay the importance of an underpinning syllabus – many teachers fail to compensate for this, and they fail to plan and deliver coherent teaching programmes. 

In our main study, most awarding organisations recognised the potential problem of incoherent teaching programmes but highlighted that they put in place several mitigations to reduce the risk of this problem. These included flexible delivery and contextualisation, providing teaching inputs, offering centres support and guidance, and embedding quality assurance processes. The professional or occupational expertise of teachers was also seen as a protective factor against this problem.

In this stakeholder study, teachers described delivering teaching programmes which included both teaching inputs devised by their awarding organisation and self-devised materials. The teaching materials provided by awarding organisations were described as useful due to their adaptability to different cohorts and settings. Interestingly, there were differences across qualifications concerning the variety of teaching materials provided by awarding organisations. For instance, in the Business_L3 and the Creative_Practice_L3 qualification, teachers described inputs such as schemes of work and delivery guides, which were said to be extensive and similar to the materials available for a general qualification (such as an A level). For the Teaching_Support_L2 and Hairdressing_L2 qualification, on the other hand, teachers seemed to be more selective in which teaching inputs provided by the awarding organisations they used. Some teachers recognised that whilst awarding organisations offer a range of inputs, they can also “create what works best for [their] students”. These differences across qualifications did not appear to be problematic for users. 

[The awarding organisation] have given a lot of information for us to be able to use that information to create what works best for our students […] we have been teaching it a while so we have got the resources that we need and it’s quite easy because we’ve got the foundation there. Teacher_Hairdressing_L2

We have specs, we have schemes of work, we have structure, we know what we need to do, etc., etc., etc., and loads and loads of guidance, so it’s very similar to an A-level. Teacher_Business_L3

Several participants in this study also recognised the importance of having occupational and professional expertise when delivering CASLO qualifications. Teachers described expanding on content and producing coherent schemes of work and learning programmes based on their teaching and occupational experience. This was emphasised in the Creative_Practice_L3 qualification, as teachers described the need for practitioners to be working within industry alongside their teaching roles. This experience supported teachers in aligning the curriculum with current industry trends, and enabled knowledge and skill transference to their students. 

One Hairdressing_L2 employer shared that maintaining continued professional development in hair salons was integral to teaching and learning. More specifically, this participant explained that having up-to-date salon experience enables teachers to maintain their own skills which could then be transferred to students. Relatedly, there was a concern from one HE recruiter, that without sufficient sector expertise, some teachers might not embed sufficient depth within their teaching programmes. This participant suggested that generalist teachers might give a “lighter touch” when delivering topics they are less knowledgeable in, which impacts on the effectiveness or clarity of teaching. As a result, certain deficits in student knowledge can surface at university level.  

Well I think if you’re going to teach painting you’ve got to know about painting. You know, if I’m teaching architecture and I’ve got to teach the students perspective drawing then I’ve got to be able to do perspective drawing myself. So I think if you’re a graphic designer and you’re going to use computer programmes you’ve got to have, you’ve got to know more than the students haven’t you. You’ve got to be good at it. You’ve got to understand, you’ve got to have knowledge of contemporary practice in the area that you’re working in. So I don’t think anybody could come in, if I’m talking to a student about some work and I want to give them references to look at […] there’s got to be kind of quite a high level of knowledge and experience and skill and practical skill within the teacher themselves otherwise how are they going to teach it. Teacher_Creative_Practice_L3

Both teachers and students across qualifications recognised the significance of having meaningful content links across units within their qualifications for supporting teaching programme coherence. It was not always clear from discussions whether these links were a consequence of how the qualification had been designed by awarding organisations or if implicit links were made at centre level by teachers contextualising the qualifications. The exception was the Creative_Practice_L3 qualification, for which teachers explicitly described themes that were embedded by the awarding organisation across units and allowed for links to be made. There was also a high degree of recognition that the broad learning outcomes in the qualification mirror actual processes found in industry. This allowed the teaching programme to begin with a developmental exploratory phase which was followed by the synoptic assessment, reflecting the process followed in the creative industries.

Another important point to emerge on the topic of the coherence of teaching programmes arose from reflections on content taught but not assessed in qualifications. A handful of teachers explained that they delivered non-assessed content as, in their view, it served an important purpose in preparing students for learning in future units “because [units] build on each other”. Whilst teachers suggested that this could be “confusing” initially, ultimately it was perceived to be useful and “the best way for the students” to learn.

In addition to the views discussed so far, considerable discussions emerged amongst stakeholders on pedagogical approaches which support coherent teaching programmes. As a consequence of unit requirements being expressed via learning outcomes written at a high enough level of generality to be taught and explored in a variety of ways, teachers and students described being able to engage in student-led, rather than teacher-led, activities such as conducting independent research, and participating in group work and whole class discussions. Students from across qualifications emphasised that “it’s not just the teacher stood at the front”, instead “it’s more [students] being able to figure out it”, which ultimately builds engagement. In addition to this, the student-led approach was described as supporting students to develop a wide range of skills, including following independent lines of enquiry, problem solving, developing autonomy and critical thinking skills. These active learning approaches often went hand in hand with contextualised and personalised learning activities, as students were provided with opportunities to independently investigate topics and areas that interested them most. Active engagement was perceived to build student learning incrementally and support its consolidation, as students could focus on all areas, including those in which they felt less secure. 

Yeah, with the teaching, obviously a lot of the time it’s not just the teacher stood at the front, it’s more us being able to figure out it by ourselves as well, which is a lot better at making us understand it, rather than with a lot of the A-levels they do tend to just stand at the front and you copy it down and it don’t really go in, but with [the Business_L3] you are kind of, you’re given hints but then you’re sent away to figure it out yourself and then it’s getting embedded a lot easier and quicker. Student_Business_L3

I think also it’s much more student-centred. It’s much more about the individual. So that adaptability and flexibility within that curriculum allows you to really work with every individual and nurture their own sort of creative pathways. Teacher_Creative_Practice_L3

Lack of holistic learning

Because CASLO qualifications represent learning outcomes one by one – and without overtly representing how those learning outcomes relate to each other – some critics say that students fail to learn holistically. This means that their learning is neither systematic, nor integrated, nor coordinated, which leaves them unable to apply their learning effectively. 

In our main study, this potential problem was recognised by only a small number of awarding organisations, and they described several mitigations to reduce the risks of this problem arising. These included contextualisation of qualifications to different local situations, ensuring the professional and occupational expertise of teachers and promoting holistic approaches to teaching, learning and assessment. Additionally, awarding organisations mentioned mitigations such as support, guidance and quality assurance processes. 

In this stakeholder study, focus group discussions did not reveal any concerns surrounding holistic learning within qualifications. In fact, throughout discussions teachers spoke about various opportunities to “teach holistically”. Several participants referenced real-life work situations such as placements as supporting holistic learning through opportunities to integrate knowledge and skills, “and take that theory back into work placement”. In the college-based qualifications which did not include a placement element, such as the Business_L3 qualification, the contextualisation of teaching and learning served as the main mitigation to this problem. Participants reported that teaching time was not focused on individual units, but there was contextualised, “in-depth investigation” and exploration across topic areas. For instance, one Business_L3 teacher described inviting a guest speaker from a large international corporation to discuss marketing in a “real world” setting. As part of this workshop, the guest speaker in addition to explaining the contemporary marketing practices used by his organisation also described the broader business functioning of the organisation. This encouraged students to think more holistically about the marketing unit.

Focus group discussions also revealed cross-unit synoptic assessments as promoting and supporting holistic learning. More specifically, synoptic assessments encouraged students to integrate and apply learning from across units to a set task. Similarly, in the Hairdressing_L2 qualification and Teaching_Support_L2 qualification, learning and assessment was applied to real world settings, reducing the risks of atomistic application of learning outcomes and assessment criteria. The application of knowledge and skills to wider contexts across the 4 qualifications was viewed positively by both teachers and students. This was, however, highly contingent on the quality of the placement environments and the access to relevant experiences that would support students in applying and developing their skills.^{[footnote 6]}

I think it’s the experiences what you get on placement. Obviously you can learn and learn and learn, but until you put it into practice it doesn’t feel as real and it doesn’t make as much sense, but once you’re doing it, it all clicks into place, so without the placements it wouldn’t be half as beneficial as it has been. Student_Teaching_Support_L2

I think that would depend again on where that student’s placed. Whether they’re in a salon which would give them the opportunities to actually, you know, not just sweep the floor and make cups of tea […] But actually would engage with that student and give them the opportunities that would help them to progress as they need to. Because I think that is just so important, you know. Employer_Hairdressing_L2

Superficial learning

Because CASLO qualifications specify learning outcomes one by one – and because they focus attention on detailed lists of criteria that need to be met for each learning outcome – critics say that this disposes students towards superficial learning. This might involve demonstrating the minimum possible performance on each criterion for each learning outcome – then moving on to the next learning outcome – and not revisiting learning outcomes that have already been achieved and, therefore, not consolidating their learning.

A small number of awarding organisations in our main study recognised and put in place mitigations to reduce the risks of superficial learning. They described mitigations such as holistic, contextualised assessments which increased the local or personal relevance of qualification content and helped to motivate students. Awarding organisations also discussed protective factors such as practitioner vocational and professional commitment and student choice and agency, both of which supported engagement within the qualification. Furthermore, the importance of a range of various qualification design features was emphasised. For instance, gaining access to higher-level qualifications was considered to motivate students who are demonstrating performance beyond the level of their current qualification. Support and guidance through regular touchpoints were also perceived to minimise issues that might lead to superficial learning and support teachers in increasing student engagement in topics. 

Whilst not widely discussed across the focus groups in this stakeholder study, a small number of teachers did recognise the potential issue of students doing the “bare minimum” to “get the criteria covered” and to, therefore, pass a unit. This potential problem was in some instances seen as a consequence of becoming demotivated by receiving a pass rather than a distinction or merit within an assessment, which in turn was deemed to negatively impact student engagement in future assessments (if it meant that the student could achieve no higher than a pass in the qualification overall). In other cases, teachers explained that students unintentionally provided evidence that was just enough to pass an assessment but could have been more detailed to demonstrate their understanding. One teacher said that this was sometimes linked to command verbs such as “identify” which led students to provide “one word […] sentence […] or bullet point” without elaborating or providing examples. Many teachers also described stretching and challenging students to go above and beyond the assessment criteria to ensure that they completed the course with high levels of competency.

It’s a shame, I’ll use unit one as the example where they are capped at a pass if they only get a pass in the first one, because a lot of my ones on the extended certificate now are in the first year and they’re still learning how to study. I’ve got a youngster now who got a pass in his first unit one assignment, and for that individual it’s a bit demotivating because when assignment two is set he is not motivated to apply himself as much as he could because he got a pass in the first one. Teacher_Business_L3

I’ve got some really nice feedback from my work, which is really encouraging, and where I have perhaps been, I don’t know, maybe a bit lazy with my words or just minimalising my words, she’s told me, you could do more, and I just put the extra effort in and now I’m back on track again.Student_Teaching_Support_L2

One higher education admissions tutor in our sample spoke explicitly about wider pressures put on teachers delivering qualifications, including the pressure related to assessment. This pressure was considered to result in many teachers becoming “outcome driven” and culminating in switching their main focus from instilling a “wider knowledge” and “broader base of learning” in students onto conducting assessments. This participant attributed the overemphasis that some teachers have on assessments to wider factors within centres, such as pressures from senior management to ensure high levels of attainment and the need to ensure that students progress into work or higher-level qualifications.

In 3 out of the 4 qualifications in this sample, participants described content that is taught or learned but not assessed, which was suggested by some to reduce the risks of “teaching to the test”. Some teachers explained that some of the non-assessed learning within their qualifications supported or scaffolded teaching in later units. There was also a significant amount of non-assessed content mentioned that students could encounter in placement settings or through contextualisation of the qualification. Teachers explained that these wider experiences are sometimes logged in student learning journeys despite not being assessed. Creative_Practice_L3 participants also highlighted the positive effects of being able to explore the industry without worrying about achieving specific grades in the developmental units. This supported student engagement and progression “without the fear of assessment hanging over them”. These elements of qualifications supported breadth and depth in the learning experiences that went beyond simply meeting assessment criteria.

In our main study, awarding organisations described having a progression route to a higher-level qualification (from a non-graded lower-level qualification) as motivating students to demonstrate performance beyond the standard associated with just passing (the lower-level qualification). This resonated with what teachers and students referred to spontaneously in this study about the value of higher-level qualifications and pathways within their subject area. Students in particular explained that they were motivated and engaged to pass their qualification to move onto the next level. Teachers in the Hairdressing_L2 qualification agreed that, with the cohort’s progression through the pathway, they often could identify those students who demonstrated the “attendance”, “behaviour” and “skills” needed for the next level of study. These students often met the demands of the course, rather than superficially completing teaching and assessment tasks. Relatedly, students moving into employment were also motivated to learn all aspects of their course to ensure that they would be competent in the workplace. 

It builds you know like your engagement because you want to learn everything. You want to do it. You can’t wait to actually get out there and start doing it on clients. Student_Hairdressing_L2

Undue assessment burden

According to criticisms in the literature, the mastery requirement in CASLO qualifications forces students to spend a great deal of time being assessed and documenting their assessments, resulting in a very burdensome assessment process. 

Five awarding organisations in our main study acknowledged the potential relevance of this problem to their qualification. Mitigations to reduce the chances of this problem occurring involved putting in place support, guidance and quality assurance processes, providing teaching inputs and supporting learning and encouraging holistic approaches to assessments. Positive attitudes of students and teachers were also described as a protective factor which further reduced the risk of undue assessment burden.

In this stakeholder study, focus group discussions unearthed aspects of the 4 qualifications which were motivating and engaging for teachers whilst recognising trade-offs between teaching, assessing and the volume of administration surrounding assessment. A large number of teachers shared positive sentiments and were positively disposed to teaching and assessing in their qualifications. Several teachers described assessments specifically as a “benchmark of seeing success” following teaching and learning. 

More than half of all teachers and students described burden associated with documenting assessments that stems mainly from the number and continuous nature of assessments. For teachers, assessment burden was a consequence of documenting judgements and providing students with feedback. Some teachers commented that they recognised the value and importance of quality assurance processes, as they often formed opportunities to discuss and exchange approaches and practices with other professionals, within and across specialisms. However, quality assurance processes also added to workload which reduced the time available for teaching and learning. It is important to note that not all teachers were provided with protected time to fulfil quality assurance activities and it was, therefore, challenging to carry out such activities during their working day. 

I think it comes back to that same point, the volume of what the students have to do. In terms of the balance of teaching and learning and assessing, the volume of what they have to do makes all of that difficult. My biggest issue is teaching and learning and assessing in comparison to admin. That’s my biggest issue, the amount of admin is disproportionate to the time taken for teaching, learning and assessing. Teacher_Business_L3

For several teachers, ongoing marking requirements of the continuous assessment model were also described as a source of burden. A number of teachers explained that students who had been out of education for a significant period of time, or those with additional language needs, required feedback on “grammar, spellings, sentence structure” in addition to feedback related to assessed criteria. This increased the time taken to mark and provide feedback, adding to the burden on teachers.

For other teachers, the scale of feedback required because of the size of cohorts was at times burdensome, even when using guidance from awarding organisations to streamline feedback. To reduce the burden of the assessment process, some Teaching_Support_L2 teachers shared that they had been advised by external quality assurers to adopt less burdensome assessment methods such as “oral questions”. This appeared to be effective for some teachers but did not seem to be common practice across participants in this study.

For students in this sample, continuous assessment was reported to be both motivating due to the continuous feedback received but also burdensome due to the continuous pressure to perform well. In terms of supporting motivation and engagement, teachers viewed the continuous assessment approach to provide regular opportunities to identify students who were not engaging, were falling behind, or those who were not on track to finish on time. This then enabled teachers to, in some cases, offer additional sessions, including summer schools and extra workshops to support students. An example of this was described by Hairdressing_L2 teachers, who shared that they often run workshops with students across qualification levels to bolster learning and to provide opportunities to learn or embed core skills. Teachers explained that sometimes disengagement stemmed from course demands, but the continuous assessment model usually allowed them to detect this earlier rather than later in the course.

In similar discussions in the Business_L3 qualification focus groups, teachers explained that there were instances where the drive to do well in each assessment resulted in students experiencing high levels of pressure and/or producing large amounts of work beyond the requirements of assessment criteria. Teachers explained that they were able to recognise such cases and intervene early enough to ensure that students were appropriately supported during their course. 

In other instances, the process of collecting evidence for assessment purposes was used to motivate students. In the Teaching_Support_L2 qualification in particular, teachers highlighted the value of portfolios which could be used by students to show their learning journey to future employers.

It’s like a nice creative journey and I do say to the students, you can keep this at the end of the year because when you do go for these interviews, this is your evidence of how you have supported the main teacher. So, just having that portfolio and, like you said, with the other triangulations of the assessments, that you’re doing that portfolio, they’re really proud of that and it’s really good to have and show off. Teacher_Teaching_Support_L2

Participants also shared that some students studying more than one qualification that operates a continuous assessment model experienced having assessments scheduled in parallel, which resulted in high levels of burden. Relatedly, participants also explained that resubmission opportunities at times coincided with other assessment events, increasing burden on students. They recognised that completing resits once students had moved onto a different unit created a tension as there was no designated time available to complete resubmissions once the delivery of the other units had started. More generally across qualifications, the ongoing demands of continuous assessments were suggested by many to be higher than taking a single terminal assessment, as each assessment contributed equally to the final grade or to passing the qualification.

If you’ve not passed an assignment first time and you’ve been given a resubmission, but then you’re on with another unit that’s got an assignment coming out, that causes the problems. That then causes the issues because you’ve got students that didn’t achieve what they needed to do the first time, so they’ve got a resubmission opportunity to get it doing, but they’re actually doing another unit at that point and another assignment is due out. So you’re looking at possibly three assignments for each unit, the amount that you’ve got to cover and they’ve got to complete, there’s not time to spread those out enough and that’s when you get a problem. Teacher_Business_L3

Demotivation and disengagement

Critics have identified certain specific negative impacts from heavy assessment burden in CASLO qualifications. These include not spending enough time teaching and learning and experiencing demotivation or disengagement from learning. Furthermore, the mastery model requirement to achieve every learning outcome can also be demotivating, particularly when a student begins to fall behind, potentially leading to non-completion. 

Four awarding organisations in our main study acknowledged the relevance of this problem to their qualifications. Seven awarding organisations did not entirely recognise this potential problem and an additional 2 did not recognise it at all. Mitigations put in place to reduce the risks of this problem arising included awarding organisations offering teaching inputs and supporting learning, building in specific qualification and assessment design features, promoting contextualisation and relevant assessments, providing centres with support and guidance, and ensuring the implementation of quality assurance processes.

In this stakeholder study, the potential for demotivation and disengagement from learning (in response to heavy assessment burden) was recognised by some participants. However, a large number of teachers spoke about the strengths of e-platforms, designed by awarding organisations to mitigate this challenge by providing a visual display of student progression. For instance, Teaching_Support_L2 teachers mentioned e-portfolios that enable student progress to be tracked and viewed. Similarly, teachers in the Hairdressing_L2 qualification described an online platform used to allow students to see their progress (and that of their peers) as a percentage as the work that they have already completed through the course. These e-tools were seen as more user friendly and motivating compared to navigating through tangible paper-based “big portfolios”. In the Teaching_Support_L2 qualification, where it is possible for students to work through units at their own pace, some teachers reported that students can sometimes be motivated to move quickly through the units, as a result of seeing their own progression.

Students recognised that, in some cases, the workload can differ within the cohort depending on one’s individual pathways and specialisms. Teacher support and “action plan[s]” which set out the targets for each unit were found to be useful for helping students to manage their time and ensuring all assessment criteria are met in the time available. Students also found teacher feedback reassuring in letting them know that they are on track. From the perspective of teachers, providing feedback was described as a burdensome activity in many instances, as it needed to be balanced with other roles and commitments within their college, often within limited spaces of time.

A small number of teachers highlighted that the requirement for students to repeatedly evidence criteria could at times negatively impact student engagement. More specifically, it was suggested that demonstrating the same criteria across different assessments could lead to students becoming “bored”. Consequently, teachers emphasised the need to maximise the use of holistic approaches across assessments to minimise repetition.

You’re asking the same questions in a lot of the exams where they could either do a cross-knowledge unit or they could put it all into a health and safety unit to cross that. But the students, they get a bit fed up, why are we repeating it again, why do we have to think about it again? It’s like, we know you know the answers but again we’re confirming it, but sometimes when they’ve had to confirm it three or four times in different assessments, they get a little bit bored of it as well. Teacher_Hairdressing_L2

Discussion

In our main study, we asked awarding organisations to respond to criticisms identified in the academic literature related to CASLO qualifications. More specifically, we asked awarding organisations whether they recognised the risk of potential problems in their own CASLO qualifications, and if they did, whether they put in place mitigations to reduce the risks of specific problems occurring. Awarding organisations recognised a number of potential problems described in the academic literature but explained that they are able to mitigate these risks through a range of processes and protective factors. 

To further understand how well mitigations and protective factors described by awarding organisations guard against the potential problems identified in the literature, this study aimed to triangulate the views of awarding organisations with those of wider stakeholders. In focus groups with teachers, students, employers and HE recruiters, we asked about experiences of CASLO qualifications in addition to more specific questions about mitigations described by awarding organisations. 

This triangulation of awarding organisation and stakeholder views was particularly important as it was not practical nor possible to conduct a large-scale evaluation of CASLO qualifications in this study. Nor was it feasible to ask stakeholders to engage directly with all of the criticisms raised in the academic literature, without, at the very least, establishing a common understanding of these criticisms to form a base for future discussions. While not perfect, the methodology does provide us with a relatively independent yardstick against which to validate the views expressed by awarding organisations in our main study. 

The findings from this study revealed that stakeholders hold mainly positive views related to the mitigations and protective factors put in place by awarding organisations. The aligned views of awarding organisations and stakeholders presented in the results section of this report suggests that most mitigations and protective factors built into qualifications appear to be operating as suggested. 

Awarding organisations who participated in our main study were asked whether they recognised a number of assessment problems potentially relevant to their CASLO qualifications. Some of these problems, outlined in the results sections of this report, were recognised by awarding organisations and were often mitigated through a package of activities or processes throughout the qualification life cycle. 

In focus group discussions in this stakeholder study, a commonly mentioned mitigation of assessment related problems, which was viewed particularly favourably by stakeholders, was the scope to contextualise and personalise assessments. This often happened through assessments conducted in real world situations or through project-based assessments. For many teachers and students, this added validity to the assessment process as students could demonstrate their skills and knowledge in industry relevant contexts. As a result, this reduced the risk of problems such as poorly conceived assessment tasks and inaccurate assessor judgements.

Alongside this mitigation, quality assurance processes such as standardisation events were described by stakeholders as effective in upholding quality assessments. A large number of teachers indicated that cross centre quality assurance events in particular served multiple important purposes, varying from deepening understanding of assessment criteria to learning about different approaches that can be used for delivery. The sharing of good practice and alternative approaches to delivery through these events, within a community of practitioners, seem to encourage teachers to adopt holistic assessment practices rather than assessing atomistically. 

Several teachers recognised the importance of their professional expertise within the assessment process. Professional and occupational expertise were seen as important for reducing the risks of problems such as atomistic judgements, and lenience and malpractice in assessments. It was also seen as integral to effective teaching and learning, particularly enabling teachers to adapt the curriculum based on their current experience of industry practices.

Participants in this study also described a small number of assessment related mitigations which did not function as effectively as suggested by awarding organisations. These included quality assurance activities involving external quality assurers. Across qualifications in this sample, teachers mentioned occasional inconsistencies in the advice and support received from external quality assurers. These included perceived differences in the types or amount of evidence required by different external quality assurers to demonstrate the achievement of certain assessment criteria and perceived differences in advice offered when centres were allocated new external quality assurers. These concerns emerged across qualifications, with some teachers suggesting that standardisation of external quality assurers would likely support more consistency in the advice offered to centres in the future.

Another area of perceived misalignment involved the consistency of language across qualification documentation. A handful of teachers felt that the language used in some resources, such as grading criteria and assignment briefs, did not always align with the language in assessment criteria. From their perspective, this reduced the clarity and usefulness of these resources. 

In our main study, we asked awarding organisations whether they recognised several teaching, learning and delivery problems described in the academic literature. Similarly to the potential assessment problems, awarding organisations recognised some of problems more than others, and described packages of mitigations and protective factors put in place to reduce the risks of these problems arising. In this stakeholder study, a widely recognised mitigation discussed by teachers and students related to opportunities to contextualise teaching programmes not only to suit the needs of students and their local context, but also industry. This mitigation supported teachers in ensuring qualifications were sufficiently relevant, teaching programmes were coherent, and learning was holistic rather than fragmented or incoherent. For students, this mitigation offered a sense of autonomy as they could explore areas of most interest to their students, which was often additionally motivating. This mitigation, that was made possible to implement through broad learning outcomes, underpinned many of the discussions within stakeholder focus groups. It appeared to go beyond reducing the risk of potential problems and instead added value to teaching and learning by allowing qualifications to meet the needs of diverse groups of students.

However, a small number of stakeholder comments indicated that there were also some perceptions of a narrowness in knowledge and skills included in qualifications when considering what students needed to know to enter industry post qualification. Teachers often remedied this through adding in additional content which they deemed important for preparing students for their next steps, which was also what the AOs suggested as a mitigation for the potential problems of local and personal irrelevance, and hard-to-pin-down content getting missed.

There was also noticeable alignment between awarding organisations and teachers in relation to the perceived need for occupational and professional expertise amongst practitioners when teaching CASLO qualifications. Teachers highlighted instances of using their expertise to shape teaching programmes and to share industry relevant knowledge and experience. This mitigation also enabled teachers to ensure learning was holistic and integrated due to the synergy between their knowledge and understanding of the sector and knowledge of qualification delivery. Recency of occupational and professional expertise was raised as an important component within discussions on this topic. There seemed to be a sliding scale in terms of the impact that occupational and professional expertise could have on knowledge transfer, but also on promoting deep rather than superficial learning, based on how recently teachers had worked in industry. For some, continued involvement in industry as a part-time professional was fundamental to effective delivery of CASLO qualifications, whilst others valued maintaining knowledge and skills through continued professional development activities solely as a teacher and assessor, rather than as a practitioner.

Although awarding organisation and stakeholder views on mitigations related to teaching, learning and delivery were largely aligned, there was some discord in relation to views on undue assessment burden. Across qualifications, teachers and students largely perceived a level of burden to result from requirements for continuous assessment. This burden was demotivating for some students despite their positive attitudes, according to teachers. However, this burden was not described as being significant enough to outweigh the strengths of the qualifications or aspects of the qualifications valued by participants (such as contextualised holistic teaching and assessment). Undue assessment burden was less of a focus in stakeholder discussions as compared to the positive views shared on mitigations and protective factors described in the results section of this report. 

Reflecting more generally on the focus group discussions as a whole, teachers and students in this study valued the flexibility and transparency built into their CASLO qualifications. There were many positive sentiments shared surrounding how each CASLO qualification functioned in a range of contexts, meeting the needs of different student groups. Participants also emphasised the purpose that these qualifications served in allowing students to progress into employment, higher education and apprenticeships. Focus group discussions were broad and dynamic, reflecting the diversity of each qualification, and often explored wider qualification issues, in addition to the CASLO specific features described in this report. The views shared in this report, and our main study, highlight the nuanced and insightful experiences within and across qualifications adopting the CASLO approach.

The focus on stakeholder responses to CASLO-specific issues in this report also sheds light on potential problems with the CASLO approach that are most relevant to users of current CASLO qualifications, based on our taxonomy of potential problems. Without a significant amount of prompting, participants in this study discussed many of the issues raised by awarding organisations in an organic way, with many mitigations and protective factors viewed positively. Although this was a small-scale and somewhat opportunistic research project – so we are unable to reach strong conclusions concerning the effectiveness of the proposed mitigations and protective factors – the views that were expressed independently by these stakeholders provide us with no strong rationale for questioning insights from our main study (report B).

References

Braun, V. & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101.

Newton, P.E. & Lockyer, C. (2022). How ‘CASLO’ qualifications work. Coventry: Office of Qualifications and Examinations Regulation.

Nowell, L.S., Norris, J.M., White, D.E. & Moules, N.J. (2017). Thematic analysis: Striving to meet the trustworthiness criteria. International Journal Of Qualitative Methods, 16(1), [not paginated].

Appendix: Focus group questions

Teacher focus group questions

Opening questions  

To begin with we’d really like to hear about your experiences of [name of qualification]

1. Is it well-suited to the cohort, context, progression routes of your students?

2. How does this qualification compare to teaching and assessing a more traditional qualification, like a GCSE?

Section A: Teaching, Studying and Learning

We’re interested in hearing more about how you teach , how your students study, and how your students learn throughout the course. 

Is there anything about teaching, studying or learning in this qualification that seems to work particularly well for you and your students? 

3. Is there anything about teaching, studying or learning in this qualification that seems to work less well for you and your students?

4. Is there anything else about how you teach or students study or learn on the course that’s worth mentioning?

Section B: Assessment 

Next, we would like to explore your experiences of assessing students taking [qualification name].

5. Is there anything about the approach to assessing that seems to work particularly well for you and your students?

6. Is there anything about the approach to assessing that seems to work less well for you and your students?

7. Is there anything else about assessments on the course that’s worth mentioning?

Section C: Delivery Issues 

We would now like to hear about the qualification from a practical point of view.  

8. Does the amount of time you spend assessing for this qualification encroach too much on the teaching and learning time?

9. Does the need for achieving all assessment criteria affect student motivation or engagement?

Section D: Next Steps and Final Thoughts 

In this final section, we would like to hear how well prepared your students will be for their next steps. 

10. Does this qualification support your students in developing the knowledge, behaviour and/or skills needed to do well in the sector or to move onto their next steps?

Do you have any final thoughts on the teaching, assessment or delivery of this qualification?  

Student focus group questions

To begin with we’d really like to hear more about your experiences of [name of qualification] 

1. Do you enjoy the course/qualification overall? What do you enjoy most about the course?

2. How does this qualification compare to taking GCSEs or other types of qualifications?

Section A: Teaching, Studying and Learning

We’re interested in hearing more about how you’re taught, how you study, and how you learn throughout this course.  

Beginning with how you’re taught: 

3. Is there anything about the approach to teaching that seems to work particularly well for you?

4. Is there anything about the approach to teaching that seems to work less well for you?

5. Is there anything else about how you are taught on the course that’s worth mentioning?

Next, if we think about how you study and learn on your course: 

6. Is there anything about the approach to studying and learning that seems to work particularly well for you?

7. Is there anything about the approach to studying and learning that seems to work less well for you?

As a final question in this section, when thinking about your overall learning:  

8. Do you feel that the approaches to teaching and studying may have helped you to learn, or may have hindered your learning at all?

Section B: Assessment 

Next, we would like to explore your experiences of being assessed on your course.  

9. Is there anything about the approach to being assessed that seems to work particularly well for you?

10. Is there anything about the approach to being assessed that seems to work less well for you?

11. Is there anything else about how you are assessed on the course that’s worth mentioning?

Section C: Delivery Issues  

We would now like to hear about the qualification from a practical point of view.  

12. Does the amount of time you spend completing assessments for this qualification encroach too much on the teaching and learning time?

13. Does the need for achieving all assessment criteria affect your motivation or engagement?

Section D: Next Steps and Final Thoughts

In this final section, we’d like to hear how well prepared you feel for your next steps once you complete your qualification.  

14. What has been the most valuable part of your course so far?

Do you have any final thoughts on the teaching, assessment or delivery of this qualification? 

Employer focus group questions

To begin with we’d really like to hear your general thoughts or experiences on the qualification or students who have taken this qualification 

1. Are there any particular strengths of the qualification or employees who have taken the qualification?

2. Are there any particular weaknesses of the qualification or employees who have taken the qualification?

3. How do employees who have completed [qualification name] compare to other employees who have completed the same qualification, are there any fundamental differences?

4. How do employees who have completed [qualification name] compare to other employees who have come through different routes, are there any fundamental differences?

5. Do employees who come with this qualification demonstrate skills or abilities that are relevant to your context?

Next, we would like to hear about the knowledge and skills of students who have completed [qualification name] 

6. Do the employees who come through this route have the knowledge and skills that you’d want them to have, at the level you’d want them to have?

• If they don’t then what do they lack?

• Is it something that’s particular to students who come through this route?

Prompts: 

a. Do employees come with a satisfactory level of knowledge and skill in the subject?

b. Do employees come with satisfactory skills?

c. Do employees come with a satisfactory attitude to learning (also work)?

d. Is there anything about the qualification (teaching, learning or assessment) that explains why employees demonstrate or don’t demonstrate the knowledge and skills that you want them to have?

e. Do you feel that the qualification provides enough time and support to develop knowledge and skills?

f. Is there enough input from employers during on-programme study – e.g. opportunities for students to gain experience through placements?

As a final question, we would like to hear your thoughts on how well-prepared students who have completed [qualification name] are for their next steps.

7. In your experience, are employees who come through this route well prepared for employment or the pathway they are on?

Prompts: 

a. Do you think that the level of preparedness is different when comparing them to employees who have come through other routes?

b. Is this route a preferred route from an employer perspective? If so, what makes it so? If not, why not?

Do you have any final thoughts on the quality or value of employees who have completed [qualification name] that you would like to share?  

Higher education recruiter focus group questions

To begin with we’d really like to hear your general thoughts or experiences on the qualification or students who have taken this qualification 

1. Are there any particular strengths of the qualification or students who have taken the qualification?

2. Are there any particular weaknesses of the qualification or students who have taken the qualification?

3. How do students who have completed [qualification name] compare to other students who have completed the same qualification, are there any fundamental differences?

4. How do students who have completed [qualification name] compare to other students who have come through different routes, are there any fundamental differences?

5. Do you notice any marked differences between students that come with this qualification in the level of skill or ability to adapt to your context?

Next, we would like to hear about the knowledge and skills of students who have completed [qualification name] 

6. Do the students who come through this route have the knowledge and skills that you’d want them to have, at the level you’d want them to have?

• If they don’t then what do they lack?

• Is it something that’s particular to students who come through this route?

Prompts: 

a. Do students come with a satisfactory level of knowledge and skill in the subject?

b. Do students come with satisfactory study skills?

c. Do students come with a satisfactory attitude to learning (also work)?

d. Is there anything about the qualification (teaching, learning or assessment) that explains why students demonstrate or don’t demonstrate the knowledge and skills that you want them to have?

e. Do you feel that the qualification provides enough time and support to develop knowledge and skills?

f. Is there enough input from employers during on-programme study – e.g. opportunities for students to gain experience through placements

As a final question, we would like to hear your thoughts on how well-prepared students who have completed [qualification name] are for their next steps

7. In your experience, are students who come through this route well prepared for University or the pathway they are on?

Prompts: 

a. Do you think that the level of preparedness is different when comparing them to students who have come through other routes?

b. Is this route a preferred route from an HE perspective? If so, what makes it so? If not, why not?

 Do you have any final thoughts on the quality or value of students who have completed [qualification name] that you would like to share

1. See report A: ‘Responding to Criticisms of the CASLO Approach (Report A)’. ↩

2.  See report B: ‘Responding to Criticisms of the CASLO Approach (Report B)’. ↩

3.  See report D: ‘Responding to Criticisms of the CASLO Approach (Report D)’. ↩

4.  Unit 2 of the Business_L3 qualification is assessed under supervised conditions over 2 days. On the first day, students receive Part A of their Research Pack under low-control conditions for a 2-hour preparation period with access to a computer and the internet. During this preparatory session, students may prepare summary notes that can be taken into their Part B assessment on the following day. The assessment concludes on the second day with a 3-hour supervised Part B written examination that is conducted in medium‑level control conditions. ↩

5.  National Occupational Standards were developed by Sector Skills Councils to set out the skills and knowledge required for specific roles. Although no longer used in England, they are still used by the devolved administrations.  ↩

6.  All of the qualifications in this study involved industry placements or work experience. ↩