Independent report

Annual compliance report 2023

Published 12 October 2023

1. Infographic

2. Executive summary

2.1 Compliance with legislative reporting requirements is important. Timely, good quality submissions are essential for several reasons including:

• Supporting better procurement. This is aligned with the intent of the legislation: that use of the data collected will ensure that good value for money is obtained in government expenditure on qualifying contracts, and that persons who are parties to qualifying defence contracts are paid a fair and reasonable price under those contracts.
• Encouraging utilisation. As indicated in the SSRO’s data strategy, data is more likely to be utilised if it is relevant, comparable and reliable.
• Reducing the need for the MOD seeking alternative sources of information. This is unnecessary and duplicative, as good quality, timely data should be available in a standardised format under the regime.
• Finalising contracts. Improved reporting reduces delays to, and helps the resolution of, any Final Price Adjustment (FPA) calculations.
• Effective management of contracts and suppliers. The ability to analyse cost variances and outturn profit, prepare estimates, challenge costs or deliver strategic planning is based on having a reliable set of complete data.

2.2 Our compliance work identifies that there has been some improvement in the timeliness and quality of submissions, but there is still more to be done in this area by contractors and this will be particularly important as reporting requirements are updated following the MOD’s review of the legislation, which is proposed to take effect from 2024/25. There is also potentially more enforcement action, covering both compliance and penalty notices, for the MOD to take in appropriate cases. We will continue to work with the MOD to provide management information on cases where report submissions are late and/or of a poor quality and we expect to see further improvements in 2023/24. In the absence of such progress and where we identify material levels of non-compliance, we will carefully consider what further steps we can take to drive the necessary change.

2.3 We highlight in this report the work that has been driven by the MOD to increase the review and use of supplier overhead reports. Additional effort is required from contractors, upon whom the reporting obligations lie, to pay greater attention to the quality of reports prior to submission and for an increase in the level of MOD review of individual contract reports. The assessment of the quality of information can only be properly undertaken once the MOD, as party to contract, has reviewed the information for accuracy.

2.4 We have started a programme of work with the MOD to support its utilisation of reported data and this in turn should ultimately increase the quality of the data in the system. We are also working to develop changes to our Defence Contract Analysis and Reporting System (DefCARS) which will allow for easier upload of information in future years. Our work in these areas will facilitate better utilisation of DefCARS data and provide insights to support procurement decisions and contract management by engaging directly with selected MOD decision making teams to identify users of data and their needs, providing hands-on support including on compliance and data use. We will proactively promote example Management Information (MI) that helps the MOD teams seeking to use MI in DefCARS and generate analytical requests which our analysis team can then deliver to the MOD in Power BI.

2.5 Our published Compliance and Review Methodology describes how we discharge our function under section 36(2) of the Defence Reform Act 2014 (the Act) to keep under review the extent to which persons subject to reporting requirements are complying with them, and how we use the reported information to inform our function under section 39(1) of the Act to keep under review the provision of the regulatory framework (the framework).

2.6 We examine Qualifying Defence Contracts (QDCs) and Qualifying Sub-Contracts (QSCs) entered into between 1 April 2022 and 31 March 2023 and provide a factual commentary on the data collected, highlighting emerging themes and areas where improvement is required. We consider the timeliness and quality of the reports submitted by contractors and identify the extent to which reporting requirements are met, setting a KPI in our corporate plan that 75 per cent of submissions are made by contractors on time and are of good quality.

Timeliness of submissions

2.7 The proportion of reports submitted on time into DefCARS, at 65 per cent, remained below our target in 2022/23. The overall timeliness for contract reports has stayed at 65 per cent as per 2021/22 but increased to 68 per cent for supplier reports. Most of the reports that are not submitted on time are subsequently submitted, with 89 per cent of the contract reports and 82 per cent of the supplier reports received, even though they may be late.

2.8 Around one third of contracts were added to DefCARS more than one month after becoming QDCs/QSCs, in part because of late notifications to us from both industry and MOD delivery teams of contracts being entered into. In effect, this meant that the statutory deadline for submissions had already lapsed for around one third of new contracts before these contractors had commenced preparing their initial reports. We are working with the MOD to improve the timeliness of contract notifications.

2.9 The MOD issued 82 compliance notices against late reports resulting in 68 submissions. This is a significant increase when compared to the 12 notices issued in 2021/22 and indicates increased focus by the MOD on improving timeliness of reporting. Nine contractors had not complied with 14 of the notices issued, but no penalty notices were issued as a result.

Quality of submissions

2.10 At 48 per cent, the quality of submissions was also below our target in 2022/23, with 62 per cent of contract reports and 7 per cent of supplier reports correct first time.

2.11 There has been a significant increase in MOD reviews of supplier reports, which has resulted in a larger number of quality issues being identified within such reports. A high proportion of these issues relate to contractors incorrectly reporting basic information and inconsistencies within the reports. While there has been a steady increase in quality of contract reports being submitted, it is difficult to identify the extent to which the quality of report submissions is improving, because relatively few reviews are being undertaken of contract reports by the MOD. Improvements by the MOD in the review process of contract reports would lead to a better understanding of the quality of those report submissions.

Recommendations

2.12 We have made several recommendations in the report which will increase compliance, including the need:

• for the MOD to develop the ‘acquisition pipeline’ to better identify new qualifying contracts;
• for an increase in the number of compliance notices issued against missing initial reports;
• for ongoing consideration as part of the MOD’s review of legislation to improve the identification of expected supplier reports;
• to continue the SSRO’s work on data utilisation with more MOD teams;
• for additional contract report reviews by the MOD, along with a continuation of its development of processes to review supplier reports; and
• for contractors to take greater care when completing report submissions, taking note of validation warnings within the system, especially with basic information relating to the contract.

3. Analysis of the timeliness of submissions

Significant messages

• Overall timeliness, at 65 per cent, for contract and supplier reports in 2022/23 remains below our KPI target of 75 per cent. Timeliness remained at 65 per cent for contract reports^{[footnote 1]} and increased to 68 per cent for supplier reports^{[footnote 2]}. Existing contractors initial report timeliness at 37 per cent is lower than that achieved by new contractors at 41 per cent.
• Only a quarter of contracts added to DefCARS in 2022/23 used the potential QDC feature. Contracts had already missed the statutory deadline for making the submissions in around a third of cases before getting access to the system.
• We identified six QSCs (40 per cent of the QSCs entered into in the year) that were not notified to us through the correct channels.
• Industry and the MOD both need to raise awareness internally of the need to notify both the SSAT and the SSRO of qualifying contracts and contractors to follow the notification process for QSCs.
• The MOD issued 82 compliance notices against late reports since the last compliance report. This is a significant increase to the 12 notices issued last year and resulted in 68 (83 per cent of those with a notice) additional submissions demonstrating that timely action by the MOD improves compliance with reporting requirements.

3.1 We set a KPI in our corporate plan that 75 per cent of submissions are made by contractors on time. In 2022/23, at 65 per cent overall, timeliness remained below our target.

3.2 Table 1 separates submission timeliness between contract and supplier reports and provides timeliness performance over the previous five years^{[footnote 3]}. The overall timeliness has remained the same for contract reports but has increased by five per cent for supplier reports in 2022/23.

Table 1: Contract and supplier report submission timeliness 2018/19 - 2022/23

Report type	2018/2019	2019/2020	2020/2021	2021/2022	2022/2023
Contract reports	70%	62%	67%	65%	65%
Supplier reports	71%	72%	65%	63%	68%

3.3 Figure 1 shows the timeliness of reports in 2022/23 by report type.

Figure 1: Proportion of reports submitted on time in 2022/23, by report type

3.4 DefCARS currently holds contract information in relation to qualifying contracts with a collective total estimated contract price of over £83.6bn^{[footnote 4]} and overhead information in relation to 50 Qualifying Business Units. There were 208 additional reports expected in 2022/23 compared to the prior year. Figure 2 details the increase in the expected number of reports by type compared to the prior years, showing an increase in volume of expected reports.

Figure 2: Total number of expected reports by year, by report type

3.5 There is a marked difference in the timeliness of submissions by report type For 2022/23 our analysis showed that:

• Initial report submission timeliness reduced to 36 per cent on time compared with 43 per cent for last year.
• Update report submission timeliness was the same as last year at 76 per cent and completion report submission timeliness was similar to last year at 60 per cent (59 per cent 2021/22).
• There was a 7 per cent improvement in overhead report timeliness at 81 per cent (74 per cent 2021/22).
• The majority of strategic reports continue to be submitted late, with timeliness reducing to 26 per cent (34 per cent 2021/22).

3.6 The SSRO has limited visibility of when a new QDC or QSC is entered into. We frequently receive late notification or late confirmation of new QDCs and QSCs, impeding our ability to support contractors to submit their initial reports within the timescales set by the legislative framework. In 2022/23 around one third of contracts were added to DefCARS more than a month after being entered into, meaning that submissions were already late before the contract was added to DefCARS. We rely on the MOD or the contractor to inform us of potential or new contracts and add these to DefCARS after confirmation from contracting parties. The MOD commercial toolkit describes the need for a MOD Delivery Team to inform the MOD’s Single Source Advisory Team (SSAT) that they are awarding a qualifying defence contract before the contract is entered into or, as a minimum, as soon as the contract is entered into. It would be helpful for contractors to ensure that their internal guidance and processes are aligned to this for notifications that need to be made to the SSRO.

3.7 A quarter of contracts added to DefCARS in 2022/23 used the ‘potential QDC’ feature allowing them to start preparing their initial report submissions prior to entering into a contract. It is important for the MOD and contractors to raise awareness internally of the need to notify the SSAT and the SSRO of both potential and new QDCs and QSCs. Some contractors are unaware of the ‘potential QDC’ facility within DefCARS until it is too late.

3.8 New contractors submitted 41 per cent of their initial reports on time and an additional 33 per cent were submitted late. Existing contractors submitted 37 per cent initial reports on time and an additional 48 per cent were submitted late, as at the cut-off date. Whilst new contractors have not met the KPI, it is primarily existing contractors who contribute most to the fall in the timeliness score of initial reports, with 78 per cent of late initial reports.

The QSC notification process needs to improve

3.9 Contractors making positive QSC assessments must notify the sub-contractor and the MOD of the positive assessment in accordance with Regulation 61(3) and (6), in a timely manner to avoid any delay in the time a qualifying sub-contractor has available to prepare and submit their initial reports. The SSRO developed assessment notification templates and guidance in 2022 to support primary contractors in complying with their notification obligations.

3.10 In 2022/23, we identified six sub-contracts listed within report submissions as being positively assessed as QSCs for which no record was held in DefCARS. The SSRO is responsible under the Act for keeping an up-to-date record of qualifying contracts, not actively trying to identify these. Submissions for the QSCs have been delayed as sub-contractors did not contact us to obtain access to DefCARS following notification of a positive assessment, or because primary contractors failed to make the positive assessment notification.

3.11 We also identified that sub-contracts were being incorrectly reported as QSCs by primary contractors who reported a ‘Yes, is a QSC’ within DefCARS, but in circumstances where the contracts were let competitively. We are currently waiting for two primary contractors to confirm whether the prospective sub-contracts they have reported are in fact QSCs.

3.12 The legislation does not include an express timeframe within which the primary contractor is required to undertake the QSC assessment, nor notify the prospective sub-contractor, the MOD and the SSRO. It is clear that the assessment must be undertaken where a sub-contract is proposed to be entered into. Failure to notify the sub-contractor and the MOD of a positive QSC assessment is one matter on which the MOD can issue a compliance notice. To date, there have been no compliance notices issued for such a contravention. In our 2017 recommendations and in our 2021 recommendations, we suggested that the assessment and notification timeliness should be made explicit in the legislation. In the April 2022 Command Paper^{[footnote 5]}, the MOD signalled support for the recommendation confirming that it would work with the SSRO to develop the necessary legislative changes.

Most report submissions are eventually submitted

Significant messages

• The majority of reports are eventually submitted. 89 per cent of the contract reports and 82 per cent of the supplier reports were subsequently submitted even though they may have been late. Strategic reports, however, remain outstanding for the most part.

• Information gaps created by late and outstanding report submissions may impact the MOD’s ability to manage contracts and suppliers, analyse cost variances and outturn profit, prepare estimates, challenge contractor costs or to inform strategic planning. These gaps also leave contractors open to compliance and penalty notices.

• Late and outstanding completion reports could financially impact the MOD or the contractor through delayed and unresolved FPA calculations.

3.13 Most of the reports that are not submitted on time are subsequently submitted. A total of 89 per cent of the contract reports and 82 per cent of the supplier reports expected during 2022/23 were received by the cut-off date for this report of 1 July 2023. Figure 3 shows how reports are submitted over time by each of the report types.

Figure 3: Analysis of the time taken to make report submissions, by report type, for reports due in 2022/23 by 1 July 2023

3.14 Submission of overhead reports, update reports and completion reports are the timeliest. Submission of strategic supplier reports and initial reports remain the least timely (as reported last year). Figure 4 provides the breakdown by report type. There were slightly more initial reports outstanding than strategic supplier reports, a total of 71 per cent (44) of the strategic supplier reports expected for 2022/23 were not received at the cut-off date.

Figure 4: Breakdown of contract and supplier report submissions for 2022/23 by 1 July 2023

3.15 Late and outstanding report submissions create gaps in the data set which will impact the MOD when using the data and the SSRO when creating management information for the MOD. The MOD uses the data to manage contracts, identify issues to discuss with contractors, for comparison exercises, benchmarking activities, to provide further insight, support strategic planning and to understand capacity and risks. There can also be financial impacts for both the MOD and the contractor through delayed and unresolved Final Price Adjustment calculations and Target Cost Incentive Fee adjustments and the contractor may be subject to compliance and penalty notices from the MOD.

MOD enforcement action works

3.16 The MOD issued 50 compliance notices for contract reports in 2022/23. We are aware that an additional 32 notices were also issued for supplier reports relating to 2022/23. This is a significant increase when compared to the 12 notices issued in 2021/22. Figure 5 details the breakdown of contract compliance notices issued in 2022/23. The legislation imposes a time limit of six months from the relevant submission due date by which the MOD may exercise its power to issue compliance notices for a failure to comply with reporting requirements.

Figure 5: Breakdown of compliance notices in 2022/23 for contract reports

3.17 Thirty-three of the 50 contract compliance notices issued resulted in reports being submitted, which demonstrates that enforcement action works. Twenty-six of the report submissions received were within the notice deadline stipulated in the compliance notice and seven were received after the date set. Four compliance notices were closed by the MOD without receiving the outstanding report submissions as there were changes made to existing information on the system.

3.18 The MOD raised 32 compliance notices against supplier reports in April and May 2023. This action resulted in the submission of 31 overhead reports. We will consider these compliance notices further as part of the 2023/24 compliance report.

3.19 At the cut-off date for the analysis used in this report, nine contractors had not complied with 14 compliance notices issued to them by the MOD. In one of these cases the MOD provided the contractor with an extension to the time for complying as stipulated in the compliance notice. The time limit for issuing a penalty notice is three months after the date specified in the compliance notice by which the steps specified are to be taken. This time limit has expired for four of the 14 compliance notices. No penalty notices were issued in 2022/23.

Table 2: Impact of compliance notices issued in 2022/23

Total number of compliance notices issued	Submissions received within the stipulated timescales	Submissions received after the stipulated timescales	Notice closed following update of existing information	Report submissions remain outstanding
50 Contract	26	7	4	13
32 Supplier	29	2	0	1

Recommendations to improve the timeliness of submissions

3.20 The Single Source Pricing Support (SSPS) team within the MOD has worked to improve compliance with rates and supplier reporting in 2022/23 and we are starting to see evidence of this through an increase in supplier report reviews and compliance notices issued. New processes are being developed and embedded to identify missing supplier reports and to improve timeliness and quality; align the receipt of reports to the rates agreement process; make use of statutory compliance notices where necessary; and perform report reviews that complement those carried out by the SSRO. These processes should continue to develop. Replication of similar resourcing and processes within the MOD for contract reports would likely be of benefit in improving submission timeliness.

3.21 The SSRO continues to develop processes aimed at supporting the timely submission of reports. We encourage the SSAT to add contracts to the system directly rather than via us as this helps to increase the time available for contractors to prepare and submit their reports. We liaise with the MOD on a weekly basis for the purpose of reducing the instances of delays in adding contracts to DefCARS. Where a new contracting company is to be set up in DefCARS, they are sometimes not responsive in providing us with contract details, reducing the time subsequently available to them to submit their initial reports. It is important for the MOD to continue to raise awareness internally of the need to notify the SSAT and the SSRO of both potential and new QDCs and QSCs, and the MOD should consider investment of resource into the development of the ‘acquisition pipeline’ to better identify new qualifying contracts, rather than relying on ad-hoc notifications.

3.22 Initial reports achieved the lowest percentage for timeliness and have the highest number of reports outstanding when compared to other contract reports, but conversely these report types have the lowest number of compliance notices issued against them. These reports record a baseline for what was agreed in the contract. Missing and incomplete initial reports impact the data set available for analysis and restrict the potential use of the data as the audit trail of the baseline set for the costs and profit is missing. Additionally, initial reports set out when the other statutory reports become due. Without this information, the population of future expected reports is incorrect. Contractors must do more to ensure their initial reports are submitted by the statutory deadlines and, in terms of compliance, the MOD should ensure that further attention is given to these types of report. We encourage a greater level of compliance and we will work with the MOD to build upon the progress made with issuing of compliance notices in 2022/23 and the subsequent positive responses from contractors. We provide the MOD with management information on cases where report submissions are late. We will continue to support industry to make the submission of reports straightforward through our on-boarding training, DefCARS demonstrations, helpdesk support, reporting guidance and DefCARS developments. We expect to see an improvement in 2023/24. In the absence of such progress and where we identify material levels of non-compliance, we will carefully consider what further steps we can take to drive the necessary change.

3.23 The SICR is reported at the level of the Ultimate Parent Undertaking (UPU) of the contracting company. Some UPUs are based overseas and much of the work undertaken by the ultimate parent will have little relevance to the pricing or performance of QDCs and QSCs. Legislative restrictions within some countries may also prevent the disclosure of information required in a SICR due to its sensitivity. Some UPUs based overseas submit SICRs by restricting the level of information disclosed to that which is attributable to their UK operations.

3.24 The April 2022 Command Paper issued by the MOD, lists proposed changes to the legislation that would allow, at the MOD’s discretion, for the submission of SICRs at a level below that of the UPU and, in some cases, to exempt the requirement to submit a SICR entirely. If taken forward, we expect that this will increase the timeliness of SICR submissions as those UPUs currently unwilling or unaware of the need to submit a SICR will be more inclined to do so if the content of the report is limited to their UK operations. Our analysis identified that 11 of the 21 outstanding SICR submissions belonged to UPUs based overseas. Additionally, those non-submissions that the MOD considers to be less valuable will be excluded from the baseline of expected reports. The MOD will need to notify the SSRO of such exclusions to ensure our records are kept up to date and we will work with the MOD to ensure that we have a joint understanding of those that have been excluded.

3.25 The Command Paper also indicates that there will be changes to reporting requirements following the MOD’s review of the legislation, proposed to take effect from 2024/25. Contractors will need to ensure that they have systems and processes in place to be able to consider and address these changes and are able to meet reporting requirements going forward.

4. Analysis of the quality of submissions

Significant messages

• Forty-eight per cent of contract and supplier reports were correct first time in 2022/23 which is below our KPI target of 75 per cent. Sixty-two per cent of contract reports and seven per cent of supplier reports were correct first time in 2022/23.
• The MOD has accessed 57 per cent of contract and supplier report submissions which is an improvement on 2021/22.
• There has been a significant increase in the level of MOD reviews when compared to prior years for overhead reports. This has, expectedly, resulted in a greater number of quality issues being identified by the SSPS. This could and should be replicated by the MOD across contract reporting, which has seen a decrease in the level of reviews from the previous year meaning the quality of contract report submissions in terms of the accuracy of information is unclear.
• The quality of the submissions increases to 94 per cent for contract reports and for supplier reports to 61 per cent when taking account of the subsequent resolution of issues raised.
• The MOD has made good progress on the supplier report recommendations listed in the SSRO’s 2021/22 Targeted Review.

4.1 Our assessment of the quality key performance indicator is currently based on a ‘pass or fail’ assessment, so a single error in a report will result in a failure. In line with our compliance methodology we consider a report to be ‘reviewed’ if issues are raised for the contractor to respond to or if the report is set to ‘no issues’, both actions that use the compliance functionality built into DefCARS. MOD and SSRO reviews are undertaken independently. This allows us to provide data based upon whether report submissions were correct first time or following subsequent submissions through the submission of correction reports.

4.2 Our quality KPI for reports that are complete and meet the requirements of the legislation at the first attempt is set at 75 per cent. In 2022/23, at 48 per cent, this was below our target. This is assessed based upon those reviews conducted by both the MOD and the SSRO. Sixty-two per cent of contract reports and seven per cent of supplier reports were correct first time in 2022/23. Figure 6 details correct first time submissions.

Figure 6: Proportion of reports submitted correct first time in 2022/23

4.3 The MOD’s Commercial Toolkit sets out the expectations on MOD Delivery Teams with respect to performing reviews of contract report submissions. As at the cut-off date for this report, the MOD accessed just over 57 per cent of contract and supplier reports in DefCARS in 2022/23, subsequently reviewing 22 per cent of contract reports and 36 per cent of supplier reports.

4.4 The MOD’s review of contract reports has decreased from 29 per cent in 2021/22 but the review of supplier reports has significantly increased from 13 per cent in 2021/22, resulting in an additional 232 issues being raised with contractors when compared to last year. This is a positive change that will improve the quality of report submissions already made, will set expectations for future submissions and explains why the number of supplier reports that were correct first time reduced to 7 per cent from 48 per cent in 2021/22.

4.5 The SSRO reviewed 99 per cent of contract reports and 95 per cent of supplier reports. Report submissions reviewed by the SSRO check for completeness and consistency. The quality of submissions in terms of their accuracy can only be considered by the MOD as a party to the contract but given the low percentages of MOD Delivery Team reviews, it is not possible to draw broad conclusions in relation to the quality of report submissions in terms of accuracy of information. Consistent with our general approach, the remainder of the quality analysis within this compliance report is based upon those report submissions reviewed by either the MOD or the SSRO since this provides a greater population for the analysis.

4.6 It is important to note that the MOD reviews of contract and supplier reports are not the sole determinant of whether the data within a report was used. For example, in our Targeted Review of contract reports, which is considered in more detail below, we identified that the MOD had reviewed five contract reports out of the eight selected. However, all eight reports had been accessed by the MOD with seven accessed multiple times. If there were no issues in the additional three contract reports and this had been recorded using the compliance functionality of DefCARS, this would have improved the level of recorded MOD reviews.

4.7 A total of 80 per cent of contract reports were submitted with no issues raised during 2022/23 and 29 per cent of supplier reports, when considering submissions that had been reviewed by either the SSRO or the MOD.

4.8 During the past four years there has been a steady increase in the number of contract reports without issues raised. There was a decline of 43 per cent in the number of issues raised in 2022/23, but this should not be interpreted solely as quality is increasing given the low level of MOD reviews. The SSRO is also raising less issues as, in accordance with our compliance methodology, we now rely primarily on the results of the automatic checking within DefCARS when reviewing contract reports. The percentage of supplier reports without issues raised has decreased from 60 per cent in 2020/21, to 50 per cent in 2021/22 and to 29 per cent per cent in 2022/23. This decrease in supplier report quality is to be expected as the MOD increases its reviews of supplier reports and identifies issues of accuracy.

4.9 During 2021/22 the SSRO carried out Targeted Reviews on four sets of supplier overhead reports submitted in DefCARS and provided the MOD with a report containing seven recommendations. This work was summarised in last year’s compliance report. Subsequent to that Targeted Review exercise, the MOD has invested in external support to commence work on the recommendations we made and to improve compliance within rates and supplier reporting. MOD team members dealing with the supplier reports are now trained in the use of DefCARS and raise queries to contractors within the system. The SSRO and SSPS are working together by sharing information on the identification of Qualifying Business Units (QBUs) and the timeliness of supplier reports, which has resulted in an increase in compliance notices being raised as described earlier in this report.

4.10 The April 2022 Command Paper proposes reforms to the legislative framework including the simplification of the definition of a QBU in the legislation. It explains how the criteria used to determine whether a supplier has one or more QBUs can be technically complex and gives rise to uncertainty about whether a supplier unit is a QBU or not. This makes it difficult to identify when supplier reports are due and remains an issue for the MOD to consider as part of its current Review of Legislation.

4.11 The SSRO will continue its work in reviewing and identifying data quality issues in supplier reports in a way that complements the MOD’s reviews. Defined processes have been developed by the MOD to review supplier report submissions for accuracy and quality, which include report specific compliance checklists to ensure a consistent assessment amongst the relevant MOD team across different suppliers. The MOD’s reviews, using these standardised processes, began on submissions due at the end of March 2023. We will provide further analysis on these reviews and the progress made by the MOD in the next compliance report.

4.12 When considering the subsequent resolution of issues raised, the quality of the submissions increases to 94 per cent for contract reports and for supplier reports to 61 per cent. Figure 7 shows the proportion of report submissions correct first time and following subsequent submissions through correction reports, for the different types of reports in 2022/23. Overhead reports are responsible for the overall supplier reports percentage being lower than contract reports, but the reason for this is explained below.

Figure 7: Proportion of report submissions correct first time and in subsequent submissions, by report type, for reports submitted in 2022/23

4.13 Initial, update, completion and strategic reports continued to achieve high correct percentages when reports following subsequent submissions were included. This demonstrates that issues raised by the MOD and the SSRO are being addressed by contractors which is improving the overall quality of the data held. In line with the SSRO’s review approach, for submissions in respect of which reporting issues raised by the SSRO have not been addressed by the contractor, these issues are closed after six months and have not been counted as being correct. There were 60 such issues in 2022/23. Forty-one of these issues were forwarded to the MOD and closed without response. The SSRO does not raise the same issue again on further submissions in relation to the same contract. While the closing of issues in this way relates to a relatively small proportion of submissions, it does mean that unaddressed data quality issues remain without satisfactory resolution.

4.14 Overhead reports achieved the lowest percentages for correct first time and in subsequent correction reports, at 28 per cent and 60 per cent respectively. We have explained earlier how the MOD has reviewed more overhead reports and raised more issues, so we expected this to affect the correct first time percentages. The number of overhead reports that were correct following subsequent submissions at 60 per cent is similar to the number we reported last year. This percentage is calculated as of the 1 July cut-off date each year and is impacted by timing issues. The majority of overhead reports are submitted in March and June each year. There were 196 overhead reports submitted for 2022/23, but 165 of these reports were submitted in March, April and May 2023. The 1 July cut-off date for this report does not provide enough time for these later report submissions to be reviewed, issues raised, and for correction reports to be submitted and closed. Figure 7 shows that the 2021/22 performance increased to 92 per cent for overhead reports with no issues raised on subsequent reports and given time, we expect a similar increase for 2022/23 reports.

4.15 Strategic reports achieved 100 per cent subsequent submission score, but without the MOD’s input into the quality of SICRs this is based solely on SMER reviews as no issues are raised on SICRs by the SSRO. The 2021/22 Annual Compliance Report included a recommendation for the MOD to provide feedback to contractors where a SICR submission does not meet the requirements and expectations for improvement to the quality of information available. We also asked the MOD to inform us when it has informed a contractor that a SICR is not required. The reviews of SICR submissions take place outside of DefCARS. We are therefore unaware of whether the MOD has provided feedback to contractors in 2022/23 and we have not been informed of those contractors who were exempted by the MOD from the requirement to provide a SICR.

Most issues are raised in initial contract report submissions

Significant messages

• We have developed our quality assessment of submissions to provide the number of issues raised against report submissions.
• Twenty per cent of contract reports contain all the contract report issues raised. Initial reports have the highest number of issues raised at 39 per cent.
• Seventy-one per cent of supplier reports contain all the supplier report issues raised. Overhead reports contain the highest number of issues raised at 72 per cent with the majority of issues being raised on the QBUCAR submissions.

4.16 In our compliance and review methodology: consultation response dated March 2023 we explained our intention to develop the quality assessment of submissions beyond the ‘pass or fail’ assessment. This year we have detailed the number of issues (MOD and/or SSRO) raised against report submissions reviewed by either or both parties. We will continue to develop this indicator for future compliance reporting.

4.17 Since the publication of the updated methodology we have analysed further the number of issues being raised by the MOD and/or the SSRO in individual reports. Nine hundred and five (72 per cent) of contract and supplier reports contain no issues, 224 reports (18 per cent) contain between one and two issues, 56 reports (4 per cent) between three and four issues and 70 reports (6 per cent) five or more issues. Twenty-eight per cent of reports contain 100 per cent of the issues.

4.18 Figure 8 shows the number of issues (MOD and/or SSRO) raised against those report submissions following a review by either or both parties.

Figure 8: Number of issues raised per report submission in 2022/23

4.19 This analysis shows that not many issues are raised on contract reports, although we explained earlier there are low percentages of MOD reviews. Twenty per cent of contract reports contain all of the contract report issues raised. Initial reports have the highest number of issues raised at 39 per cent, and of these, 14 per cent contain three or more issues. Initial reports are likely to contain a higher number of issues in comparison to the other contract reports as contractors or contractor teams are often new to the reporting requirements, unfamiliar with the contract and may be unaware that the timescale for making the submission is only one month from entering into the contract.

4.20 Seventy-one per cent of supplier reports contain 100 per cent of the supplier report issues raised. Overhead reports contain the highest number of issues raised at 72 per cent, and of these, 13 per cent contain three to four issues and 18 per cent have five or more issues raised on them. Our analysis shows that QBUCARs contained the highest number of issues at 87 per cent and rates reports were at 61 per cent.

4.21 DefCARS undertakes several automatic validation checks (warnings and errors) to support contractors with their report submissions by indicating areas that the user may wish to recheck or consider further. These validation checks are well established on contract reports but less so on supplier reports. We have also highlighted earlier in the report that the MOD has increased the amount of supplier report reviews at 36 per cent. The limited number of validation checks and the higher amount of MOD reviews explains the increased number of issues being raised on supplier reports. Ultimately, responsibility for quality of the data rests with the contractor and industry processes need to improve to get more reports right first time.

4.22 We will continue to develop the quality indicator to be able to provide more detailed information in the 2023/24 compliance report, including further clarity on the issues raised in reports beyond that provided in Figure 8 and the analysis of themes.

Targeted Review - Learning from reviews of DefCARS contract reports

The Targeted Reviews of selected report submissions assist with our understanding of how well MOD reviews of contract reports are working in practice. The exercise is undertaken in addition to our standard report reviews and, this year, it complemented our data utilisation engagement and DefCARS training with a specific MOD Delivery Team. The SSRO acknowledges the assistance and openness of the MOD Delivery Team in discussing and responding to its questions in discussions held during data utilisation meetings in March and April 2023 and thanks team members for their time in this regard.

We completed reviews of eight report submissions which covered initial, update and closure reports against six contracts. The small sample selected for review was not representative of the population of submissions. We provided detailed findings to the team, but the key issues and recommendations arising from the work this year are summarised below:

• three of the eight report submissions were submitted late by the contractors; six of the eight reports contained validation warnings in DefCARS; the MOD Delivery Team had reviewed three of the eight report submissions selected for this review;
• there were cases where the MOD Delivery Team reviewed the report submissions but had not changed the status of the reports to “no further issues”; MOD Delivery Team members informed us that they had not reviewed previous reports as * they were relatively new in post and the submissions were made two years prior; information that was known to MOD Delivery Team members, such as contractual assumptions, were not reported but were also not identified as missing by the MOD Delivery Team; and
• contract duration changes were reporting in On-Demand Reporting Plans, but consequential changes to reporting schedules for other reports were not accurately reflected in submissions.

We identified a number of conclusions and recommendations from this work for the MOD:

• when a report submission is late the MOD Delivery Team can raise this with contractors during their scheduled meetings, and with the SSAT for potential enforcement action;
• upon submission of each report the MOD Delivery Team should review all validation warnings in DefCARS, as well as considering the accuracy of the submission, and check for relevance and applicability of any outstanding validations prior to considering whether to raise issues;
• MOD Delivery Teams can support the effective operation of the legislative framework by reviewing and utilising DefCARS information in relation to individual contracts;
• MOD Delivery Teams need to pro-actively close reports that have been reviewed and where issues have been satisfactorily resolved for all submissions;
• issues that are subsequently corrected by contractors to improve the accuracy of the reports should be reviewed by MOD Delivery Teams and marked as reviewed on the system;
• it would be beneficial for new MOD Delivery Team members to receive a handover from their predecessor(s), including for report submissions in DefCARS and review previous report submissions in DefCARS to help understand their contract(s) and future report submissions; and
• when agreeing contract extensions, MOD Delivery Teams should ensure that the contractor takes care with any consequential changes to their report submissions which should be reflected in an up to date Contract Reporting Plan.

Basic report information is often incorrect

Significant messages

• The MOD raised 272 issues on contract reports and 246 issues on supplier reports, while the SSRO raised 311 issues on contract reports and 286 issues on supplier reports.
• A high proportion of issues raised in reports relate to contractors incorrectly reporting basic information and inconsistencies within the reports.
• Contractors must take greater care when completing report submissions on DefCARS, including the review of validation warnings prior to submission.
• Training is required in how to raise and log an issue in DefCARS, as the MOD has not always selected the correct issue category or has raised duplicate issues within reports.

4.23 Issues raised by the SSRO and the MOD on the statutory reports due in 2022/23 cover a wide range of issues, but some general themes can be identified. Figure 9 shows the top five themes identified from the issues raised by the MOD. In 2022/23, 114 of the 272 issues raised by the MOD on contract reports and 201 of the 246 issues raised on supplier reports centred around these top five themes. Some issues are applicable to several reports.

Figure 9: Top five themes from contract and supplier report issues raised by the MOD in 2022/23

4.24 We have further reviewed the top three themes for both contract and supplier reports raised by the MOD.

4.25 On the contract reports, contractors are required to report a quantified analysis of the causes of 90 per cent of variances between any estimated Allowable Costs used to determine the contract price and the total actual and forecast Allowable Costs. The MOD raised 37 issues, (21 issues relating to blank fields and 13 to incomplete explanations). This theme did not appear in last year’s top 5. Twenty-seven of these issues, however, were raised against one contract which could have been summarised more succinctly. The ‘Contract’ page requires basic information about the contract, which requires a cross check against the contract by the MOD. Six issues included missing contract information and ten related to incorrect dates. Risk contingency information is required on several DefCARS pages. Our analysis identified that only seven of 20 issues actually related to risk contingency. These issues could have been classified better on the system by MOD reviewers.

4.26 With respect to supplier reports, the Report Submission Admin page requires basic information. When raising issues including 40 relating to insufficient information provided by the contractor in the attachment, the MOD correctly used the Report Submission Admin category. The Profit before interest and tax category was selected for 52 issues that related to the people / payroll row in the QBUCAR pages. A number of these fields were populated with zeros and in 26 cases the MOD sought to clarify this. Thirty issues were raised by the MOD against the QBUCAR Head Count page, 28 due to missing data or to confirm that the data was correct.

4.27 Most of these could only be identified by the MOD as a party to the contract and shows that it is important that the MOD complete timely reviews.

Thematic Review – Assumptions

The SSRO’s compliance and review methodology states that we will seek to conduct Thematic compliance reviews. The aim of these is to enhance our understanding of issues by providing insight into the effectiveness of routine compliance reviews and how the provisions of the regulatory framework are being applied in practice. This year we selected the reporting of contract pricing assumptions as a theme to review.

Assumptions made about pricing QDCs/QSCs are required to be reported by contractors in the CPS. This information provides the basis upon which contracts are agreed. It also assists with contract management and helps to improve cost estimation for future contracts. Our reporting guidance (paragraphs 4.59 to 4.67) sets out what contractors should report in relation to assumptions, facts and calculations, ideally using the DefCARS fields across three pages - the assumption inputs, inflation assumptions and exchange rate assumptions pages. Alternatively, contractors can upload supporting documentation but this is sub-optimal for future analysis.

Our thematic review looked at the assumption information reported in the period May 2015 to April 2023 for QDC’s submitted up to and including 30 April 2023 (535 contracts representing a total of £83.6bn).

A total of 95 contracts (18 per cent) had no assumptions reported at all and four contracts contained no data in DefCARS but did attach an assumptions document. For the remaining 436 contracts a total of 2963 assumptions had been reported by contractors, an average of 5.5 assumptions per contract, with 117 (22 per cent) of the 535 contracts including attachments.

Data can be entered as ‘input assumptions’ or ‘calculation assumptions’. For ‘input assumptions’, the description was clear in 845 (44 per cent) of the assumptions, clear in 530 cases (27 per cent) once additional comments were reviewed and unclear 565 cases (29 per cent). For ‘calculation assumptions’ types, the description was clear in 106 (36 per cent) but there were 190 cases (64 per cent) where the assumption description and any supporting comments were unclear. The reporting of exchange rate and inflation assumptions was better, with 160 (87 per cent) of the exchange rate assumptions and 524 (97 per cent) of the inflation assumptions having a clear description. The requirements to annually profile the assumption value was less well completed. A total of 345 compliance issues were raised across 166 individual contracts. Two hundred and thirty-eight (69 per cent) of these were raised by SSRO and 107 (31 per cent) by MOD. The SSRO issues predominantly related to completeness whereas the MOD issues were much more closely aligned to the actual details of the contract for which the submission had been made.

Areas that we consider require improvement in relation to the reporting of pricing assumptions include:

• contractors taking greater care in completing assumption information, heeding validation warnings and familiarising themselves with reporting requirements and our guidance;
• the MOD improving its compliance checks to ensure that the reported assumption data is in line with contractual information. Where the data is incomplete, or inconsistent with the contract, the MOD should utilise the compliance functionality within DefCARS to raise issues with the contractor; and
• the SSRO providing further definitions and examples to assist contractors, simplifying the reporting capability within DefCARS, making use of data upload functionality, and better guidance on attachments.

4.28 We further reviewed the top three themes for both contract and supplier reports raised by the SSRO, identified in Figure 10, and identified areas for contractors to improve their reporting.

Figure 10: Top five themes from contract and supplier report issues raised by the SSRO in 2022/23

4.29 Contractors are required to provide the date that the report is due in every submission. This field is important for determining whether submissions are being made on time or not. Last year’s report found 120 issues against the Report Submission Admin category, which this year has reduced to 32. This reduction is attributable to 2022/23 being the first full year that our DefCARS development has provided contractors with a list of regular statutory reports and associated due dates as a drop down for selection based on the latest submitted reporting plan. Issues remain where there is no automation on the reporting of due dates in initial and on-demand reports. In these cases, contractors need to exercise greater care clearly linking report due dates to the legislative timescales or agreed dates.

4.30 A validation warning implemented in March 2017 warns contractors when the Total Contract Price on the contract page does not match that on the price page. A further validation implemented in May 2019 warns where the total contract price has changed, but the latest pricing amendment date field is blank. We raised 29 issues against contract price inconsistencies with 28 of these being cases the contractor would have received one of the validation warnings, but made the submission regardless.

4.31 Sixteen of the eighteen issues raised against inconsistencies across reports were raised against update reports, with the remaining two raised against CCRs. Seven related to contractors changing the contract price but not reporting a pricing amendment date. The regulatory framework requires the contractor to input the estimates as well as actuals for the relevant accounting period in the QBUCARs and explain any material difference. A total of 54 issues raised were attributable to failure to do so. Twenty-nine issues were raised because the date the report is due or relevant financial year was incorrect in overhead reports. We also identified 30 cases in the QBUCAR where the business unit total column fields for annual total new joiners and annual total leavers were not completed against the total number of employees rows.

4.32 All of these issues highlight the ongoing need for better adherence to reporting guidance and system requirements by contractors to ensure better quality data.

4.33 Recommendations to improve the quality of submissions and the use of data The SSRO has commenced work as part of its data utilisation project, which involves liaising with teams within the MOD, to identify data held in DefCARS that teams find useful, develop this into regular MI and to train teams on compliance in DefCARS. It is through increased use of the data that the MOD’s engagement with quality and timeliness matters will increase, which in turn will improve compliance levels.

4.34 The quality of submissions in terms of their accuracy can only be considered by the MOD as a party to the contract. We acknowledge within this report that the MOD’s SSPS team has increased its review of supplier reports and has commenced work to improve compliance with rates and supplier reporting in 2023/24. The MOD should continue to develop the supplier report processes to implement the 2021/22 Targeted Review recommendations and should carefully consider changes to the legislative definition of a QBU to ensure it supports the SSRO and the MOD in identifying when supplier reports are due. Alongside this, the SSRO should develop its own supplier report review processes to compliment the work of the MOD as part of our ongoing engagement with the MOD on the data utilisation project.

4.35 To increase the quality of the contract report data held in DefCARS, contractors should improve their data input to ensure reports are right first time and must take greater care when completing the ‘contract’ page which requires basic information relating to the contract, but there continues to be incorrect information reported. Validation warnings should be reviewed and addressed by revising the report or adding a comment prior to submission. Contractors should also take greater care when completing assumptions information in the system.

4.36 The MOD Delivery Teams should review more contract reports and the MOD can improve its compliance checks to ensure the assumptions data aligns with the contract and raise issues where this is not the case. The SSRO should develop its reporting guidance on assumptions to support contractors by including further definitions and examples.

4.37 Contract report reviews should be undertaken within 15 days from report submission as set out in the MOD’s Commercial toolkit, using the compliance functionality built into the system to evidence reviews. The increase in the review of supplier reports may be difficult to replicate in the same way with contract reports, as the SSPS is a dedicated team focusing on overhead submissions, whereas MOD Delivery Teams are larger and disperse across a number of contracts. This is, however, an area that requires attention from the MOD.

4.38 Handover arrangements and training to MOD teams new to the requirements of the regime should cover legislative requirements, commercial toolkit requirements and the DefCARS report submissions relating to the contractor(s) for which they are responsible. Reviewing and understanding previous report submissions will support teams to understand their contract(s) and future report submission requirements.

4.39 We have successfully reduced the number of issues being raised against the report due date field by developing a list of regular statutory reports and associated due dates for contractors to select once their initial report is submitted. The SSRO should investigate whether this facility can be extended to cover initial reports, thereby further reducing the number of issues raised against this field and report type.

4.40 Proposed changes to the regulatory framework may result in a reduction in the number of SICR submissions due in future years and may also improve the timeliness of remaining submissions, but this alone is unlikely to improve quality. Areas of poor quality remain, despite clear requirements being set out in Regulations and our reporting guidance, and this will be particularly important as other reporting requirements are updated following the MOD’s review of the legislation. The MOD should provide feedback to the contractor where a SICR submission does not meet those requirements and its expectations.

1. Initial reports (the Contract Pricing Statement (CPS), Contract Reporting Plan (CRP) and Contract Notification Report (CNR)), update reports (the Quarterly Contract Report (QCR) and Interim Contract Report (ICR)), and completion reports (the Contract Completion Report (CCR) and Contract Costs Statement (CCS)). ↩

2. Overhead reports (Qualifying Business Unit Cost Analysis Report (QBUCAR), Estimated Rates Claim Report (ERCR), Actual Rates Claim Report (ARCR), Estimated Rates Agreement Pricing Statement (ERAPS), Rates Comparison Report (RCR)) and strategic reports (Strategic Industry Capacity Report (SICR) and Small and Medium Enterprise report (SMER)). ↩

3. Throughout this report data attributable to previous years may be restated when compared to previous publications of the compliance report. Such changes occur where report submissions which were not received by a previous reporting cut-off date are later received, where suppliers have provided corrected or updated reports with new data, or where the SSRO has been made aware of different reporting requirements than originally thought. For further information please refer to Appendix 1: Data Sources and Methodology. ↩

4. Annual Qualifying Defence Contract Statistics: 2022/23 ↩

5. The Defence and Security Industrial Strategy: reform of the Single Source Contract Regulations ↩