Research and analysis

Technical Note: UKTI's monitoring and evaluation evidence

Published 5 June 2014

1. Part A: Measurement of Financial Benefits Generated by UKTI Trade Services

1.1 Introduction

Evidence about the financial benefits generated by UKTI trade services is gathered quarterly through the Performance and Impact Monitoring Survey (PIMS) carried out by OMB Research, an independent market research company specialising in business surveys. In addition to the evidence collected through PIMS, estimates of the impact of UKTI trade services are obtained through a rolling programme of independent evaluations of particular trade services. These evaluations use a range of alternative techniques, in order to derive estimates of the magnitude of the impact of trade services on business sales and other measures of performance. Further detail on the evaluation programme is in Part B of this note.

An estimate of total financial benefits generated by UKTI trade services is published annually by UKTI - PIMS reports by OMB Research (https://www.gov.uk/government/publications/ukti-performance-and-impact-monitoring-surveys-q4-201314-summary-results-pims-31-34-march-2014). Results published at the end of March 2014, based on surveys carried out during the financial year 2013-14, reported estimated total benefits of £51.8 billion measured in terms of additional sales attributed by UKTI trade clients to the support provided.

The PIMS estimates reflect businesses’ own judgments about the value of additional revenues and profits which they have achieved, or expect to achieve, as a direct result of the help provided. Figures for additional profits and revenues are obtained from UKTI trade clients interviewed for PIMS, using a sequence of questions which has been rigorously tested to ensure that it elicits well informed and carefully considered judgments about these values, taking account of the specific ways in which UKTI support had benefited a business. These judgments are then used by OMB Research to calculate a figure for estimated mean and total additional sales. Clients who said they would have achieved similar results without UKTI support are counted as having received zero additional sales.

PIMS interviews are conducted at two stages, the first being between 4-7 months after the provision of the support, and a follow-up, with a smaller sample, a year later. Interviews are designed to capture a rounded picture of the quality and impact of the support, taking into account the business context.

The values of financial benefit given by the firms themselves, in the context of PIMS interviews, are then adjusted downwards through application of discounting and a number of robustness checks. These adjustments cover the following:

• Discounting: Expected future profits are discounted at 8% and counted over a limited period, normally up to five years, exceptionally up to a maximum of 10 years. After this period, the discount rate is, in effect, increased to 100%.

• Additionality: Two separate additionality tests are applied. Benefits which are not explicitly attributed by the client directly to the support are excluded. Clients who state that they could have achieved similar results without support are classified as having achieved zero additional sales.

• Consistency: Additional profits attributed to UKTI by the client are not counted, unless the client has also reported significant impact on one or more qualitative indicators, showing how the service had enabled the additional profits to be made.

Details of these adjustments, and of their effects on the mean reported additional profit attributable to UKTI support, are published quarterly in the summary reports by OMB Research on PIMS results. A more detailed description is provided by OMB Research in its full annual report on PIMS. Analysis of PIMS data shows that the significant qualitative impacts most frequently reported by UKTI clients, who consider that the support has enabled them to generate additional profit and sales revenues are:

• gained access to customers/business partners not otherwise available,

• gained access to information not otherwise available,

• improved profile or credibility overseas,

• improved knowledge of the competitive environment,

• improved overseas marketing strategy, and

• gained confidence to explore or expand in an overseas market or markets.

These findings confirm that the reported financial benefit is generated as a result of UKTI trade services enabling clients to upgrade their approach to overseas business and overcome barriers to accessing overseas opportunities.

The very high financial benefits reported by UKTI clients have been a consistent finding over the 7 years covered by the PIMS research, and are also consistent with evidence from evaluations using other methodologies. Further qualitative insight into how the services achieve such high impact can be gleaned from case studies carried out as part of PIMS follow up qualitative research. Reports are available at:

http://webarchive.nationalarchives.gov.uk/20131124144306/http://www.ukti.gov.uk/uktihome/aboutukti/ourperformance/performanceimpactandmonitoringsurvey/qualitativesurveys.html

Alternative estimates of the additional sales resulting from UKTI trade support are available from quantitative evaluations of specific trade services, which are carried out by independent contractors as part of UKTI’s rolling programme of depth evaluations. These studies use econometric techniques to compare the performance of supported firms with that of a matched comparison group of non-supported firms. Two quantitative impact evaluations were completed during 2012:

• Evaluation of Trade Advisory Services delivered by the International Trade Teams in the English Regions (London Economics 2012): The analysis used data from a purpose designed survey of 300 users, and a matched sample of 300 non-user businesses. It found that UKTI support resulted in 11% faster growth over a 2 year period, and was also associated with a higher probability of entering new markets.

• Economic Impact Evaluation of UKTI’s Overseas Market Introduction Service (Breinlich, Mion, et al Essex, LSE, and Warwick University; 2012): The study matched client data to data on the wider population of UK businesses ^{[footnote 1]}. Econometric techniques were then used to evaluate the impact of service use on the performance of the client businesses. The study estimated that the mean effect of OMIS on employment and total turnover growth, over a two year period, was £1.5m additional turnover, and 7 additional jobs. At the median, the impact was £611,000 additional turnover, and 3 additional jobs.

These estimates are of a similar order of magnitude to those obtained from PIMS. Full reports on these and other evaluation research are available at:

http://webarchive.nationalarchives.gov.uk/20131124144306/http://www.ukti.gov.uk/uktihome/aboutukti/ourperformance/evaluation.html

A further two quantitative impact studies were commissioned in 2013 and will report later in 2014. The evaluations aimed to:

• Assess the impact of UKTI trade services on UK goods exports by linking UKTI client reported to HMRC data and data on the UK business population

• Assess the impact of UKTI trade and investment services by linking UKTI client records to data on the UK business population.

1.2 Other Possible Measures of Benefit

Two other measures of economic benefit, which are sometimes used in evaluation of trade services or other business support services, are:

• Additional exports: This measure was commonly used in evaluations of trade services carried out in the UK prior to 2002, and featured in the Public Service Agreement Targets for British Trade International for the period 1999-2002^{[footnote 2]}. For the Spending Review period 2002-04, it was replaced by a measure of improved business performance of users of the trade services, where performance was defined in terms of improved productivity and profitability over the medium term. This change reflects the fact that export activity is not an end in itself, but benefits businesses and the UK economy through enabling companies to improve their performance, and achieve stronger growth than would otherwise be possible. Focus on additional exports can potentially be misaligned with the business development needs of a company, whereas focus on a business performance improvement target ensures clear alignment with these needs. Moreover, focus on exports would substantially understate the impact of trade services, both in terms of benefit to the supported firm, and in terms of impact on net exports. This is because the improvements to products and services, and to productivity, which result from the services, have a favourable impact on the firm’s competitiveness and performance in all its markets, including in the UK domestic market.

• Gross Value Added (GVA): This measure is often used in the context of evaluating other forms of publicly funded business support. However, as the measure includes wages, there is a need to take into account the likelihood that staff in the supported business might be able to earn similar wages in another company or sector, possibly in another UK region. The measure should therefore be reported net of displaced alternative wage earnings, either at regional or national level, as appropriate in context. If this displacement is not taken into account, estimates of GVA will substantially overstate the impact of business support. However, displacement is very difficult to measure.

1.3 Calculation of Total Benefit

The method of calculation of the £51.8bn estimated total additional revenues generated by UKTI trade services is summarised below, in three parts:

• estimation of mean benefit per business supported,

• de-duplication to count the number of businesses that received support during the year: de-duplication is necessary because some businesses will have used more than one service in the course of a year; and

• grossing up from the mean, The total de-duplicated number of businesses is multiplied by the mean.

Calculation of the mean benefit, in terms of additional profit and sales attributed specifically to the help provided by UKTI, uses the following steps:

• Firms asked to estimate the expected benefit in terms of bottom-line profit £, or in terms of additional sales, if easier for the client to estimate.

• Future expectations allowed for (using annual discounting rate of 8%); number of years in the future is normally capped at five years, for example when the respondent says the revenues will continue “indefinitely”. Exceptionally, up to 10 years are counted, where the respondent is specific about the number of years

• Allowance made for actions not taken as a result of support. This adjustment takes account of the client’s assessment of the costs and benefit which might have been achieved through these foregone actions.

• Adjustment for non-additionality Total profit is weighted by proportion “would have realised anyway”, based on the client’s own judgment,

• Consistency check and further additionality check: Adjusted to zero if no impact has been recorded against at least one of two qualitative impact measures, namely, “change in behaviour (PIMS measure A83)” or “barriers to market access overcome” (PIMS measure A92). Impact is classified as zero in these qualitative measures if the firm has said it would have achieved similar results in any case.

All outliers are checked by a qualitative call back to the respondent by one of the OMB Research directors. (If the respondent is not able to provide a credible or consistent explanation, the observation is reduced to zero).

2. Part B: Key Sources of UKTI Data

The key source of data for measuring how UKTI is performing against our targets is UKTI’s CRM system, which provides the foundation information used within PIMS.

This data is transformed and reported upon using UKTI’s Business Reporting and Intelligence Tool (BRIT), introduced during 2009-10.

CRM – provides UKTI staff worldwide with a single view into customer-history transactions, allowing us to share relevant customer information across the organisation, and avoid duplication of effort. In doing this, it enhances the quality and professionalism of the service we provide to customers.

PIMS – our key performance measurement tool. It is an independent survey of our performance, carried out on our behalf by a leading market research organisation, OMB^{[footnote 3]}.

Economic and Evaluation Research – Evidence from PIMS is complemented by evidence from a rolling annual programme of in-depth evaluation, carried out by independent research teams, with technical advice and scrutiny also provided by senior academics that are not part of the lead research team. Reports on PIMS and on economic evaluation and research projects, commissioned by UKTI, are published in full on the UKTI website at:

http://webarchive.nationalarchives.gov.uk/20131124144306/http://www.ukti.gov.uk/uktihome/aboutukti/ourperformance.html

It is vital for measuring our progress, as well as for the delivery of high-quality, professional services to our customers, that all customer interactions, service deliveries, and active and successful inward investment projects are recorded fully and accurately on the CRM system throughout the year. To underline the importance of this to UKTI, all CRM users are required to meet a 100% accuracy target for entering customer data onto the CRM system within 48 hours.

2.1 Customer Data Management System (CDMS)

UKTI’s Customer Data Management System (CDMS) will be launched in three phases during 2013. It will replace and build upon the functionality currently delivered by a number of UKTI’s on-line systems which hold Customer Data. CDMS will integrate these systems and bring together in one place UKTI’s key customer data.

The systems to be replaced are:

• Customer Relationship Management (CRM)

• Overseas Market Introduction Service (OMIS)

• Corporate Reporting

2.2 The Performance and Impact Monitoring Survey (PIMS)

PIMS is an independent central monitoring survey of users of UKTI’s business services. It measures the performance and impact of UKTI support. PIMS covers all significant customer-facing trade services and provides evidence about service quality and what difference UKTI makes to businesses. It uses a range of measures, including information on the overall performance of UKTI against its performance measures.

The percentage figures in the measure used to report improved business performance against UKTI’s target reflect those firms reporting that they have achieved sustainable (i.e. longer-term) improvements in productivity and profitability, after they have secured additional export business as a result of support from UKTI.

Measures of a range of other business activities, that are covered by PIMS, have improved UKTI’s measurement of other aspects of its impact on business capabilities.

PIMS quarterly surveys are based on telephone interviews with a sample of users of UKTI’s principal services. The interviews are carried out in two waves. The first wave captures clients’ initial assessment of the difference that UKTI’s support has made, taking into account changes the business may have made to its products, practices, or marketing strategies, or impact on other business decisions. These interviews are carried out four to seven months after support has been provided.

The second wave of interviews is designed to assess the longer-term impact of UKTI services, and to capture any revisions to the clients’ initial assessment of this impact. These second-wave interviews are carried out 10 to 12 months later. The surveys concentrate on gathering information on business performance and processes, how these have changed over the period since service delivery, and the factors which lie behind the reported changes, such as improved knowledge and capabilities, or help with overcoming other barriers to overseas market entry. The emphasis is on factors affecting business competitiveness, and the measurement methodology aims to capture sustainable rather than unsustainable improvements in performance and competitiveness.

Issues covered by the survey questions include:

• barriers overcome, and new business, both in the target market as well as new sales in any other market, and any (positive or negative) effect on domestic sales,

• impact on skills and business behaviour, including improvements in products, processes or strategies, and impact on investment in research and new-product development,

• quality, relevance, and usefulness of information, advice, or contacts provided by UKTI,

• the extent to which similar benefits could have been achieved through other means,

• willingness to pay more for services,

• business-profile characteristics, and

• strategic motives for exporting.

As well as being used to measure performance against UKTI’s targets, results from the initial wave of interviews also provide UKTI managers with early indications of how well different services are performing, so that any necessary adjustments can be made to drive up service quality and effectiveness.

Analysis of correlations between reported qualitative benefits, reported impact on productivity, and profitability, shows that these hard business performance improvements most frequently occur when clients report substantial benefit from:

• gaining access to contacts not otherwise accessible,

• gaining access to information not otherwise accessible, including understanding how to navigate unfamiliar business environments,

• raising the firm’s profile or credibility in overseas markets,

• improving knowledge of the competitive environment overseas,

• improving the firm’s overseas marketing strategy and

• gaining the confidence to explore a new market or expand in an existing one.

For some clients, the key factor has been help to overcome a particular problem or difficulty with a legal or regulatory issue, including intellectual property protection.

Data gathered on business-profile characteristics and strategic motives for exporting, are used in analysis of the survey results, to help identify characteristics of companies most likely to benefit from UKTI support. The questions also contain cross checks, ensuring that any inconsistent responses or outliers can be identified, and checked through a follow-up call where necessary.

The follow-up interviews provide a further opportunity for cross checks with firms’ initial responses and also provide evidence about the time profile of benefits resulting from the support. Clients who reported in the initial interview that they had gained access to information or contacts not otherwise accessible, or had benefited substantially from raising their profile in the market, are also asked whether these had turned out to be more or less useful than they had initially thought, or about the same.

Analysis shows that across most impact measures there is little change between the results derived from the initial interview wave and the follow-up. Exceptions are:

• initial assessments of the business benefit resulting from raising a firm’s profile, and from gaining access to information not otherwise accessible, are significantly more often revised upwards in the follow-up interview, and

• innovation impacts tend to increase, with firms reporting improvements to products or processes, or impact on R&D, which had not been expected at the first interview.

Full details of the PIMS survey results, and in-depth analysis of these and other issues, can be found in the PIMS reports, published on the UKTI website at www.ukti.gov.uk.

The PIMS survey of UKTI trade service users is complemented by a smaller annual survey of UK exporters who have not used UKTI services. The aims of this non-user survey are to provide evidence on usage of non-UKTI export support, measure the extent to which firms encounter barriers which give rise to the need for such services, investigate attitudes towards support among those firms not yet accessing it, and provide data on the profile of non-user firms to allow comparison with the profile of UKTI users. Those firms who have used some form of non-UKTI export support are asked about its quality and effectiveness, using questions designed to be consistent with the measures used for UKTI trade services.

Comparison of data on the profile of users and non-users shows that UKTI clients differ from many non-users in a number of important ways. UKTI clients are:

• more likely to be innovative and to undertake R&D,

• more growth-oriented, and more likely to report strong growth and growth expectations,

• more likely to report substantial business benefits from exporting, especially in terms of benefits from exposure to new ideas. ^{[footnote 4]}

Nevertheless, the non-user surveys also show that there is a significant minority of non-users which match the profile of users, but had not been aware of the availability of UKTI trade services.

In order to inform ongoing development of the survey methodology and to provide deeper insights into the results, UKTI also commissions two small qualitative studies annually, to follow up respectively on the user and non-user surveys. These qualitative studies involve a small number of in-depth interviews with companies that had taken part in one of the initial survey interviews, and had expressed willingness to take part in further research. The specific focus of these follow-up interviews is guided by topical priorities. The 2013 studies focused on ‘UKTI Regional Network Advisory Services - Understanding Service Quality and Impact, and What Makes a Successful Intervention” and “Performance of Export Services Delivered by UKTI’s US Network: Understanding Client Perceptions of Service Quality and Impact, and Exploring Ideas for Improvement.

Development of the monitoring surveys is also informed by evidence and analysis derived from UKTI’s research and evaluation programme.

2.3 Economic and Evaluation Research

In addition to its programme of monitoring surveys, UKTI also commissions an annual programme of independent evaluations of the economic impact and rationale for specific services. These studies use a range of quantitative and qualitative research techniques, typically including econometric analysis of the performance of UKTI-supported firms, and a comparison group of non-supported companies, to test for service impact on key business performance variables. The methodology for evaluation of the economic rationale for services typically involves a review of relevant literature, together with collection of evidence on various aspects of additionality, and the most likely counterfactual, to provide a basis for assessing what would have happened without the service. Each evaluation project is guided by an advisory group, involving a number of academic researchers with relevant expertise.

A key principle which guides UKTI’s approach to evaluation is that analysis of the policy logic of an intervention should precede and inform detailed decisions about evaluation methodology. This principle helps to ensure that the economic rationale for the policy is properly understood, and that the evaluation methodology will be both robust and credible. Evidence from literature reviews accordingly plays an important part in the evaluation process in UKTI. This helps to ensure that individual evaluation research projects are grounded within existing theory and evidence, and that design of evaluation methodology is in each case informed by a sound understanding of relevant issues.

Economic impact evaluations carried out for UKTI, also seek to make optimal use of the large datasets on the performance of UK businesses which are available, based on business returns to Companies House, and on registrations of intellectual property ownership. By using statistical techniques to control for other factors which might influence performance, researchers are able to use these datasets to compare the performance of UKTI-supported businesses with that of other similar UK firms who have not used these services.

For example, using these techniques, research on businesses which used UKTI’s Overseas Market Introduction Service (OMIS) ^{[footnote 5]} indicates that users achieved over £1m in mean additional revenue as a result. The researchers point out that PIMS can only capture direct effects attributed by the client to the service, whereas the data on performance capture other effects as well. These other effects are likely to include sales resulting from contacts made as a longer- term consequences of engaging in the new business, which the client had attributed to UKTI support at the time of the PIMS interview.

Using similar statistical techniques, an evaluation of the impact of UKTI support provided by the International Trade Advisers in the English regions estimated similarly large positive impacts on the performance of supported businesses. ^{[footnote 6]}

New academic research is also commissioned on occasion, where there is a clear need to fill a gap in the wider evidence relating to the economic rationale for UKTI services. For example, UKTI commissioned new quantitative research to investigate the causal links between exporting and productivity, at firm level and at aggregate economy level (Harris and Li 2007^{[footnote 7]}). This study found that, on average, firms gained a 34% productivity up-lift, taking account of selection effects, as a result of beginning to export. The study also found that some 60% of UK productivity growth was attributable to exporters, with non-exporters mainly contributing to aggregate productivity growth through net exit.

UKTI regards its monitoring, evaluation, and research programme as a dynamic interactive process, through which each of the three sources of evidence and analysis is enriched. To this end, events are hosted regularly at which evidence from monitoring, evaluation and academic research are all discussed, with participation by officials with policy responsibility as well as by Government economists, evaluation specialists, and academic researchers. This process fosters cross-fertilisation across the monitoring, evaluation, and academic research programme, and ensures development of a richer and more robust evidence base. It also ensures that policy-relevant insights and lessons from all three sources of evidence can be identified and debated in the round, thus providing a more rounded and reliable basis for policy.

Evidence from research and evaluation carried out for UKTI, and from wider economic literature, is reviewed in depth in two BIS Economics Papers, both available on the BIS website:

• BIS Economics Paper No 5 – Internationalisation of Innovative and High Growth SMEs (2010)

• BIS Economics Paper No 13 – International Trade and Investment – the Economic Rationale for Government Support

1. The study used the OFLIP database which was compiled by Christian Helmers and the late Mark Rogers (Oxford), and incorporates data from FAME based on Companies House records, together with publicly available data on intellectual property ownership from IPO and other sources. ↩

2. The target was set on “the value of additional exports generated for each £1 DTI/FCO expenditure”, with the baseline set at £20:£1. ↩

3. PIMS Research Reports, providing quarterly assessments of our performance can be found at https://www.gov.uk/government/collections/uk-trade-investment-performance-and-impact-monitoring-survey ↩

4. PIMS Non-User Survey Reports available at: http://webarchive.nationalarchives.gov.uk/20131124144306/http://www.ukti.gov.uk/uktihome/aboutukti/ourperformance/performanceimpactandmonitoringsurvey/nonusersurveys.html ↩

5. Impact study of OMIS (2012) Breinlich et al ↩

6. Evaluation of the International Trade Teams (2012) London Economics ↩

7. Firm level empirical study of the contribution of exporting to UK productivity growth. Harris and Li (2007). Report to UKTI ↩