Guidance

Public Design Evidence Review: Literature Review Paper 3 - Public Design and Public Value (HTML)

Published 17 July 2025

Commissioned by the Policy Design Community

Authors:

Liz Richardson, Professor of Public Administration, Department of Politics, University of Manchester

Catherine Durose, Professor of Public Policy, and Co-Director of the Heseltine Institute for Public Policy, Practice and Place, University of Liverpool

Iacopo Gronchi, Doctoral Candidate, Institute for Innovation and Public Purpose, University College London

Charlie Mealings, Doctoral Candidate and Adjunct Lecturer, Department of Political Economy, King’s College London

How to cite:

Cabinet Office (2025) Public Design Evidence Review: Literature Review Paper 3 – Public Design and Public Value.

Background

This document is the third of three literature reviews commissioned by the cross-government Policy Design Community, written by an interdisciplinary team of academics. It explores the relationship between public design and public value, highlighting the potential of public design to enhance policymaking and governance by fostering innovation, co-design, and effective policy framing. 

The wider project was commissioned as a non-exhaustive exploration of the relationship between public design and public value. It was conducted within rapid timeframes and prioritised cross-disciplinary working. The authors began drafting in September 2023. They finalised their drafts in March 2024 and published in May 2025.

‘Literature Review Paper 1 - Public Design’^{[footnote 1]} Literature Review Paper 1 (‘Public Design’) and ‘Literature Review Paper 2 - Public Value’^{[footnote 2]} are published alongside Paper 3 (‘Public Design and Public Value’) as part of the Public Design Evidence Review.

1. Executive Summary

Public value has become an important mobilising narrative for reform, including within UK government, challenging public management to establish the ‘so what’ of public policy (Bason, 2017) by developing public value propositions. These propositions should reflect what the public wants and is concerned about, as well as ensuring good governance.

So, how can public design support the delivery of public value? And what does the academic evidence base tell us about the relationships between public design and public value? In this report, we draw upon an integrative review of published academic empirical research, then summarise existing literature that seeks to test claims for the contribution of public design to public value.

This is the third report in a series of three companion reports, commissioned as part of a wider review of public design in policymaking: the Public Design Evidence Review. It was led by the UK Government’s Policy Design Community, which is a group of civil servants across departments in central government, alongside colleagues in local government, who form a community of practice for those interested in developing the application of public design in public policy. The report has been produced by a team of academics across design research and political science from the University of Manchester, University of Liverpool, University College London, and King’s College London.

Despite its influence, public value remains an idea in development conceptually (O’Flynn, 2021) and in practice can be a challenging ambition to achieve. Seeking to do so however, opens the opportunity to do things differently. It is in this context that public design has emerged as a promising approach to supporting public value.

In summary, the findings from the academic literature are:

• The evidence base is at an early stage of development but suggests a potential future research agenda.
• Promising approaches in the academic literature provided some evidence of the role of elements of public design contributing to elements of public value (see more below). This evidence was concentrated in policy-specific and discipline-specific domains.
• The available evidence suggests a persuasive direction of travel, indicating the positive potential of design for public value, which could be further evidenced.
• The evidence base is not yet sufficiently comprehensive to demonstrate findings at the level of a ‘meta-effect’ of public design as an enabler of public value creation. Future research could seek to meet the ‘high bar’ of this criterion.

Reflecting the evidence base, we focus on two key potential contributions of public design to public value: co-design and framing.

• Co-design - understood as a form of ‘collective creativity’ or ‘collective ownership of outcomes’ (Bebbington et al. 2022) - is recognised to ‘transpose readily’ to generating public value in their shared emphasis on relational, ambitious, open and outcome-focused ways of working.
  • Sanders et al. (2010) for example, differentiate between three types of co-design tools and techniques: talking, making, and playing. Each of these is evidenced in the literature to have an application relevant to public value generation; for example, driving creativity and innovation.
  • A range of further design contributions are recognised to be useful in ‘scaffolding’ (Bebbington et al. 2022) the co-design process. These include user experience, process design, digital tools, and team-working.
• Complex policy challenges, such as climate change, require ‘framing’ to sustain public perceptions of legitimacy and feasibility. Design is recognised as offering a valuable contribution to generating persuasive public value propositions in this context.
  • The contribution of design to policy persuasiveness is supported by evidence across a wide range of rigorous studies on policy challenges, such as making sustainable choices, civic participation, and public health measures.
  • In many of these studies, design makes a central contribution to the construction of behavioural insights, for example, in the construction of ‘nudges’ (Mukherjee and Mukherjee, 2018).
• Design contributions also help policymakers to communicate to each other and establish more useful policy outcomes.

The direction taken now will shape the ongoing credibility and momentum of design research and, by implication, the application of design approaches in policy settings. Building on the extant evidence base, we identify four areas for future research:

• Evidencing the impacts of design in policymaking and understanding how to communicate this in ways that are credible and persuasive to diverse stakeholders.
• Understanding how to develop organisational capacity and skills to support design in policy. This includes learning from other approaches being introduced into government, such as co-production, data science, behavioural insights and preventative early intervention, and how they have addressed resource implications and barriers to innovation.
• Looking across the policy environment to differentiate the role of public design in different spaces, including different levels of government, in policy implementation, and across sectors.
• Given the context of uncertainty, complexity and contestation, anticipatory policy design is of increasing importance, as is specifying the distinctive contribution of design thinking to futures thinking, in policy and operational delivery in government.

The field of work on the contribution of public design to public value involves continued excitement from proponents, healthy debate, and critical thinking within the field, alongside some scepticism from those who are less convinced, and calls for the next phase of development (O’Flynn, 2021).

We can discern the beginnings of a transformation, but what that transformation looks like will depend on the next phase of the research agenda. We hope the recommendations from this review will help shepherd this part of the discipline through to maturity. A stronger evidence base will also help to socialise public design approaches in public policy and governance.

2. Method

We used an integrative review (Toracco 2005; Snyder 2019) of published academic research and evidence. Below, we describe the process, including search terms used, journals searched, guiding questions, and descriptive results of the search.

2.1 Process

We conducted a targeted search of academic literature, in several rounds, using a set of guiding questions (set out in Appendix 1). We used a series of search terms that covered the key concepts of public design, public value, and robust and rigorous methodologies (Appendix 2). We iterated our strategies based on the results within and between rounds (for example see Appendices 3 to 6). While this was in no way a systematic review, we have been mindful of guiding principles from Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) (Moher et al., 2009) and the Cochrane principles for qualitative systematic reviewing.

After collating the papers, we developed an annotated bibliography (Appendices 7-10). In this we annotated notes on each paper summarising the content, noting its relevance for our review and any key limitations. We then synthesised the results of the search and annotated bibliography into the findings in section 3 of this report.

2.2 Search terms

We started with an initial indicative set of search terms that covered the four core elements of the search. That is:

• Includes design (or similar – e.g., co-production) as independent variable
• AND/OR includes public value (or similar – e.g., trust) as dependent variable
• AND showcases hard evidence (qualitative, quantitative, or systematic review)
• AND addresses (directly or indirectly) our core research questions

We used different Boolean search combinations and adapted our search terms between each round, for example, to widen an unfruitful search, or to narrow down a search that yielded too many non-relevant results. We started with the most rigorous criteria (for example, design AND value AND RCT) then moved to less exact criteria to expand the search - particularly replacing the AND requirement with OR for the research methodologies’ criteria.

2.3 Journals

In the first round, we identified a list of journals in political science and design research, in which we might reasonably expect to find relevant material (see list in Appendix 2). The list comprised highly ranked journals and those selected for their remit and coverage of either/both content and/or methodologies. We amended the list in subsequent rounds, based on ‘snowballing’ from references in relevant papers.

2.4 Guiding questions

The following guiding questions were used by the team in the searches. We anticipated that we would need to take the questions in the round, rather than individually, depending on the level of sophistication of the evidence base. Where the evidence base has proven to be in the preliminary stages, we have reflected these questions in the identified areas for future research:

• How do the features of public design practice contribute to the delivery of public value outcomes?
• What is the evidence for design’s direct contribution to aspects that could be broadly considered as aspects of public value (even if not named as such)?
• What is the upstream effect of public design on policymaking and the downstream effect on delivery (including operational and digital channels)?
• How do institutions measure and track the impact of public design on public value (including what it prevents)?
• Where has the financial return on investment of design been quantified?

In the next section, we summarise some of the themes from the substantive content generated by the search.

3. Findings from the literature review

As set out above in section 2, we focused on finding existing empirical evidence, of as high a methodological quality as possible, using a structured search of academic journals.

In this section we situate our analysis in contemporary debates on public value, and highlight the potential scope of public design to respond to current limitations. Whilst acknowledging that literature exploring the holistic intersections between public design and public value is at an early-stage of development, we can identify a wide-range of specific studies linking aspects of public design to aspects of public value. To cohere these studies, we have used heuristics to pull together the findings in ways that reflect the evidence base. Doing so, leads us to focus on two key potential contributions of public design and public value: co-design; and framing.

3.1 Background

Public value has become a significant, if contested, idea in public administration and management (O’Flynn, 2021), providing a mobilising narrative for reform, including within UK government. It initially developed as part of a broader strategic approach to help guide public managers in achieving valued results, and as a means of integrating “substantive purpose, political authorisation and organisational capacity” (Moore, 1984, p. 21). Mark Moore’s (1995) highly influential ‘strategic triangle’ for public value, establishes three challenges for public managers.

First, their strategy must be aimed at creating something that is substantively valuable; second, it must be legitimate and politically sustainable; and third, it must be operationally and administratively feasible (O’Flynn, 2021, p. 869).

Public managers therefore must work on developing public value propositions, which can articulate “public aspirations and concerns as well as the procedural norms and values associated with good public sector governance” (Alford et al., 2017, p. 590); or as Bason (2017) terms it, establishing the ‘so what’ of public policy. The implication being that, by doing so, public value has the potential in tangible terms to enhance democratic values within policymaking such as legitimacy, justice and effectiveness (Stoker, 2006; Fung, 2015; Crosby et al., 2016).

Despite its influence, public value remains an idea in development conceptually (O’Flynn, 2021) and in practice. In terms of the latter, for example, practical efforts to establish public value can be in tension with “cost, project approval, policy, governance and culture” (Baxter et al., 2023) in public policymaking and government.  These tensions are exacerbated by the broader context of socio-ecological-technical challenges that increasingly “strain the capacity of 20th century policy design approaches” (Nogueira and Schmidt, 2022, p. 32).

For every challenge, however, there are opportunities that open. Searching for feasible ways to develop public value, in inhospitable conditions, has strengthened resolve to evolve new ways of doing things. Strains on attempts to develop public value have raised productive questions about the approaches that might boost efforts and resolve possible tensions. As some scholars have pointed out, we are starting to re-think “the intent of policy […] how policy is constructed – and for whom” (Nogueira and Schmidt, 2022, p. 32). Torfing et al. (2020) identified the need for a mediation, or a mediator, to help foster constructive insights from conflict, and a catalyst to prompt policy actors to ‘think out of the box’. To do so in a way that nurtures public value requires, as phrased by Torfing et al. (2020, 592): “taming the snake in paradise”. In other words, to nurture a diversity of “views, ideas and forms of knowledge whilst still establishing a common ground for learning” (see also Taoka et al., 2018; Nguyen and Mougenot, 2022).

Public design is emerging as one potentially promising approach (Bason and Austin, 2022; Bebbington et al., 2022) that can help to ‘tame the snakes’. In this review, we focus on the question of how public design may support the generation of compelling public value propositions, in the face of policy challenges that are uncertain, complex and contested (van der Steen et al., 2018). Advocates have made some compelling claims about the roles of public design in contributing to public value. In this review, we asked how far the existing evidence base can support those claims. The substantive findings follow in the sections below.

3.2 The evidence base is at an early stage of development

Our search was for robust analyses of the causal or contributory role of public design on public value. In this context, ‘robust’ was defined by initially looking for gold-standard methodologies, including Randomised Controlled Trials (RCTs) - particularly through field experiments - and reviews of robust evidence bases, through systematic reviews. 

We ideally wanted to see robust assessment of the causal role of the explanatory/independent variable (public design) on the outcome/dependent variable (public value) considering and controlling for confounding variables like organisational reform). Additionally, the research should be transferable or replicable across settings, i.e. have good external validity, as well as internal validity.

Unsurprisingly, given the high bar, we did not find work at this order of magnitude. We did not find ‘meta’ level studies. That is, we failed to find much work that took a holistic definition of public design. By this we mean including the six components/elements set out in the companion literature review in this series (Paper 1) and using experimental or other similar methods to causally connect it to a holistic definition of public value, with the elements as set out in the other companion review in this series (Paper 2). 

While it might seem a big ask, these standards have been partially or even more fully met in other fields, such as work in health/medical treatment, public health (to a lesser extent), in political science areas such as studies of aspects of democratic regimes such as voting, trust and so on. We did not however anticipate that we would find such work in the field of design, hence the dearth of work was as we had anticipated. 

Variations were found within the area broadly defined as public design. We expanded our definition to include ‘thinner’ definitions of design such as design’s sister field of behavioural insights. Including behavioural insights expanded the evidence base, as this field has excelled in establishing a robust evidence base from its inception, including exemplary transparent publishing of null results. We also note here that some work we uncovered from the field of commercial design is relevant and could be potentially replicated in relation to public design, to test transferability between contexts.

The companion papers in this series of literature reviews also detail some of the origins of the success of interaction design on public value, through cases of the use of design in the digital sphere, for example through case studies of the UK Government Digital Service (GDS), and the BBC. Other work packages in the Public Design Evidence Review are collating case study examples. We narrowed down our searches to particular forms of evidence. The evidence base did support claims that interaction design contributed to generating persuasive public value propositions. We also saw studies that demonstrated the role for digital tools in providing scaffolding for design. However, digital applications and interaction design were not perhaps as prominent in the academic studies as we might have expected, given their role in establishing the field in practice and policy.

In addition, we started with a clear expectation about the direction of travel in causal pathways. We set out an expectation that design (or equivalent/adjacent variables) was the explanatory or independent variable, and public value (or equivalent/adjacent variables) was the outcome or dependent variable. Instead, we found that the inter-changeable use of design practices was an explanatory variable or factor in some studies and an outcome in others. Concepts like creativity, citizen engagement, lived experience and innovation frequently appeared across different studies, as both input and output factors. Given that design processes have multiple iterations and therefore potentially multiple outputs, at different phases or stages (as well as various intermediate outcomes) this is entirely in keeping with design’s underlying theories of change, and the modelling of pathways being set out in related work packages.

3.3 Wide range of specific studies

What we found were more specific studies linking aspects of public design to aspects of public value. While specific, the studies covered a wide range of different specific aspects of both concepts. Our search generated a wide range of types of design interventions or practices. They encompassed ‘design for polity’ as well as ‘design for policy’, i.e. system-wide processes of policy formulation, as well as specific service level examples.

Examples of design tools or practices used in the studies included:

• changes in policy tool mixes, pooled policy tools (policy packaging)
• communications campaigns
• behavioural insights-informed interventions (reminders, prompts and other salience enhancing devices, and framing devices)
• citizen engagement and mobilisation methods
• heterogeneous inputs to policymaking/introduction of more diverse expertise
• policy labs, innovation labs e.g. Public Sector Innovators (PSIs)
• agile methods and adaptive models
• transformational and other reforms of leadership styles and approaches

Imputed outcomes being tested, under the banner of public value, were similarly wide-ranging and covered the following.

Organisational/administrative capacity in government and other governance institutions, such as:

• understandings of evidence by policymakers
• strengthening the resilience and future preparedness of public action/foresight and anticipatory planning
• closing the gap between policy ideation and implementation
• strengthening creativity and ideas generation
• insulating government from high levels of risk and uncertainty
• changing culture, mindsets, shared values/mission
• organisational collaboration, joined up working and trans-disciplinary working
• problem-solving capacity of public sector organisations
• connections to lived experience and proximity to users
• strengthened attitudes related to place

Citizen behaviour change including civic participation, such as:

• charitable donations
• sharing personal data for public good
• lifestyle changes towards health and well-being, eating choices, vaccine take-up, and welfare service take-up

Citizen perceptions of policymaking and government, such as:

• trust
• accountability
• legitimacy
• policy effectiveness or its impact
• service quality

System or organisational reform, such as:

• more effective and efficient procurement practices
• service quality and processes, for example: hand hygiene in hospitals and prompt tax returns

Greater eco-resilience at individual, organisational and system levels, such as:

• sustainable food choices
• flood preparedness

These materials offered a promising direction of travel towards establishing contributory and causal links between some aspects of design practices and certain types of positive outcomes for public bodies. Many of the studies specified plausible pathways from public design to public value. In section 3.4 we summarise a digest of what the current evidence base tells us about the links between public design and public value.

3.4   Digest of academic literature on links between public design and public value

Section 3.3 illustrated the wide range of specific policy topics and design tools covered by the literature we reviewed. While positive, in terms of potential comprehensiveness, it means that it can be harder to understand what the literature tells us in a more coherent way. 

Therefore, in our digest of the literature, we use heuristics to pull together the findings in ways that are supported by the thrust of the evidence. We centre the findings around two sets of key potential contributions of public design to public value: ‘co-design’ and ‘framing’. Many of these insights and claims about the links between public design and public value are aligned with the broader search we carried out in the companion Paper 1 in the series of literature reviews.

3.4.1. Co-design contributing to enhanced policy processes

Design researchers have begun to recognise the resonance between the aspirations of public value and the ‘inclusive and participatory foundations of co-design’, and the potential to encourage creativity in the policy process (Bason, 2017; Blomkamp, 2018; Thøgersen et al., 2021; Bebbington et al., 2022). For example, Bason and Austin (2022) identified design’s exploration, generation, and enacting of innovation as key pathways to public value creation. Similarly, Deserti et al. (2020) identifies three pathways from co-design to public value creation: i) bridging ideation and implementation; ii) adapting to context; iii) allowing continuous reflection on the very notion of public value. In this way, public design practices may indicate the emergence of human-centred models of public governance that offer new openings for creative influences and a counterpoint to more bureaucratic and analytical traditions of policymaking (Bason and Austin, 2022). Building on this, we consider the potential for design to fulfil this function.

As Bebbington et al. (2022) note interest goes both ways, with public value theorists (Benington and Moore, 2011; Brown, 2021) highlighting the potential role for participatory methods to support the generation of public value. However, this link appears to have not yet been thoroughly explored.

Design practices have long been premised on working in a way that foregrounds multi-disciplinarity and diversity, and supports co-design (Mitchell et al., 2016; Wang et al., 2022; van Dam et al., 2021) which has been evidence as providing pathways to public value creation (Criado et al., 2021; French and Raman, 2021). Co-design is most simply understood as a form of ‘collective creativity’ or ‘collective ownership of outcomes’ (Bebbington et al. 2022). Design researchers have increasingly sought to identify underpinning principles for co-design. For example, Cruickshank et al. (2013, cited in Bebbington et al., 2022, p. 10) propose eight key guidelines:

1. Agree how the success of the project will be recognised
2. Move in and beyond your normal design practice
3. Involve and respect lots of people in the idea-generating parts of the process
4. Use the expertise of all participants in the process
5. Let everyone be creative in their own way
6. Explore and challenge assumptions
7. Expect to go beyond the average
8. Bring the process to the best possible conclusion with the best possible design outcome

Similarly, Blomkamp (2018) refers to principles of being adaptive, inclusive, outcomes-focused, participative and respectful. Bebbington et al. (2022, p. 10) have argued that these guidelines ‘transpose readily’ to generating public value in their emphasis on relational, ambitious, open and outcome-focused ways of working.

Resonating with the broader connection between ‘learning by doing’ as a means of public service innovation (Criado et al., 2023), Sanders et al. (2010) differentiate between three types of co-design tools and techniques: talking, making, and playing - each of which is also established in the literature:

1. Talking, telling and explaining - including stories, self-observation, experience maps and prioritising. For example, Villalba et al. (2019), evidence how translating research and practice findings, presented in an accessible card format, was used as part of a storytelling activity. This supported both service professionals and service users to:

• apply the insight from the card to the character/story;
• discuss real life; and
• discuss experiences; triggering empathy and informing service re-design.

2. Making tangible things, including sketches, collages, mock-ups and prototypes. For example, sketching is a design practice which organises ‘strategically built knowledge space[s]’ (Brun et al., 2016) that facilitate shared understanding, generates new insights, and supports the emergence of ideas (Brun et al., 2016), as well as the evaluation of ideas (Sun et al., 2013). This can promote good or better ideas coming forward, supporting the development of the “most promising” ideas (Sun et al., 2013). Prototyping activities have been evidenced to challenge embedded norms and privileges in existing policy approaches, and enhance ownership of problem definition and solution finding in policy design processes (Nogueira and Schmidt, 2022).

3. Acting, enacting and playing, including personas, games, scenario-making and role-playing. For example, Bornet and Brangier’s (2016) study evidenced that the translation of research and evaluation into personas, rather than presenting them as a written list of findings, prompted fuller debate and generated more appropriate ideas and responses.

Evidence from robust methods suggests that using design techniques like talking, making and playing in co-design processes, drives outcomes of creativity and innovation. Evidence includes measurable improvements in terms of both the quality and quantity of ideas (Mitchell et al., 2016; Lee and Ostwald, 2022; Ou et al., 2023; Guaman-Quintanilla et al., 2023).

For example, a study by Mitchell et al. (2016) evidenced how co-design can generate a more holistic perspective on problem-solving, by generating more ideas (see also, Perttula et al. 2006) and more systematic thinking about potential solutions.

Another study showed the long-term transformational impact of engagement in a design process on both creative skill sets of participants and their ability to navigate team dynamics (Sadowska and Laffy, 2019). This was supported by Balakrishnan (2016), whose study evidenced how design thinking can help to boost motivation and creative skill sets. A further study also illustrated the wider wellbeing benefits of co-design by enhancing a sense of personal growth and a collective sense of place-related optimism (Corcoran et al., 2018).

A range of further design contributions are recognised to be useful in ‘scaffolding’ (Bebbington et al. 2022) the co-design process. These include:

• User experience. Design has long been considered a means to foreground user experience in policymaking. Design processes can offer an explicit means to allow policymakers greater proximity with service users, which is evidenced to boost public service motivation and performance (Bellé, 2014), support resilience of public services (Leite and Hodgkinson, 2023), and generate empathetic insight from policymakers and public service professionals, and dialogue between different stakeholders (van Rijn, 2011).
• Process design. For example, linking taskwork and teamwork in an interactive pattern and through a ‘heartbeat’ rhythm of interaction (regular peaks in action separated by pauses) (Cash et al. 2020).
• Digital tools. Design has been at the forefront of digitalisation within policymaking, particularly in supporting new citizen-government interfaces. This can support improved service performance and address complex policy challenges involving multiple agencies (Allen et al., 2020). Sustained input from designers can help to avoid interactive tools and platforms that can unintentionally undermine shared understanding (Steen et al. 2013).
• Team working. Public value generation can rest on the development of a ‘one team culture’ (Baxter et al. 2023) across traditional policymaking silos. Design can enhance attention on the impact of team dynamics (for example, skills diversity, and flexibility) which can positively enhance conflict resolution, problem solving, goals and performance, tasks and planning (Fraser, 2009).

3.4.2. Framing contributing to perceptions of policy legitimacy

Classically complex policy challenges, such as climate change, require transformative policy responses to enhance effective outcomes. This often involves ‘policy packages’ bringing together different policy instruments, which can be difficult to communicate to the public.

Policy framing in this context is important to sustain public perceptions of policy legitimacy and feasibility (Montpetit, 2008; Fesenfeld, 2022; Reynolds et al., 2022) and of positive impact (Christensen, 2021). Design practices are useful in generating persuasive public value propositions to address these concerns (Lee et al. 2017; Katsonis, 2019).

The contribution of information and interaction design, in generating persuasive public value propositions, is evidenced by a wealth of rigorous studies on a wide range of critical policy challenges. These include:

• Citizens’ COVID vaccination uptake (Keppeler et al., 2022; Chen et al., 2023, Kantorowicz-Reznichen et al., 2022).
• Taking of prescribed medicines (Jachimowicz et al., 2021).
• Tand hygiene compliance (Van Roekel et al., 2022).
• Making sustainable lifestyle choices (Gravert and Kurz, 2021; De-loyde et al., 2022; Chandra, 2023).
• Supporting climate change mitigation (Mol et al. 2021; Dolšak et al., 2020).
• Promoting voter registration (Kolle et al., 2020).

In many of these studies, design makes a central contribution to the construction of behavioural insights, for example, in the construction of ‘nudges’ (Mukherjee and Mukherjee, 2018).

Policy framing is also important in how policymakers communicate to each other. Again, design is useful here in communicating underlying evidence to policymakers (Brick and Freeman, 2021).

‘Innovative abduction’ (Dong et al., 2015) refers to a process of working creatively between detailed practices, specific concrete examples, and larger-scale or more generalised understandings. Abductive processes have been shown to support new ways to frame ideas and propositions, and to explore new working principles underpinning them. Visualisation by a designer, particularly where it involves both imagination and images of the end user or outcomes, helps to create abduction, resulting in more useful outcomes (Dahl et al., 2001).

4. Future research agendas

The digest of the academic material in section 3.4 illustrates some of the potential for this ‘teenage’ field to blossom into a mature area of study. The direction that is taken now will shape the ongoing credibility and momentum of design research and, by implication, the application of design approaches in policy settings. 

In section 4, we build on both the strengths and the gaps in the extant literature, to identify four areas for future research.

4.1 Building an evidence base for the impacts of design in policymaking

A recurrent question in the academic literature is how to evaluate and build an evidence base for how design impacts the policymaking process and, more significantly, its outcomes (Whicher and Crick, 2019).

The use of design has been claimed to generate a wide range of impacts, at the level of practices: changing mindsets, practices, skills (Ferrarezi, 2021; Perez et al., 2022), and tools; and at a strategic level, shifts in framing and language, relationship-building, critical reflection, situated understanding and connection with lived experience (Cole and Hagen, 2023). However, there is more work needed to develop the evidence base and do that in ways that are credible and persuasive across different groups of stakeholders. 

More debate is needed about which standards and types of evidence are valued. Some tensions are evident between emphasis by certain groups (including in this report) on gold-standard methodologies, such as systematic reviews, field experiments and so on, and other groups, who have concerns about the limitations of the dominant approaches to evaluation (Mergel et al., 2018). The latter group would support a case for a more pluralistic approach to evidence, for example, through calls for more creative forms of evaluation, to evidence the value or “capture the ‘how’ in terms of (co-)design” (Manohar et al. 2016). Such approaches would need to be adaptive, responsive to context and demonstrate value beyond financial return (Owen et al. 2022). However, it is unclear whether alternative forms of impact evaluation could present convincing data to people in the former group.

Our review of the literature used the idea of public value to cohere across very different areas of outcomes resulting from, or linked to, public design. In some cases, we have imputed the association between the factors being tested and their connection to public design and/or public value. Public value provides a potentially useful framing for such inquiries (Soe and Drechsler, 2018). Future work could strengthen rigour in the field by more explicitly operationalising and testing these concepts.

4.2 Research on organisational capacity and skills to support design in policy

Design is not alone in facing the challenge of establishing itself and expanding its application. These are common challenges for innovative approaches being introduced into government. While their trajectories are each quite distinct, there are insights that can be drawn from ideas such as co-production (Loeffler and Bovaird, 2016), data science, behavioural insights (Whicher and Crick, 2019) and preventative early intervention (Cairney, 2020). Lessons from such approaches indicate the need to address resource implications, including skills development (Loeffler and Bovaird, 2016).

Barriers may constrain the establishment of new approaches within policymaking, even where there is a level of support for it (Kleinhans, 2017). Barriers include: a lack of knowledge or understanding of the approach within the policymaking environment; friction with legal tasks and responsibilities; tensions around institutional ownership; limited durability of relationships between different stakeholders; and ‘risk averse’ administrative cultures.

Therefore, a second key area for future research is the organisational capacity and skills that could support uptake of design approaches.

4.3 Research on the use of design across policy environments

As the evidence base on design matures, one area for exploration is how to differentiate its role in separate spaces in the policymaking environment. This includes all levels of government: central, local and city-regional. Local governments in the UK, and internationally, are developing their own design capabilities and capacity, and the use of decentralised design approaches is currently under-explored (Hermus, van Buuren, and Bekkers, 2019). Focusing on local, and decentralised forms of design, also suggests a greater spatial element to future research that looks at the role of design in place-making, as suggested by emerging work (Collier and Gruendel, 2022).

Future research could usefully look at how design is mobilised by different groups of policy actors, within each of the levels, recognising that policy is ‘made’ on the front-line, through implementation, as much as it is in policy formulation stages (Durose, 2011, Durose and Lowndes, 2023). Some work has already begun to explore the role of front-line innovation in generating public value (Gofen et al., 2023), and this avenue could be pursued further. Looking at different policy actors would then start to include work on everyday design and the use of design by civil society actors. This recommendation is supported by existing reviews of the field, one example of which identifies an over-representation in the literature of work focusing on civil servants and less on other actors (Hermus, van Buuren, and Bekkers, 2019).

A third dimension could include distinctions between policy areas. Each of the three dimensions also intersect in multiple ways, generating possibilities for a three-dimensional ‘cube’ of research questions.

4.4 Developing knowledge on design approaches for futuring

Given the context of uncertainty, complexity and contestation, anticipatory policy design is of increasing importance (Capano and Pavan, 2019), as is specifying the distinctive contribution of design thinking to futures thinking in policy. There is a growing sense that meeting these challenges requires engaging a range of actors in policymaking, including citizens (Gouache, 2022). However, such practices, including participatory agenda-setting, are “relatively experimental, atypical and sporadic” (Gouache, 2022). This suggests an important agenda for further research on how public design approaches can be more grounded in foresight and deliberation.

4.5 Conclusion

Ideas have their own lifecycle, typically moving in stages from conceptual and empirical development (Mullins, 1973), through emerging excitement, followed by critique and either transformation or decline (Hirsch and Levin, 1999). The field of work, on the contribution of public design to public value, has some elements of all three stages simultaneously: continued excitement from proponents, healthy debate and critical thinking within the field, some scepticism from those who are less convinced, and calls for the next phase of development (O’Flynn, 2021). We can discern the beginnings of a transformation, albeit in a shape that is taking form across a disparate set of outlets and that, so far, lacks coherence.

What that transformation looks like will depend on the next phase of the research agenda. The evidence base in the field might be characterised as going through its ‘teenage years’. We hope that the recommendations from this review will help shepherd this part of the discipline through to maturity. A stronger evidence base will also help to socialise public design approaches in public policy and governance.

References

Alford, J., Douglas, S., Geuijen, J., and ‘t Hart, P. (2017). Ventures in public value management: Introduction to the symposium. Public Management Review, 19(5), 589–604. Available at: Ventures in public value management: Introduction to the symposium

Allen, B., Tamindael, L. E., Bickerton, S. H., and Cho, W. (2020). Does citizen coproduction lead to better urban services in smart cities projects? An empirical study on e-participation in a mobile big data platform. Government Information Quarterly, 37(1), 101412. Available at: Does citizen coproduction lead to better urban services in smart cities projects? An empirical study on e-participation in a mobile big data platform

Balakrishnan, B. (2022). Exploring the impact of design thinking tool among design undergraduates: a study on creative skills and motivation to think creatively. International Journal of Technology and Design Education, 32(3), 1799-1812. Available at: Exploring the impact of design thinking tool among design undergraduates: a study on creative skills and motivation to think creatively

Bason, C. (2017a). Leading Public Design. Discovering Human-centred Governance. Bristol: Policy Press. Available at: Leading Public Design. Discovering Human-centred Governance (PDF file. Size 4.38MB)

Bason, C. and Austin, R. D. (2022). Design in the public sector: Toward a human centred model of public governance. Public Management Review, 24(11): 1727-1757. Available at: Design in the public sector: Toward a human centred model of public governance

Baxter, D. et al. (2023). Institutional challenges in agile adoption: Evidence from a public sector IT project. Government Information Quarterly, p. 101858. Available at: Institutional challenges in agile adoption: Evidence from a public sector IT project

Bebbington, J, Cruickshank, L. and Haig, N. (2022). Co-designing public value: collective ownership of outcomes in the public sphere. Design Research Society. Available at: Co-designing public value: collective ownership of outcomes in the public sphere

Bellé, N. (2014). Leading to Make a Difference: A Field Experiment on the Performance Effects of Transformational Leadership, Perceived Social Impact, and Public Service Motivation. Journal of Public Administration Research and Theory, 24(1), pp. 109–136. Available at: Leading to Make a Difference: A Field Experiment on the Performance Effects of Transformational Leadership, Perceived Social Impact, and Public Service Motivation

Benington, J. and Moore, M. (2011) Public value: theory and practice. Macmillan.

Blomkamp, E. (2018). The promise of co‐design for public policy. Australian Journal of Public Administration, 77(4), 729-743. Available at: The Promise of Co-Design for Public Policy

Bornet, C., and Brangier, E. (2016). The effects of personas on creative codesign of work equipment: an exploratory study in a real setting. CoDesign, 12(4): 243-256. Available at: The effects of personas on creative codesign of work equipment: an exploratory study in a real setting

Brick, C., and Freeman, A. (2021). Communicating evidence in icons and summary formats for policymakers: What works? Behavioural Public Policy, 1-29. Available at: Communicating evidence in icons and summary formats for policymakers: What works?

Brown, P. R. (2021). Public Value Measurement vs. Public Value Creating Imagination – the Constraining Influence of Old and New Public Management Paradigms. International Journal of Public Administration. 44(10), 808–817. Available at: Public Value Measurement vs. Public Value Creating Imagination – the Constraining Influence of Old and New Public Management Paradigms

Brun, J., Le Masson, P., and Weil, B. (2016). Designing with sketches: The generative effects of knowledge preordering. Design Science, 2, E13. Available at: Designing with sketches: The generative effects of knowledge preordering

Cairney, P. (2023) Understanding public policy: theories and issues. Third edition. Bloomsbury.

Capano, G., Pavan, E. (2019). Designing anticipatory policies through the use of ICTs. Policy and Society, 38(1): 96–117. Available at: Designing anticipatory policies through the use of ICTs

Cash, P., Dekoninck, E., and Ahmed-Kristensen, S. (2020). Work with the beat: How dynamic patterns in team processes affect shared understanding. Design Studies, 69: 100943. Available at: Work with the beat: How dynamic patterns in team processes affect shared understanding

Chandra, G. (2023). Non-monetary intervention to discourage consumption of single-use plastic bags. Behavioural Public Policy, 7(1): 143-156. Available at: Non-monetary intervention to discourage consumption of single-use plastic bags

Chen, N., Trump, K., Hall, S. and Le, Q. (2023). The effect of postcard reminders on vaccinations among the elderly: A block-randomized experiment. Behavioural Public Policy, 7(2), 240-265. Available at: The effect of postcard reminders on vaccinations among the elderly: A block-randomized experiment

Christensen, H.S. (2021). A conjoint experiment of how design features affect evaluations of participatory platforms. Government Information Quarterly, 38(1) 101538. Available at: A conjoint experiment of how design features affect evaluations of participatory platforms

Cole, L. and Hagen, P. (2023). Scaling deep through transformative learning in public sector innovation labs – experiences from Vancouver and Auckland. Public Management Review, 0(0), pp. 1–28. Available at: Scaling deep through transformative learning in public sector innovation labs – experiences from Vancouver and Auckland

Corcoran, R., Marshall, G., and Walsh, E. (2018). The psychological benefits of cooperative place-making: a mixed methods analyses of co-design workshops. CoDesign, 14(4): 314-328. Available at: The psychological benefits of cooperative place-making: a mixed methods analyses of co-design workshops

Criado, I., Ferreira Dias, T., Sano, H., Rojas-Martín, F., Silvan, A. and Isidro Filho, A. (2021). Public Innovation and Living Labs in Action: A Comparative Analysis in post-New Public Management Contexts. International Journal of Public Administration, 44(6): 451-464. Available at: Public Innovation and Living Labs in Action: A Comparative Analysis in post-New Public Management Contexts

Criado, J.I., Alcaide-Muñoz, L. and Liarte, I. (2023). Two decades of public sector innovation: building an analytical framework from a systematic literature review of types, strategies, conditions, and results. Public Management Review, 0(0), pp. 1–30. Available at: Two decades of public sector innovation: building an analytical framework from a systematic literature review of types, strategies, conditions, and results

Crosby, B. C., Hart, P. and Torfing, J. (2016). Public value creation through collaborative innovation. Public Management Review. 19(5), 655-669. Available at: Public value creation through collaborative innovation

Cruickshank, L., Coupe, G. and Hennessy, D. (2013). Co-design: fundamental issues and guidelines for designers: Beyond the Castle case study. Swedish Design Research Journal, 2: 48-57. Available at: Co-design: fundamental issues and guidelines for designers: Beyond the Castle case study

Dahl, D. W., Chattopadhyay, A., and Gorn, G. J. (2001). The importance of visualisation in concept design. Design Studies, 22(1), 5-26. Available at: The importance of visualisation in concept design

De-loyde, K., Pilling, M., Thornton, A., Spencer, G., and Maynard, O. (2022). Promoting sustainable diets using eco-labelling and social nudges: A randomised online experiment’. Behavioural Public Policy, 1-17. Available at: Promoting sustainable diets using eco-labelling and social nudges: A randomised online experiment

Deserti, A., Rizzo, F. and Smallman, M. (2020). Experimenting with co-design in STI policy making. Policy Design and Practice, 3(2), 135-149. Available at: Experimenting with co-design in STI policy making

Dolšak, N, Adolph, C, Prakash A. (2020). Policy design and public support for carbon tax: Evidence from a 2018 United States national online survey experiment. Public Administration. 98: 905–921. Available at: Policy design and public support for carbon tax: Evidence from a 2018 United States national online survey experiment

Dong, A., Lovallo, D., and Mounarath, R. (2015). The effect of abductive reasoning on concept selection decisions. Design studies, 37: 37-58. Available at: The effect of abductive reasoning on concept selection decisions

Durose, C. and Lowndes, V. (2024). Gendering discretion: why street-level bureaucracy needs a gendered lens. Political Studies, 72(3): 1026-1049. Available at: Gendering discretion: why street-level bureaucracy needs a gendered lens

Durose, C. (2011). Re-visiting Lipsky: front line work in UK local governance. Political Studies, 59(4): 978-995. Available at: https://doi.org/10.1111/j.1467-9248.2011.00886.x

Ferrarezi, E., Brandalise, I. and Lemos, J. (2021). Evaluating experimentation in the public sector: learning from a Brazilian innovation lab. Policy Design and Practice, 4(2): 292-308. Available at: Evaluating experimentation in the public sector: learning from a Brazilian innovation lab

Fesenfeld, L. (2022). The effects of policy design complexity on public support for climate policy. Behavioural Public Policy, 1-26. Available at: The effects of policy design complexity on public support for climate policy

Fraser, K. (2009). Labour flexibility: impact of functional and localised strategies on team-based product manufacturing. CoDesign, 5(3): 143-158. Available at: Labour flexibility: impact of functional and localised strategies on team-based product manufacturing

French, T. and Raman, S. (2021). Engaging People with Lived Experience in Co-design of Future Palliative Care Services. International Journal of Public Administration, 44(9): 778-789. Available at: Engaging People with Lived Experience in Co-design of Future Palliative Care Services

Fung, A. (2015). Putting the Public Back into Governance: The Challenges of Citizen Participation and Its Future. Public Administration Review, 75(4): 513-522. Available at: Putting the Public Back into Governance: The Challenges of Citizen Participation and Its Future

Gouache, C. (2022). Imagining the future with citizens: participatory foresight and democratic policy design in Marcoussis, France. Policy Design and Practice, 5(1): 66-85. Available at: Imagining the future with citizens: participatory foresight and democratic policy design in Marcoussis

Gravert, C., and Kurz, V. (2021). Nudging à la carte: A field experiment on climate-friendly food choice. Behavioural Public Policy, 5(3): 378-395. Available at: Nudging à la carte: A field experiment on climate-friendly food choice

Guaman-Quintanilla, S., Everaert, P., Chiluiza, K., and Valcke, M. (2023). Impact of design thinking in higher education: a multi-actor perspective on problem solving and creativity. International Journal of Technology and Design Education, 33(1): 217-240. Available at: Impact of design thinking in higher education: a multi-actor perspective on problem solving and creativity

Hermus, M., van Buuren, A. and Bekkers, V. (2020). Applying design in public administration: a literature review to explore the state of the art. Policy and Politics, 8(1): 1–20. Available at: Applying design in public administration: a literature review to explore the state of the art

Hirsch, P. M. and Levin, D.Z. (1999). Umbrella advocates versus validity police: A life-cycle model’. Organization Science, 10(2): 199-212. Available at: Umbrella advocates versus validity police: A life-cycle model

Jachimowicz, J., Gladstone, J., Berry, D., Kirkdale, C., Thornley, T., and Galinksy, A. (2021). Making medications stick: Improving medication adherence by highlighting the personal health costs of non-compliance. Behavioural Public Policy, 5(3), 396-416. Available at: Making medications stick: Improving medication adherence by highlighting the personal health costs of non-compliance

Kantorowicz-Reznichenko, E., Kantorowicz, J., and Wells, L. (2022). Can vaccination intentions against COVID-19 be nudged? Behavioural Public Policy, 1-25. Available at: Can vaccination intentions against COVID-19 be nudged?

Katsonis, M. (2019). Designing effective public engagement: the case study of Future Melbourne 2026. Policy Design and Practice, 2(2): 215-228. Available at: Designing effective public engagement: the case study of Future Melbourne 2026

Keppeler, F., Sievert, M., and Jilke, S. (2022). Increasing COVID-19 vaccination intentions: A field experiment on psychological ownership. Behavioural Public Policy, 1-20. Available at: Increasing COVID-19 vaccination intentions: A field experiment on psychological ownership

Kleinhans, R. (2017). False promises of co-production in neighbourhood regeneration: the case of Dutch community enterprises. Public Management Review, 19(10), 1500–1518. Available at: False promises of co-production in neighbourhood regeneration: the case of Dutch community enterprises

Kolle, F., Lane, T., Nosenzo, D. and Starmer, C. (2020). Promoting voter registration: The effects of low-cost interventions on behaviour and norms. Behavioural Public Policy, 4(1), 26-49. Available at: Promoting voter registration: The effects of low-cost interventions on behaviour and norms

Lee, H., Tsohou, A., Choi, Y. (2017). Embedding persuasive features into policy issues: Implications to designing public participation processes. Government Information Quarterly, 34(4), 591-600. Available at: Embedding persuasive features into policy issues: Implications to designing public participation processes

Lee, J. H., and Ostwald, M. J. (2022). The relationship between divergent thinking and ideation in the conceptual design process. Design Studies, 79, 101089. Available at: The relationship between divergent thinking and ideation in the conceptual design process

Leite, H. and Hodgkinson, I. R. (2023). Examining resilience across a service ecosystem under crisis. Public Management Review, 25:4, 690-709. Available at: Examining resilience across a service ecosystem under crisis

Loeffler, E. and Bovaird, T. (2016). User and community co-production of public services: what does the evidence tell us? International Journal of Public Administration, 39(13), 1006–1019. Available at: User and community co-production of public services: what does the evidence tell us?

Manohar, A., Smith, M., and Calvo, M. (2016). ‘Capturing the “how”: showing the value of codesign through creative evaluation’. In: Lloyd, P. and Bohemia, E. (eds.), Future focused thinking - DRS International Conference 2016, 27 - 30 June, Brighton, United Kingdom. Available at: Capturing the “how”: showing the value of codesign through creative evaluation

Mergel, I., Gong, Y. and Bertot, J. (2018). ‘Agile government: systematic literature review and future research’, Government Information Quarterly. 35(2), 291–298. Available at: Agile government: systematic literature review and future research

Mitchell, V., Ross, T., May, A., Sims, R., and Parker, C. (2016). Empirical investigation of the impact of using co-design methods when generating proposals for sustainable travel solutions. Co-Design, 12(4), 205-220. Available at: Empirical investigation of the impact of using co-design methods when generating proposals for sustainable travel solutions

Moher, D., A. Liberati, J. Tetzlaff, and D. G. Altman (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement’. Annals of Internal Medicine 151(4), 264. Available at: Empirical investigation of the impact of using co-design methods when generating proposals for sustainable travel solutions

Mol, J., Botzen, W., Blasch, J., Kranzler, E., and Kunreuther, H. (2021). All by myself? Testing descriptive social norm-nudges to increase flood preparedness among homeowners. Behavioural Public Policy, 1-33. Available at: All by myself? Testing descriptive social norm-nudges to increase flood preparedness among homeowners

Montpetit, É. (2008). Policy design for legitimacy: expert knowledge, citizens, time and inclusion in the United Kingdom’s biotechnology sector. Public Administration, 86: 259-277. Available at: Policy design for legitimacy: expert knowledge, citizens, time and inclusion in the United Kingdom’s biotechnology sector

Moore, M. (1984, May 1–12). A conception of public management. In Teaching public management. Proceedings of a workshop to assess materials and strategies for Teaching Public Management (pp. 9–11). Seattle: Public Policy and Management Program for Case and Course Development, Boston University. Available at: A conception of public management

Moore, M. (1995). Creating public value: Strategic management in government. Harvard University Press.

Mukherjee, I. and Mukherjee, N. (2018). Designing for sustainable outcomes: espousing behavioural change into co-production programmes. Policy and Society, 37(3), 326-346. Available at: Designing for sustainable outcomes: espousing behavioural change into co-production programmes

Mullins, N. (1973). Theories and theory groups in contemporary American sociology. Harper and Row.

Nogueira, A. and Schmidt, R. (2022). Participatory policy design: igniting systems change through prototyping. Policy Design and Practice, 5(1): 32-50. Available at: Participatory policy design: igniting systems change through prototyping

O’Flynn, J. (2021). Where to for public value? Taking stock and moving on. International Journal of Public Administration. 44:10, 867-877. Available at: Where to for public value? Taking stock and moving on

Ou, X., Goldschmidt, G., and Erez, M. (2023). The effect of disciplinary diversity on design idea generation in dyadic teams. Design Studies, 86, 101184. Available at: The effect of disciplinary diversity on design idea generation in dyadic teams

Owen, V., Ceyhan, P., Cruickshank, L., and Christou, E. (2022). Evaluating social innovations: How creative evaluation can help articulate their values and impacts. In: Lockton, D., Lenzi, S., Hekkert, P., Oak, A., Sádaba, J., Lloyd, P. (eds.), DRS2022: Bilbao, 25 June - 3 July, Bilbao, Spain. Available at: Evaluating social innovations: How creative evaluation can help articulate their values and impacts

Perez, R.E., Ng A.C.L. and Tiglao, N.C.C. (2022). Enhancing policy capacity through Co-design: the case of local public transportation in the Philippines. Policy Design and Practice, 5:1, 103-121. Available at: Enhancing policy capacity through Co-design: the case of local public transportation in the Philippines

Perttula, M. K., Krause, C. M., and Sipilä, P. (2006). Does idea exchange promote productivity in design idea generation? Co-Design, 2(3), 125-138. Available at: Does idea exchange promote productivity in design idea generation?

Reynolds, J., Hobson, A., Ventsel, M., Pilling, M., Marteau, T., and Hollands, G. (2022). Effect of visualising and re-expressing evidence of policy effectiveness on perceived effectiveness: A population-based survey experiment. Behavioural Public Policy, 1-19. Available at: Effect of visualising and re-expressing evidence of policy effectiveness on perceived effectiveness: A population-based survey experiment

Sadowska, N., and Laffy, D. (2019). Measuring the impact of strategic design learning experience long after the classroom delivery. The Design Journal, 22(sup1), 1305-1315. Available at: Measuring the impact of strategic design learning experience long after the classroom delivery

Sanders, E., Brandt, E., and Binder, T. (2010). A framework for organizing the tools and techniques of participatory design. Proceedings of the 11th Biennial Participatory Design Conference, 195-198. Available at:  A framework for organizing the tools and techniques of participatory design

Snyder, H. (2019). Literature review as a research methodology: an overview and guidelines. Journal of Business Research, 104, 333-339. Available at: Literature review as a research methodology: an overview and guidelines

Soe, R.-M. and Drechsler, W. (2018). Agile local governments: Experimentation before implementation’. Government Information Quarterly, 35(2): 323–335. Available at: Agile local governments: Experimentation before implementation

Steen, M., Arendsen, J., Cremers, A., De Jong, A., De Jong, J., and De Koning, N. (2013). Using interactive model simulations in co-design: An experiment in urban design. CoDesign, 9(1): 2-16. Available at: Using interactive model simulations in co-design: An experiment in urban design

Stoker, G. (2006). Public value management: a new narrative for networked governance? American Review of Public Administration, 36(1): 41-57. Available at: Public value management: a new narrative for networked governance?

Sun, L., Xiang, W., Chai, C., Wang, C., and Liu, Z. (2013). Impact of text on idea generation: an electroencephalography study. International Journal of Technology and Design Education, 23, 1047-1062. Available at: Impact of text on idea generation: an electroencephalography study

Thøgersen, D., Waldorff, S.B. and Steffensen, T. (2021). Public value through innovation: Danish public managers’ views on barriers and boosters. International Journal of Public Administration, 44(14), pp. 1264–1273. Available at: Public value through innovation: Danish public managers’ views on barriers and boosters

Torfing, J., Cristofoli, D., Gloor, P.A., Meijer, A. J. and Trivellato, B. (2020). Taming the snake in paradise: combining institutional design and leadership to enhance collaborative innovation. Policy and Society, 39(4), 592–616. Available at: Taming the snake in paradise: combining institutional design and leadership to enhance collaborative innovation

Torraco, R.J. (2005). Writing integrative literature reviews: Guidelines and examples. Human Resource Development Review, 4(3), 356-367. Available at: Writing integrative literature reviews: Guidelines and examples

Van Dam, S., Sleeswijk Visser, F., and Bakker, C. (2021) The impact of co-creation on the design of circular product-service systems: Learnings from a case study with washing machines. The Design Journal, 24(1), 25-45. Available at: The impact of co-creation on the design of circular product-service systems: Learnings from a case study with washing machines

Van der Steen, M., van Twist, M. (2018). Strategies for robustness: five perspectives on how policy design is done. Policy and Society, 37(4), 491–513. Available at: Strategies for robustness: five perspectives on how policy design is done

Van Rijn, H., Sleeswijk Visser, F., Stappers, P. J., and Özakar, A. D. (2011) Achieving empathy with users: the effects of different sources of information. CoDesign, 7(2), 65-77. Available at: Achieving empathy with users: the effects of different sources of information

Van Roekel, H., Reinhard, J., and Grimmelikhuijsen, S. (2022). Improving hand hygiene in hospitals: comparing the effect of a nudge and a boost on protocol compliance. Behavioural Public Policy, 6(1), 52-74. Available at: Improving hand hygiene in hospitals: comparing the effect of a nudge and a boost on protocol compliance

Villalba, C., Jaiprakash, A., Donovan, J., Roberts, J., and Crawford, R. (2019). Testing literature-based health experience insight cards in a healthcare service co-design workshop. CoDesign. 17(1), 70-82. Available at: Testing literature-based health experience insight cards in a healthcare service co-design workshop

Wang, Z., Jiang, T., Huang, J., Tai, Y., and Trapani, P.M. (2022). How might we evaluate co-design? A literature review on existing practices. In: Lockton, D., Lenzi, S., Hekkert, P., Oak, A., Sádaba, J., Lloyd, P. (eds.), (2022). Bilbao, 25 June - 3 July, Bilbao, Spain. Available at: How might we evaluate co-design? A literature review on existing practices

Whicher, A. and Crick, T. (2019) ‘Co-design, evaluation and the Northern Ireland Innovation Lab’, Public Money and Management, 39(4),290–299. Available at: Co-design, evaluation and the Northern Ireland Innovation Lab

List of appendices

• Appendix 1. Internal team brief for Topic 3 literature search
• Appendix 2. Journal lists, query terms and example queries
• Appendix 3. Search results (by journal) – design journals round 1
• Appendix 4. Search results (by journal) – design journals round 2
• Appendix 5. Search results (by journal) – political science journals round 1
• Appendix 6. Search results (by journal) – political science journals round 2
• Appendix 7. Bibliography – design journals, round 1
• Appendix 8. Bibliography – design journals, round 2
• Appendix 9. Bibliography – political science journals, round 1
• Appendix 10. Bibliography – political science journals, round 2

Appendix 1. Internal team brief for Topic 3 literature search

Overall idea

An integrative review (Toracco 2005; Snyder 2019) of published research and evidence, academic outputs.

Guiding questions

• How do the features of public design practice contribute to the delivery of public value outcomes?
• What is the evidence for design’s direct contribution to aspects that could broadly be considered to be aspects of public value (even if not named as such).
• What is the upstream effect of public design on policymaking and the downstream effect on delivery (including operational and digital channels)?
• How do institutions measure and track the impact of public design on public value (including what it prevents)?
• Where has the financial return on investment of design been quantified?

What we want

We are looking here ideally for ‘hard’ evidence of causal mechanisms, using ‘gold standard’ impact evaluation methods, following a hierarchy of methods, with field experiments at the top, along with all other forms of experiment like survey experiment, lab experiment – any RCT.  Then quasi-experimental, i.e. anything with a control group even if not randomly assigned.  Then any ‘observational’ study e.g. case study, cross-sectional analysis, etc. We also follow a hierarchy of evidence/outlets so academic publications i.e. books and peer-reviewed journals are top. 

We can use the evolving operative concepts to lightly guide the search, partly to further specify the hypothesised mechanisms and links, but also looking for ‘proof’/evidence of those links.  But we don’t need to be restricted to that.

Method

Brainstorm and keep a record of search terms. Note the number of hits and how you narrow down or expand the search, depending on number and quality of hits. Possible search terms in various combinations: ‘design’; ‘public design’; ‘public value’; ‘impact’/’impacts’; ‘benefits’; ‘field experience’; ‘randomised controlled trial’; ‘RCT’; ‘field trial’; ‘quasi-experiment’; ‘natural experiment’; ‘control’; ‘counter-factual’/’counter factual’; ‘evaluation’; ‘systematic reviews of evidence’; ‘return-on investment’; ‘cost benefit’/’cost-benefit analysis’.

For each article, do a light-touch annotated bibliography (ref, weblink, copy of abstract, a short paragraph on what’s most useful for us, and any reflections.

Appendix 2. Journal lists, query terms, and example queries

Design journals

• Design Methods and Theories
• Creativity and Innovation Management
• International Journal of Operations and Production Management
• Journal of Management Information Systems
• Journal of Organisational Behaviour
• Government Information Quarterly
• Co-design
• Journal of Service Design
• Design for Health
• Journal of Health Design
• Designs for Learning
• International Journal of Technology and Design Education
• Journal of Design Research
• DISCERN, International Journal of Design for Social Change, Sustainable Innovation and Entrepreneurship
• Social Design Review
• International Journal of Design in Society
• Strategic Design Research Journal

Public admin, public policy and political science journals

• International Journal of Public Administration
• Journal of Public Administration Research and Theory
• Public Management Review
• Public Administration
• Public Administration Review
• Policy Studies Journal
• Social Policy and Administration
• Policy and Society
• Government Information Quarterly
• Journal of Public Policy
• Behavioural Public Policy
• Policy Design and Practice
• Policy and Politics

Query terms

Round 1

• RCT
• randomised/randomized control(led) trial
• field trial
• quasi-experiment
• systematic review of evidence
• return on investment / ROI
• natural experiment
• control
• cost benefit/cost-benefit analysis
• counter-factual/counter factual
• benefits OR impact
• public value
• public design
• design

Round 2

Search terms related to evaluation methodologies

• RCT
• randomised/randomized control(led) trial
• experiment
• quasi-experiment
• natural experiment
• counter-factual
• control
• systematic review
• ROI
• Return on investment
• cost benefit
• benefits
• impact
• evaluation

Search terms related to types of design and public design

• design
• public design
• policy design
• service design
• design for health
• design for education
• strategic design
• systems design
• government innovation
• experience design
• interaction design
• co-design

Search terms related to public value

• public value
• public sector innovation
• public innovation

Example query

Round 1

(“RCT”) OR (“randomised control”) OR (“trial”) OR (“experiment”) OR (“quasi- experiment”) OR (“natural experiment”) OR (“systematic review”) OR (“ROI”) OR (“return on investment”) OR (“control”) OR (“cost benefit”) OR (“counter-factual”) OR (“benefits”) OR (“impact”) OR (“public design”) OR (“public value”) OR (“evaluation”)

Round 2

(“RCT” OR “randomised control” OR “trial” OR “experiment” OR “quasi-experiment” OR “natural experiment” OR “systematic review” OR “ROI” OR “return on investment” OR “control” OR “cost benefit” OR “counter-factual” OR “benefits” OR “impact” OR “public design” OR “evaluation”) AND (“public value” OR “public sector innovation” OR “public innovation” “policy design” OR “service design” OR “design for health” OR “design for education” OR “strategic design” OR “systems design” OR “government innovation” OR “experience design” OR “interaction design” OR “co-design” OR “design”)

Appendix 3. Search results (by journal) – design journals round 1

Design Studies

Query:

(“randomised control”) OR (“trial”) OR (“experiment”) OR (“systematic review”) OR (“return on investment”) OR (“control”) OR (“cost benefit”) OR (“counter-factual”) OR (“impact”)

[in: Title, abstract or author-specified keywords; Design Studies; 2000-2023]

Hits:

159

Selected: 15

References:

Alipour, L., Faizi, M., Moradi, A. M., and Akrami, G. (2017). The impact of designers’ goals on design-by-analogy. Design Studies, 51, 1-24. Available at: The impact of designers’ goals on design-by-analogy

Balters, S., Weinstein, T., Mayseless, N., Auernhammer, J., Hawthorne, G., Steinert, M., and Reiss, A. L. (2023). Design science and neuroscience: A systematic review of the emergent field of Design Neurocognition. Design Studies, 84, 101148. Available at: Design science and neuroscience: A systematic review of the emergent field of design neurocognition

Belmonte, M. V., Millán, E., Ruiz-Montiel, M., Badillo, R., Boned, J., Mandow, L., and Pérez-de-la-Cruz, J. L. (2014). Randomness and control in design processes: An empirical study with architecture students. Design Studies, 35(4), 392-411. Finding: preferences for control or randomness among designers. Available at: Randomness and control in design processes: An empirical study with architecture students

Cash, P., Elias, E., Dekoninck, E., and Culley, S. (2012). Methodological insights from a rigorous small scale design experiment. Design Studies, 33(2), 208-235. Available at: Methodological insights from a rigorous small scale design experiment

Cash, P., Dekoninck, E., and Ahmed-Kristensen, S. (2020). Work with the beat: How dynamic patterns in team processes affect shared understanding. Design Studies, 69, 100943. Finding: comparative study of how ‘dynamic patterns’ contribute to the development of shared understanding. Available at: Work with the beat: How dynamic patterns in team processes affect shared understanding

Collado-Ruiz, D., and Ostad-Ahmad-Ghorabi, H. (2010). Influence of environmental information on creativity. Design Studies, 31(5), 479-498. Finding: different sorts and levels of information – commonly available for designers – was delivered to 56 people, all of whom were asked to generate different design ideas. ‘Soft info’ reduces fixation and improves creativity. Available at: Influence of environmental information on creativity

Dahl, D. W., Chattopadhyay, A., and Gorn, G. J. (2001). The importance of visualisation in concept design. Design Studies, 22(1), 5-26. Finding: imaginative visualisation improves design, imagining the end-user improves design. Visualising the end user, in combination with imagination-based visualisation, led to designs that were significantly more appealing to the customer. Available at: The importance of visualisation in concept design

Dong, A., Lovallo, D., and Mounarath, R. (2015). The effect of abductive reasoning on concept selection decisions. Design studies, 37, 37-58. Finding: abductive reasoning plays a significant role in concept generation and selection. Available at: The effect of abductive reasoning on concept selection decisions

Flager, F., Gerber, D. J., and Kallman, B. (2014). Measuring the impact of scale and coupling on solution quality for building design problems. Design Studies, 35(2), 180-199. Finding: problem scale decreases solution quality. Available at: Measuring the impact of scale and coupling on solution quality for building design problems

Hernandez, N. V., Shah, J. J., and Smith, S. M. (2010). Understanding design ideation mechanisms through multi-level aligned empirical studies. Design studies, 31(4), 382-410. Finding: effectiveness of six different ideation components. Available at: Understanding design ideation mechanisms through multilevel aligned empirical studies

Kokotovich, V., and Purcell, T. (2000). Mental synthesis and creativity in design: an experimental examination. Design studies, 21(5), 437-449. Finding: experiments with 3d designers, 2d designers and non-designers. Designers are more creative than non-designers. Available at: Mental synthesis and creativity in design: an experimental examination

Lee, J. H., and Ostwald, M. J. (2022). The relationship between divergent thinking and ideation in the conceptual design process. Design Studies, 79, 101089. Finding: review of 35 experiments suggests a causal link between divergent thinking and ideation. Available at: The relationship between divergent thinking and ideation in the conceptual design process

Nguyen, M., and Mougenot, C. (2022). A systematic review of empirical studies on multidisciplinary design collaboration: Findings, methods, and challenges. Design Studies, 81, 101120. Finding: found five thematic clusters, all show the importance of shared understanding for MDC. Available at: A systematic review of empirical studies on multidisciplinary design collaboration: Findings, methods, and challenges

Ou, X., Goldschmidt, G., and Erez, M. (2023). The effect of disciplinary diversity on design idea generation in dyadic teams. Design Studies, 86, 101184. Finding: disciplinary diversity improves quality of design ideas. Available at: The effect of disciplinary diversity on design idea generation in dyadic teams

Tromp, N., and Hekkert, P. (2016). Assessing methods for effect-driven design: Evaluation of a social design method. Design Studies, 43, 24-47. Available at: Assessing methods for effect-driven design: Evaluation of a social design method

Design issues

N.B. no advanced search functionality.

Table 1. Search queries for “design issues”

Queries	hits	relevant
“randomised control trial”	1	0
“field trial”	4	0
“quasi-experiment”	2	0
“systematic review”	3	0
“return on investment”	11	0
“natural experiment”	2	0
“cost benefit“	6	0
“counter factual“	1	0
“public value”	5	0
“public design”	45	0

Design Science

N.B. no advanced search functionality.

Table 2. Search queries for “design science”

Queries	hits	relevant
randomised control trial	14	0
field trial	26	1
quasi-experiment	43	0
systematic review	49	1
return on investment	13	0
natural experiment	61	0
cost benefit	51	0
counter factual	0	0
public value	31	0
Experiment [not full text]	43	0

2000-2023

Brun, J., Le Masson, P., and Weil, B. (2016). Designing with sketches: The generative effects of knowledge preordering. Design Science, 2, E13. Finding: sketching helps the architect mobilise knowledge distant from the initial topic. And helps knowledge structuring. Available at: Designing with sketches: The generative effects of knowledge preordering

Hay, L., Duffy, A., McTeague, C., Pidgeon, L., Vuletic, T., and Grealy, M. (2017). A systematic review of protocol studies on conceptual design cognition: Design as search and exploration. Design Science, 3, E10. Finding: reviews protocol studies of design cognition. Mostly just taxonomies (design as search, design as exploration…). Available at: A systematic review of protocol studies on conceptual design cognition: Design as search and exploration

Design and Culture

Query:

(“randomised control”) OR (“trial”) OR (“experiment”) OR (“systematic review”) OR (“return on investment”) OR (“control”) OR (“cost benefit”) OR (“counter-factual”) OR (“impact”)

[in: Title, abstract and keywords; Design and Culture; 2000-2023]

Hits:

15

Selected:

0

International Journal of Design, Creativity and Innovation

Query:

(“RCT”) OR (“randomised control”) OR (“trial”) OR (“experiment”) OR (“quasi- experiment”) OR (“natural experiment”) OR (“systematic review”) OR (“ROI”) OR (“return on investment”) OR (“control”) OR (“cost benefit”) OR (“counter-factual”) OR (“benefits”) OR (“impact”) OR (“public design”) OR (“public value”)

[in: Title, abstract and keywords; Design and Culture; 2000-2023

Hits:

28

Selected:

5

Becattini, N., Borgianni, Y., Cascini, G., and Rotini, F. (2017). Surprise and design creativity: investigating the drivers of unexpectedness. International journal of design creativity and innovation, 5(1-2), 29-47. Finding: (experiment) effect of surprise on design idea generation. Available at: Surprise and design creativity: investigating the drivers of unexpectedness

Coelho, D. A., and Vieira, F. L. (2018). The effect of previous group interaction on individual ideation novelty and variety. International Journal of Design Creativity and Innovation, 6(1-2), 80-92. Finding: (experiment) group interaction > design creativity. Available at: The effect of previous group interaction on individual ideation novelty and variety

Masaya, S., Fujii, K., Watanabe, K., Chen, C., Hai, L., Zhang, J., … and Hirose, Y. (2023). An exploratory study for the impact of video-induced positive moods on ideation. International Journal of Design Creativity and Innovation, 11(1), 26-41. Finding: (experiment) effect of positive mood stimuli on creativity. Available at: An exploratory study for the impact of video-induced positive moods on ideation

Vasantha, G. V. A., Chakrabarti, A., Rout, B. K., and Corney, J. (2014). Influences of design tools on the original and redesign processes. International Journal of Design Creativity and Innovation, 2(1), 20-50. Finding: (experiment) 26 variables, from captured documents and video protocols, reveal that the design tools had a statistically significant impact on four key variables. These key variables are: the total time taken to solve each problem; the time spent on detailed design activity; the textual representation of structural requirements; the graphical representation of the structure of detailed concepts. Available at: Influences of design tools on the original and redesign processes

Worinkeng, E., Joshi, S., and Summers, J. D. (2015). An experimental study: analyzing requirement type influence on novelty and variety of generated solutions. International Journal of Design Creativity and Innovation, 3(2), 61-77. Finding: (experiment) non-functional design requirements > novelty scores. Available at: An experimental study: analyzing requirement type influence on novelty and variety of generated solutions

Journal of Design Research

Query:

(“RCT”) OR (“randomised control”) OR (“trial”) OR (“experiment”) OR (“quasi-experiment”) OR (“natural experiment”) OR (“systematic review”) OR (“ROI”) OR (“return on investment”) OR (“control”) OR (“cost benefit”) OR (“counter-factual”) OR (“benefits”) OR (“impact”) OR (“public design”) OR (“public value”)

[in: anywhere; Journal of Design Research; 2000-2023]

Hits:

4

Selected:

0

Human-Computer Interaction

Query:

(“RCT”) OR (“randomised control”) OR (“trial”) OR (“experiment”) OR (“quasi- experiment”) OR (“natural experiment”) OR (“systematic review”) OR (“ROI”) OR (“return on investment”) OR (“control”) OR (“cost benefit”) OR (“counter-factual”) OR (“benefits”) OR (“impact”) OR (“public design”) OR (“public value”) OR (“evaluation”)

[in: title, abstract; Human-Computer Interaction; 2000-2023]

Hits:

38 [in title]; 178 [in abstract]

Selected:

1

Gergle, D., Kraut, R. E., and Fussell, S. R. (2013). Using visual information for grounding and awareness in collaborative tasks. Human-Computer Interaction, 28(1), 1-39. Available at: Using Visual Information for Grounding and Awareness in Collaborative Tasks

She Ji: The Journal of Design, Economics, and Innovation

Query:

(“randomised control”) OR (“trial”) OR (“experiment”) OR (“systematic review”) OR (“ROI”) OR (“return on investment”) OR (“control”) OR (“impact”) OR (“evaluation”)

[in: title, abstract, keywords; she-ji; 2000-2023]

Hits:

41

Selected:

2

Cockbill, S. A., May, A., and Mitchell, V. (2019). The assessment of meaningful outcomes from co-design: a case study from the energy sector. She Ji: The Journal of Design, Economics, and Innovation, 5(3), 188-208. Available at: The assessment of meaningful outcomes from co-design: a case study from the energy sector

Price, R. A., De Lille, C., and Bergema, K. (2019). Advancing industry through design: A longitudinal case study of the aviation industry. She Ji: The Journal of Design, Economics, and Innovation, 5(4), 304-326. Available at: Advancing industry through design: A longitudinal case study of the aviation industry

The Design Journal

Query:

(“RCT”) OR (“randomised control”) OR (“trial”) OR (“experiment”) OR (“quasi- experiment”) OR (“natural experiment”) OR (“systematic review”) OR (“ROI”) OR (“return on investment”) OR (“control”) OR (“cost benefit”) OR (“counter-factual”) OR (“benefits”) OR (“impact”) OR (“public design”) OR (“public value”) OR (“evaluation”)

[in: title, abstract; The Design Journal; 2000-2023]

Hits:

23 [in title], 210 [in abstract]

Selected:

3

Corazzo, J. (2019). Materialising the Studio. A systematic review of the role of the material space of the studio in Art, Design and Architecture Education. The Design Journal, 22 (supplement 1), 1249-1265. Available at: Materialising the Studio. A systematic review of the role of the material space of the studio in Art

Sadowska, N., and Laffy, D. (2019). Measuring the impact of strategic design learning experience long after the classroom delivery. The Design Journal, 22 (supplement 1), 1305-1315. Available at: Measuring the impact of strategic design learning experience long after the classroom delivery

van Dam, S., Sleeswijk Visser, F., and Bakker, C. (2021). The impact of co-creation on the design of circular product-service systems: Learnings from a case study with washing machines. The Design Journal, 24(1), 25-45. Available at: The impact of co-creation on the design of circular product-service systems: Learnings from a case study with washing machines

Design Research Society

Query:

“RCT” OR “randomised control” OR “trial” OR “experiment” OR “quasi- experiment” OR “natural experiment” OR “systematic review” OR “ROI” OR “return on investment” OR “control” OR “cost benefit” OR “counter-factual” OR “benefits” OR “impact” OR “public design” OR “public value”

[in: all; DRS Biennial Conference Series]

Hits:

164

Selected:

4

Lievesley, M., and Yee, J. (2012) Valuing Service Design: Lessons from Social Return on Investment (SROI). In: Israsena, P., Tangsantikul, J. and Durling, D. (eds.). Research: Uncertainty Contradiction Value - DRS International Conference 2012, 1-4 July, Bangkok, Thailand. Finding: uses SROI to evaluate a public sector training programme. Available at: Valuing Service Design: Lessons from Social Return on Investment (SROI).

Manohar, A., Smith, M., and Calvo, M. (2016). Capturing the “How”: Showing the value of codesign through creative evaluation. In: Lloyd, P. and Bohemia, E. (eds.). Future Focused Thinking - DRS International Conference 2016, 27 - 30 June, Brighton, United Kingdom. Finding: using creative evaluation (CE). Available at: Capturing the “How”: Showing the value of codesign through creative evaluation.

Owen, V., Ceyhan, P., Cruickshank, L., and Christou, E. (2022). Evaluating social innovations: How creative evaluation can help articulate their values and impacts. In:  Lockton, D., Lenzi, S., Hekkert, P., Oak, A., Sádaba, J., Lloyd, P. (eds.). DRS2022: Bilbao, 25 June - 3 July, Bilbao, Spain. Finding: review of existing evaluation methods, plus intro to ‘creative evaluation’. Available at: Evaluating social innovations: How creative evaluation can help articulate their values and impacts

Wang, Z., Jiang, T., Huang, J., Tai, Y., and Trapani, P.M. (2022). How might we evaluate co-design? A literature review on existing practices. In: Lockton, D., Lenzi, S., Hekkert, P., Oak, A., Sádaba, J., Lloyd, P. (eds.). DRS2022: Bilbao, 25 June - 3 July, Bilbao, Spain. Finding: review of existing evaluation methods. Available at: How might we evaluate co-design? A literature review on existing practices.

Creativity and Innovation Management

Query:

(“RCT”) OR (“randomised control”) OR (“trial”) OR (“experiment”) OR (“quasi- experiment”) OR (“natural experiment”) OR (“systematic review”) OR (“ROI”) OR (“return on investment”) OR (“control”) OR (“cost benefit”) OR (“counter-factual”) OR (“benefits”) OR (“impact”) OR (“public design”) OR (“public value”) OR (“evaluation”)  

[in: title, abstract, keywords; Creativity and Innovation Management; 2000-2023]

Hits:

1

Selected:

0

Journal of Management Information Systems

Query:

(“RCT” OR “randomised control” OR “trial” OR “experiment” OR “quasi- experiment” OR “natural experiment” OR “systematic review” OR “ROI” OR “return on investment” OR “control” OR “cost benefit” OR “counter-factual” OR “benefits” OR “impact” OR “evaluation”) AND (“public design” OR “public value” OR “service design” OR “policy design” OR “public sector innovation”)

[in all; Journal of Management Information Systems; 2000-2023]

Hits:

19

Selected:

0

Journal of Organizational Behavior

Query:

(“RCT”) OR (“randomised control”) OR (“trial”) OR (“experiment”) OR (“quasi- experiment”) OR (“natural experiment”) OR (“systematic review”) OR (“ROI”) OR (“return on investment”) OR (“control”) OR (“cost benefit”) OR (“counter-factual”) OR (“benefits”) OR (“impact”) OR (“public design”) OR (“public value”) OR (“evaluation”)

[in title, abstract, keywords; Journal of Organizational Behavior; 2000-2023]

Hits:

20

Selected:

0

Appendix 4. Search results (by journal) – design journals round 2

Creativity and Innovation Management

Query:

(“RCT” OR “randomised control” OR “trial” OR “experiment” OR “quasi-experiment” OR “natural experiment” OR “systematic review” OR “ROI” OR “return on investment” OR “control” OR “cost benefit” OR “counter-factual” OR “benefits” OR “impact” OR “public design” OR “evaluation”) AND (“public value” OR “public sector innovation” OR “public innovation” “policy design” OR “service design” OR “design for health” OR “design for education” OR “strategic design” OR “systems design” OR “government innovation” OR “experience design” OR “interaction design” OR “co-design” OR “design”)          

[in: all; Creativity and Innovation Management]

Hits:

8

Selected:

1

Tang, C., Zhang, Y., and Reiter‐Palmon, R. (2020). Network centrality, knowledge searching and creativity: The role of domain. Creativity and Innovation Management, 29(1), 72-84. Available at: Network centrality, knowledge searching and creativity: The role of domain

Journal of Management Information Systems

Query:

(“RCT” OR “randomised control” OR “trial” OR “experiment” OR “quasi-experiment” OR “natural experiment” OR “systematic review” OR “ROI” OR “return on investment” OR “control” OR “cost benefit” OR “counter-factual” OR “benefits” OR “impact” OR “public design” OR “evaluation”) AND (“public value” OR “public sector innovation” OR “public innovation” “policy design” OR “service design” OR “design for health” OR “design for education” OR “strategic design” OR “systems design” OR “government innovation” OR “experience design” OR “interaction design” OR “co-design” OR “design”)          

[in: all; Journal of Management Information Systems]

Hits:

3

Selected:

0

Journal of Organizational Behavior

Query:

(“RCT” OR “randomised control” OR “trial” OR “experiment” OR “quasi-experiment” OR “natural experiment” OR “systematic review” OR “ROI” OR “return on investment” OR “control” OR “cost benefit” OR “counter-factual” OR “benefits” OR “impact” OR “public design” OR “evaluation”) AND (“public value” OR “public sector innovation” OR “public innovation” “policy design” OR “service design” OR “design for health” OR “design for education” OR “strategic design” OR “systems design” OR “government innovation” OR “experience design” OR “interaction design” OR “co-design” OR “design for policy” OR “design”)      

[in: all; Journal of Organizational Behavior]

Hits:

3

Selected:

0

Government Information Quarterly

Query:

(“experiment” OR “evaluation” OR “systematic review”) AND (“service design” OR “policy design” OR “strategic design” OR “systems design” OR “public sector innovation” OR “co-design”)        

[in: all; Government Information Quarterly; 8 booleans max]

Hits:

272

Selected:

1

Allen, B., Tamindael, L. E., Bickerton, S. H., and Cho, W. (2020). Does citizen coproduction lead to better urban services in smart cities projects? An empirical study on e-participation in a mobile big data platform. Government Information Quarterly, 37(1), 101412. Available at: Does citizen coproduction lead to better urban services in smart cities projects? An empirical study on e-participation in a mobile big data platform

Co-design

Query:

(“experiment” OR “evaluation” OR “systematic review” OR “rct” OR “impact”)

[in: all; CoDesign]

Hits:

327

Selected:

13

Bornet, C., and Brangier, E. (2016). The effects of personas on creative codesign of work equipment: an exploratory study in a real setting. CoDesign, 12(4), 243-256. Available at: The effects of personas on creative codesign of work equipment: an exploratory study in a real setting

Bowen, S., McSeveny, K., Lockley, E., Wolstenholme, D., Cobb, M., and Dearden, A. (2013). How was it for you? Experiences of participatory design in the UK health service. CoDesign, 9(4), 230-246. Available at: How was it for you? Experiences of participatory design in the UK health service

Corcoran, R., Marshall, G., and Walsh, E. (2018). The psychological benefits of cooperative place-making: a mixed methods analyses of co-design workshops. CoDesign, 14(4), 314-328. Available at: The psychological benefits of cooperative place-making: a mixed methods analyses of co-design workshops

Fraser, K. (2009). Labour flexibility: Impact of functional and localised strategies on team-based product manufacturing. CoDesign, 5(3), 143-158. Available at: Labour flexibility: impact of functional and localised strategies on team-based product manufacturing

Hong, S. W., Jeong, Y., Kalay, Y. E., Jung, S., and Lee, J. (2016). Enablers and barriers of the multi-user virtual environment for exploratory creativity in architectural design collaboration. CoDesign, 12(3), 151-170. Available at: Enablers and barriers of the multi-user virtual environment for exploratory creativity in architectural design collaboration

Karlgren, K., and Ramberg, R. (2012). The use of design patterns in overcoming misunderstandings in collaborative interaction design. CoDesign, 8(4), 231-246. Available at: The use of design patterns in overcoming misunderstandings in collaborative interaction design

Kleinsmann, M., Valkenburg, R., and Buijs, J. (2007). Why do(n’t) actors in collaborative design understand each other? An empirical study towards a better understanding of collaborative design. CoDesign, 3(1), 59-73. Available at: Why do(n’t) actors in collaborative design understand each other? An empirical study towards a better understanding of collaborative design

Perttula, M. K., Krause, C. M., and Sipilä, P. (2006). Does idea exchange promote productivity in design idea generation? CoDesign, 2(3), 125-138. Available at: Does idea exchange promote productivity in design idea generation?

Mitchell, V., Ross, T., May, A., Sims, R., and Parker, C. (2016). Empirical investigation of the impact of using co-design methods when generating proposals for sustainable travel solutions. CoDesign, 12(4), 205-220. Available at: Empirical investigation of the impact of using co-design methods when generating proposals for sustainable travel solutions

Steen, M., Arendsen, J., Cremers, A., De Jong, A., De Jong, J., and De Koning, N. (2013). Using interactive model simulations in co-design: An experiment in urban design. CoDesign, 9(1), 2-16. Available at: Using interactive model simulations in co-design: An experiment in urban design

Taoka, Y., Kagohashi, K., and Mougenot, C. (2018). A cross-cultural study of co-design: the impact of power distance on group dynamics in Japan. CoDesign, 17(1), 22-49. Available at: A cross-cultural study of co-design: the impact of power distance on group dynamics in Japan

Van Rijn, H., Sleeswijk Visser, F., Stappers, P. J., and Özakar, A. D. (2011). Achieving empathy with users: the effects of different sources of information. CoDesign, 7(2), 65-77. Available at: Achieving empathy with users: the effects of different sources of information

Villalba, C., Jaiprakash, A., Donovan, J., Roberts, J., and Crawford, R. (2019). Testing literature-based health experience insight cards in a healthcare service co-design workshop. CoDesign, 17(1), 70-82. Available at: Testing literature-based health experience insight cards in a healthcare service co-design workshop

Design for Health

Query:

(“experiment” OR “evaluation” OR “systematic review” OR “rct” OR “impact)

[in: all; Design for Health]

Hits:

137

Selected:

0

Journal of Health Design

Query:

(“experiment” OR “evaluation” OR “systematic review” OR “rct” OR “impact”)

[in: all; Journal of Health Design]

Hits:

20

Selected:

0

Designs for Learning

Query:

(“experiment” OR “evaluation” OR “systematic review” OR “rct” OR “impact”)        

[in: all; Designs for Learning

Hits:

129

Selected:

0

International Journal of Technology and Design Education

Query:

(“experiment” OR “evaluation” OR “systematic review” OR “rct” OR “impact” OR “control” OR “evaluation”) 

[in: all; International Journal of Technology and Design Education; via Google Scholar]

Hits:

60

Selected:

7

Ateş Akdeniz, A. (2022). Exploring the impact of self-regulated learning intervention on students’ strategy use and performance in a design studio course. International Journal of Technology and Design Education, 33(5), 1923-1957. Available at: Exploring the impact of self-regulated learning intervention on students’ strategy use and performance in a design studio course

Balakrishnan, B. (2022). Exploring the impact of design thinking tool among design undergraduates: a study on creative skills and motivation to think creatively. International Journal of Technology and Design Education, 32(3), 1799-1812. Available at: Exploring the impact of design thinking tool among design undergraduates: a study on creative skills and motivation to think creatively

Conradie, P. D., Van Acker, B. B., De Vos, E., and Saldien, J. (2021). Impact of user involvement on design students’ motivation and self-confidence. International Journal of Technology and Design Education, 31, 183-197. Available at: Impact of user involvement on design students’ motivation and self-confidence

Guaman-Quintanilla, S., Everaert, P., Chiluiza, K., and Valcke, M. (2023). Impact of design thinking in higher education: a multi-actor perspective on problem solving and creativity. International Journal of Technology and Design Education, 33(1), 217-240. Available at: Impact of design thinking in higher education: a multi-actor perspective on problem solving and creativity

Huffman, T. J., and Mentzer, N. (2021). The impact of modeling-eliciting activities on high school student design performance. International Journal of Technology and Design Education, 31(2), 255-280. Available at: The impact of modeling-eliciting activities on high school student design performance

Ji, Y., Qiu, Q., Feng, P., and Wu, J. (2019). Empirical study on the impact of knowledge in international patent classification on design inspiration of undergraduate students. International Journal of Technology and Design Education, 29, 803-820. Available at: Empirical study on the impact of knowledge in international patent classification on design inspiration of undergraduate students

Sun, L., Xiang, W., Chai, C., Wang, C., and Liu, Z. (2013). Impact of text on idea generation: an electroencephalography study. International Journal of Technology and Design Education, 23, 1047-1062. Available at: Impact of text on idea generation: an electroencephalography study

Journal of Design Research

Query:

(“experiment” OR “evaluation” OR “systematic review” OR “RCT” OR “impact” OR “control” OR evaluation”)

[in: all; Journal of Design Research; via Google Scholar]

Hits:

323

Selected:

4

Casakin, H. (2004). Visual analogy as a cognitive strategy in the design process. Expert versus novice performance. Journal of Design Research, 4(2), 197-217. Available at: Visual analogy as a cognitive strategy in the design process. Expert versus novice performance

Kim, Y. S., Lee, S. W., Kim, M. S., Park, J. A., and Jeong, J. Y. (2011). An exercise programme for cognitive elements of design creativity. Journal of Design Research, 9(2), 185-202. Available at: An exercise programme for cognitive elements of design creativity

Mougenot, C., Bouchard, C., Aoussat, A., and Westerman, S. (2008). Inspiration, images and design: an investigation of designers’ information gathering strategies. Journal of Design Research, 7(4), 331-351. Available at: Inspiration, images and design: an investigation of designers’ information gathering strategies

Sun, G., Yao, S., and Carretero, J. A. (2016). An experimental approach to understanding design problem structuring strategies. Journal of Design Research, 14(1), 94-117. Available at: An experimental approach to understanding design problem structuring strategies

DISCERN: International Journal of Design for Social change, Sustainable Innovation and Entrepreneurship

[browsed all issues (4)]

Selected:

0

Social Design Review

Query:

(“experiment” OR “evaluation” OR “systematic review” OR “RCT” OR “impact” OR “control” OR “evaluation”)

[in: all; Social Design Review; via Google Scholar]

Hits:

2

Selected:

0

International Journal of Design in Society

Query:

(“experiment” OR “evaluation” OR “systematic review” OR “RCT” OR “impact” OR “control” OR “evaluation”)

[in: all; International Journal of Design in Society; via Google Scholar]

Hits:

148

Selected:

0

Strategic Design Research Journal

Query:

(“experiment” OR “evaluation” OR “systematic review” OR “RCT” OR “impact” OR “control” OR “evaluation”)

[in: all; Strategic Design Research Journal; via Google Scholar]

Hits:

260

Selected:

1

Marques da Rosa, V., Meyer, E., Wolff, F., Miletto Tonetto, L., and Brust-Renck, P. G. (2022). A Systematic Review of Design and Wellbeing. Strategic Design Research Journal, 15(1). Available at: A Systematic Review of Design and Wellbeing (PDF file. Size 444KB)

Link to Google Scholar search query used: https://scholar.google.com/scholar?start=100andq=experiment+OR+evaluation+OR+%22systematic+review%22+OR+rct+OR+impact+OR+control+OR+evaluat+source:%22Strategic+Design+Research+Journal%22andhl=enandas_sdt=0,5

Other references

Green, G. (2011). Experimental validation of the R³I design evaluation methodology. Journal of Design Research, 9(1), 1-15. Available at: Experimental validation of the R³I design evaluation methodology

Jukić, T., Pluchinotta, I., and Vrbek, S. (2022). Organizational maturity for co-creation: towards a multi-attribute decision support model for public organizations. Government Information Quarterly, 39(1), 101623. Available at: Organizational maturity for co-creation: towards a multi-attribute decision support model for public organizations

Seravalli, A., Agger Eriksen, M., and Hillgren, P. A. (2017). Co-Design in co-production processes: jointly articulating and appropriating infrastructuring and commoning with civil servants. CoDesign, 13(3), 187-201. Available at: Co-Design in co-production processes: jointly articulating and appropriating infrastructuring and commoning with civil servants

Vaajakallio, K., and Mattelmäki, T. (2014). Design games in codesign: as a tool, a mindset and a structure. CoDesign, 10(1), 63-77. Available at: Design games in codesign: as a tool, a mindset and a structure

Vaz, F., and Ferreira, M. (2021). Assessing design approaches’ political role in the public sector. Journal of Design Research, 19(4-6), 197-212. Available at: Assessing design approaches’ political role in the public sector

Appendix 5. Search results (by journal) – political science /public administration journals round 1

Search query methodology

The first round focused on top 10 journals in the ‘Public Policy and Administration’ sub-category of Google Scholar. On top of those, I have added two journals in the same category that I expected to see as relevant for the subject at hand: ‘Public Money and Management’ and ‘Public Performance and Management Review’. The same query was used and adapted, depending on each journal’s available solutions for advanced search (i.e. some journals permitted the combined use of nested Boolean connectors, others did not) and focused on the period 2000-2023. The research yielded 17 papers. A few additional notes on these results:

• Almost no paper focuses exactly on ‘design’ and ‘public value’ as such. To get results, one has to loosen the definition of both and include tangential strands of research, e.g. many empirical papers focus on co-production and trust.
• Adding ‘public design’ instead of ‘design’ in the query brings the count to zero in all the journals I researched.
• Despite the mandatory inclusion of ‘public value’ in the query, few papers reference public value theory and none of them quantifies it.
• Having a stronger conceptualisation of the connection between public design and public value would help us ‘expand’ the search for evidence, beyond the boundaries imposed by these two words. For example: paper (2) below identifies how greater proximity to users increases public service performance and motivation. In the paper, the determinant of proximity is not design but leadership. Greater proximity is also a feature that can be reasonably attributed to the practice of design. By looking only at the ‘macro-level’ causal chain (public design à public value) are we missing out on the ‘micro-level’ causal chain (public design à features of public design à public value)?
• The criteria used for paper selection are the following:
  • Includes design or similar (e.g. co-production) as an independent variable
  • AND/OR includes public value or similar (e.g. trust) as a dependent variable
  • AND showcases hard evidence (qualitative, quantitative or systematic review)
  • AND addresses (directly or indirectly) one of the four targeted research questions within Topic 3

Government Information Quarterly

Query:

(“design” OR “agile”) AND (“public value”) AND (“impact” OR “evaluation” OR assessment”) AND (“randomized controlled trial” OR “experiment” OR “systematic review”)

Year:

2000-2023

Result:

79

Selected:

3

Public Management Review

Query:

[[All: ”design”] OR [All: ”agile”]] AND [All: ”public value”] AND [[All: ”impact”] OR [All: ”evaluation”] OR [All: assessment”]] AND [[All: ”randomized controlled trial”] OR [All: ”experiment”] OR [All: ”systematic review”]] AND [in Journal: Public Management Review] AND [Publication Date: (01/01/2000 TO 12/31/2023)]

Result:

56

Selected:

6

Public Administration Review

Query:

[[All: ”design”] OR [All: ”agile”]] AND [All: ”public value”] AND [[All: ”impact”] OR [All: ”evaluation”] OR [All: assessment”]] AND [[All: ”randomized controlled trial”] OR [All: ”experiment”] OR [All: ”systematic review”]] AND [in Journal: Public Administration Review] AND [Publication Date: (01/01/2000 TO 12/31/2023)]

Result:

33
Selected:

0

American Review of Public Administration

Query:

[[All: ”design”] OR [All: ”agile”]] AND [All: ”public value”] AND [[All: ”impact”] OR [All: ”evaluation”] OR [All: assessment”]] AND [[All: ”randomized controlled trial”] OR [All: ”experiment”] OR [All: ”systematic review”]] AND [in Journal: American Review of Public Administration] AND [Publication Date: (01/01/2000 TO 12/31/2023)]

Result:

 5

Selected:

0

Journal of Public Administration Research and Theory

Query:

[Keywords: “design”] AND “public value” AND “impact” AND “evidence”; 2000-2023 

Result:

0

Selected:

2 (found afterwards)

Policy Studies Journal

Query:

[[All: ”design”] OR [All: ”agile”]] AND [All: ”public value”] AND [[All: ”impact”] OR [All: ”evaluation”] OR [All: assessment”]] AND [[All: ”randomized controlled trial”] OR [All: ”experiment”] OR [All: ”systematic review”]] AND [in Journal: Policy Studies Journal] AND [Publication Date: (01/01/2000 TO 12/31/2023)]

Result:

 5

Selected:

0

Social Policy and Administration

Query:

[[All: ”design”] OR [All: ”agile”]] AND [All: ”public value”] AND [[All: ”impact”] OR [All: ”evaluation”] OR [All: assessment”]] AND [[All: ”randomized controlled trial”] OR [All: ”experiment”] OR [All: ”systematic review”]] AND [in Journal: Social Policy and Administration] AND [Publication Date: (01/01/2000 TO 12/31/2023)]

Result:

0

Selected:

0

Policy and Society

Query:

[Keywords: “design”] AND “public value” AND “impact” AND “evidence”; 2000-2023 

Result:

16

Selected:

1

International Journal of Public Administration

Query:

[[All: ”design”] OR [All: ”agile”]] AND [All: ”public value”] AND [[All: ”impact”] OR [All: ”evaluation”] OR [All: assessment”]] AND [[All: ”randomized controlled trial”] OR [All: ”experiment”] OR [All: ”systematic review”]] AND [in Journal: International Journal of Public Administration] AND [Publication Date: (01/01/2000 TO 12/31/2023)]

Result:

16

Selected:

3

Public Administration

Query:

[[All: ”design”] OR [All: ”agile”]] AND [All: ”public value”] AND [[All: ”impact”] OR [All: ”evaluation”] OR [All: assessment”]] AND [[All: ”randomized controlled trial”] OR [All: ”experiment”] OR [All: ”systematic review”]] AND [in Journal: Public Administration] AND [Publication Date: (01/01/2000 TO 12/31/2023)]

Result:

13

Selected:

0

Extra: Public Money and Management

Query:

[[All: ”design”] OR [All: ”agile”]] AND [All: ”public value”] AND [[All: ”impact”] OR [All: ”evaluation”] OR [All: assessment”]] AND [[All: ”randomized controlled trial”] OR [All: ”experiment”] OR [All: ”systematic review”]] AND [in Journal: Public Money and Management] AND [Publication Date: (01/01/2000 TO 12/31/2023)]

Result:

14

Selected:

2

Extra: Public Performance and Management Review

Query:

[[All: ”design”] OR [All: ”agile”]] AND [All: ”public value”] AND [[All: ”impact”] OR [All: ”evaluation”] OR [All: assessment”]] AND [[All: ”randomized controlled trial”] OR [All: ”experiment”] OR [All: ”systematic review”]] AND [in Journal: Public Performance and Management Review] AND [Publication Date: (01/01/2000 TO 12/31/2023)]

Result:

10

Selected:

0

Appendix 6. Search results (by journal) – political science /public administration journals round 2

Journal list

• International Journal of Public Administration
• Journal of Public Administration Research and Theory
• Public Management Review
• Public Administration
• Public Administration Review
• Policy Studies Journal
• Social Policy and Administration
• Policy and Society
• Government Information Quarterly
• Journal of Public Policy (new relative to first round)
• Behavioural Public Policy (new relative to first round)
• Policy Design and Practice (new relative to first round)
• Policy and Politics (new relative to first round)

Query terms

(RCT‘/‘randomised control trial‘, ‘field trial‘, ‘quasi-experiment‘, ‘systematic review of evidence‘, ‘ROI‘/‘return on investment‘, ‘natural experiment‘, ‘control‘, ‘cba‘/‘cost-benefit analysis‘, ‘counter factual‘, ‘benefits OR impact‘, ‘public value‘, ‘public design‘, ‘design‘)

Example query

(“randomised control trial” OR “experiment” OR “systematic review” OR “return on investment” OR “cost benefit” OR “counter-factual” OR “impact” OR “evaluation”) AND (“service design” OR “social design” OR “co-design” OR “policy design” OR “design for policy” OR “public design”)

Criteria used for selection

• Includes design or similar (e.g. co-production) as an independent variable
• AND showcases hard evidence (qualitative, quantitative or systematic review)
• AND addresses (directly or indirectly) one of the four targeted questions in Topic

Found 46 potentially useful papers (25 quantitative, 21 qualitative, 2 reviews). N.B.: 19 out of 25 quantitative-based papers come from Behaviour and Public Policy, and thus based on ‘nudging’ rather than ‘design thinking’ as such.

International Journal of Public Administration

Query:

(“randomised control trial” OR “experiment” OR “systematic review” OR “return on investment” OR “cost benefit” OR “counter-factual” OR “impact” OR “evaluation”) AND (“service design” OR “social design” OR “co-design” OR “policy design” OR “design for policy” OR “public design”)

[in: Title, abstract or author-specified keywords; 2000-2023]

Hits:

94

Added the condition “AND [(“design”) in: keyword]” due to sprawling results.

Hits:

4

Selected:

2

References:

Criado, J.I., Ferreira Dias, T.F., Sano, H., Rojas-Martín, F., Silvan, A. and Filho, A.I. (2021). Public Innovation and Living Labs in Action: A Comparative Analysis in post-New Public Management Contexts. International Journal of Public Administration, 44:6, 451-464. Available at: Public Innovation and Living Labs in Action: A Comparative Analysis in post-New Public Management Contexts

French, T. and Raman, S. (2021). Engaging People with Lived Experience in Co-design of Future Palliative Care Services. International Journal of Public Administration, 44:9, 778-789. Available at: Engaging People with Lived Experience in Co-design of Future Palliative Care Services

Journal of Public Administration Research and Theory

Query:

“design”

[in: Title or author-specified keywords; 2000-2023]

N.B. Oxford Academic’s search query doesn’t support nested Boolean connectors.

Hits:

34

Selected:

0

Public Management Review

Query:

(“randomised control trial” OR “experiment” OR “systematic review” OR “return on investment” OR “cost benefit” OR “counter-factual” OR “impact” OR “evaluation”) AND (“service design” OR “social design” OR “co-design” OR “policy design” OR “design for policy” OR “public design”)

[in: Title, abstract or author-specified keywords; 2000-2023]

Hits:

176

Added the condition “AND [(“design”) in: keyword]” due to sprawling results.

Hits:

5

Selected:

3

References:

Bason, C. and Austin. R.D. (2022). Design in the public sector: Toward a human centred model of public governance. Public Management Review, 24:11, 1727-1757. Available at: Design in the public sector: Toward a human centred model of public governance

Gofen, A., Meza, O. and Moreno-Jaimes, C. (2023). Frontline organizations as experimental settings for policy change: why public management matters even more. Public Management Review. Available at: Frontline organizations as experimental settings for policy change: why public management matters even more

Leite, H. and Hodgkinson, I.R. (2023). Examining Resilience Across a Service Ecosystem under Crisis. Public Management Review, 25:4, 690-709. Available at: Examining Resilience Across a Service Ecosystem under Crisis

Public Administration

Query:

(“randomised control trial” OR “experiment” OR “systematic review” OR “return on investment” OR “cost benefit” OR “counter-factual” OR “impact” OR “evaluation”) AND (“service design” OR “social design” OR “co-design” OR “policy design” OR “design for policy” OR “public design”)

[in: Title, abstract or author-specified keywords; 2000-2023]

Hits:

129

Added the condition “AND [(“design”) in: Keyword]” due to sprawling results.

Hits:

0

Added the condition “AND [(“design”) in: Title]” due to sprawling results.

Hits:

17

Selected:

2

References:

Dolšak, N, Adolph, C, Prakash, A. Policy design and public support for carbon tax: Evidence from a 2018 United States national online survey experiment. Public Admin. 2020; 98: 905–921. Available at: Policy design and public support for carbon tax: Evidence from a 2018 United States national online survey experiment

Montpetite, É. (2008). Policy design for legitimacy: expert knowledge, citizens, time and inclusion in the United Kingdom’s biotechnology sector. Public Administration, 86: 259-277. Available at: Policy design for legitimacy: expert knowledge, citizens, time and inclusion in the United Kingdom’s biotechnology sector

Public Administration Review

Query:

(“randomised control trial” OR “experiment” OR “systematic review” OR “return on investment” OR “cost benefit” OR “counter-factual” OR “impact” OR “evaluation”) AND (“service design” OR “social design” OR “co-design” OR “policy design” OR “design for policy” OR “public design”)

[in: Title, abstract or author-specified keywords; 2000-2023]

Hits:

145

Added the condition “AND [(“design”) in: Keyword]” due to sprawling results.

Hits:

0

Added the condition “AND [(“design”) in: Title]” due to sprawling results.

Hits:

4

Selected:

1

References:

Chan, F.K.Y., Thong, J.Y.L., Brown, S.A. and Venkatesh, V. (2021). Service Design and Citizen Satisfaction with E-Government Services: A Multidimensional Perspective. Public Admin Rev, 81: 874-894. Available at: Service Design and Citizen Satisfaction with E-Government Services: A Multidimensional Perspective

Policy Studies Journal

Query:

(“randomised control trial” OR “experiment” OR “systematic review” OR “return on investment” OR “cost benefit” OR “counter-factual” OR “impact” OR “evaluation”) AND (“service design” OR “social design” OR “co-design” OR “policy design” OR “design for policy” OR “public design”)

[in: Title, abstract or author-specified keywords; 2000-2023]

Hits:

158

Added the condition “AND [(“design”) in: Keyword]” due to sprawling results.

Hits:

17

Selected:

1

References:

Koski, C.and Siddiki, S. (2022). Linking policy design, change, and outputs: Policy responsiveness in American state electricity policy. Policy Studies Journal, 50, 553–574. Available at: Linking policy design, change, and outputs: Policy responsiveness in American state electricity policy

Social Policy and Administration

Query:

(“randomised control trial” OR “experiment” OR “systematic review” OR “return on investment” OR “cost benefit” OR “counter-factual” OR “impact” OR “evaluation”) AND (“service design” OR “social design” OR “co-design” OR “policy design” OR “design for policy” OR “public design”)

[in: Title, abstract or author-specified keywords; 2000-2023]

Hits:

124

Added the condition “AND [(“design”) in: Keyword]” due to sprawling results.

Hits:

1

Added the condition “AND [(“design”) in: Title]” due to sprawling results.

Hits:

4

Selected:

0

Policy and Society

Query:

“design”

[in: Title or author-specified keywords; 2000-2023]

N.B. Oxford Academic’s search query doesn’t support nested Boolean connectors.

Hits:

36

Selected:

4

References:

Bloemen, P., Van Der Steen, M. and Van Der Wal, Z. (2019). Designing a century ahead: climate change adaptation in the Dutch Delta. Policy and Society, 38(1), 58–76. Available at: Designing a century ahead: climate change adaptation in the Dutch Delta

Capano, G. and Pavan, E. (2019). Designing anticipatory policies through the use of ICTs. Policy and Society, 38(1), 96–117. Available at: Designing anticipatory policies through the use of ICTs

Mukherjee, I. and Mukherjee, N. (2018). Designing for sustainable outcomes: espousing behavioural change into co-production programmes. Policy and Society, 37(3), 326–346. Available at: Designing for sustainable outcomes: espousing behavioural change into co-production programmes

Van der Steen, M. and van Twist, M. (2018). Strategies for robustness: Five perspectives on how policy design is done. Policy and Society, 37(4), 491–513. Available at: Strategies for robustness: Five perspectives on how policy design is done

Government Information Quarterly

Query:

(“randomised control trial” OR “experiment” OR “systematic review” OR “return on investment” OR “cost benefit” OR “counter-factual” OR “impact” OR “evaluation”) AND (“service design” OR “social design” OR “co-design” OR “policy design” OR “design for policy” OR “public design”)

[in: Title, abstract or author-specified keywords; 2000-2023]

Hits:

67

Selected:

4

References:

Bharosa, N., Janssen, M., van Wijk, R., de Winne, N., van der Voort, H., Hulstijn, J. and Tan, Y. (2013). Tapping into existing information flows: The transformation to compliance by design in business-to-government information exchange. Government Information Quarterly, 30(S1), S9-S18. Available at:  Tapping into existing information flows: The transformation to compliance by design in business-to-government information exchange

Christensen, H. S. (2021). A conjoint experiment of how design features affect evaluations of participatory platforms. Government Information Quarterly, 38(1). Available at:  A conjoint experiment of how design features affect evaluations of participatory platforms

Lee, H., Tsohou, A. and Choi, Y. (2017). Embedding persuasive features into policy issues: Implications to designing public participation processes. Available at: Embedding persuasive features into policy issues: Implications to designing public participation processes

van der Giessen, M. and Bayerl, P. S. (2022). Designing for successful online engagement: Understanding technological frames of citizen and police users of community policing platforms. Government Information Quarterly, 39(3). Available at: Designing for successful online engagement: Understanding technological frames of citizen and police users of community policing platforms

Journal of Public Policy

Query:

(“design”)

[in: Search within full text; 2000-2023]

N.B. Cambridge Core’s search query doesn’t support nested Boolean connectors.

Hits:

68

Selected:

1

References:

Moore, R., Gan, K., Minnich, K., and Yokum, D. (2022). Anchor management: A field experiment to encourage families to meet critical programme deadlines. Journal of Public Policy, 42(4), 615-636. Available at: Anchor management: A field experiment to encourage families to meet critical programme deadlines

Behavioural Public Policy

Query:

(“design”)

[in: Search within full text; 2000-2023]

N.B. Cambridge Core’s search query doesn’t support nested Boolean connectors.

Hits:

54

Selected:

8

References:

Baggio, M., and Motterlini, M. (2022). Testing donation menus: On charitable giving for cancer research – evidence from a natural field experiment. Behavioural Public Policy, 6(3), 417-438. Available at: Testing donation menus: On charitable giving for cancer research – evidence from a natural field experiment

Bradt, J. (2022). Comparing the effects of behaviorally informed interventions on flood insurance demand: An experimental analysis of ‘boosts’ and ‘nudges’. Behavioural Public Policy, 6(3), 485-515. Available at: Comparing the effects of behaviorally informed interventions on flood insurance demand: An experimental analysis of ‘boosts’ and ‘nudges’

Brick, C., and Freeman, A. (2021). Communicating evidence in icons and summary formats for policymakers: What works? Behavioural Public Policy, 1-29. Available at: Communicating evidence in icons and summary formats for policymakers: What works?

Chen, N., Trump, K., Hall, S., and Le, Q. (2023). The effect of postcard reminders on vaccinations among the elderly: A block-randomized experiment. Behavioural Public Policy, 7(2), 240-265. Available at: The effect of postcard reminders on vaccinations among the elderly: A block-randomized experiment

Fesenfeld, L. (2022). The effects of policy design complexity on public support for climate policy. Behavioural Public Policy, 1-26. Available at: The effects of policy design complexity on public support for climate policy

Keppeler, F., Sievert, M., and Jilke, S. (2022). Increasing COVID-19 vaccination intentions: A field experiment on psychological ownership. Behavioural Public Policy, 1-20. Available at: Increasing COVID-19 vaccination intentions: A field experiment on psychological ownership

Renaud, K., and Zimmermann, V. (2019). Nudging folks towards stronger password choices: Providing certainty is the key. Behavioural Public Policy, 3(2), 228-258. Available at: Nudging folks towards stronger password choices: Providing certainty is the key

Reynolds, J., Hobson, A., Ventsel, M., Pilling, M., Marteau, T., and Hollands, G. (2022). Effect of visualising and re-expressing evidence of policy effectiveness on perceived effectiveness: A population-based survey experiment. Behavioural Public Policy, 1-19. Available at: Effect of visualising and re-expressing evidence of policy effectiveness on perceived effectiveness: A population-based survey experiment

Query:

(“randomised control trial” AND “design”)

[in: Search within full text; 2000-2023]

NB: Cambridge Core’s search query doesn’t support nested Boolean connectors.

Hits:

61

Selected:

4

References:

De-loyde, K., Pilling, M., Thornton, A., Spencer, G., and Maynard, O. (2022). Promoting sustainable diets using eco-labelling and social nudges: A randomised online experiment. Behavioural Public Policy, 1-17. Available at: Promoting sustainable diets using eco-labelling and social nudges: A randomised online experiment

Gravert, C., and Kurz, V. (2021). Nudging à la carte: A field experiment on climate-friendly food choice. Behavioural Public Policy, 5(3), 378-395. Available at: Nudging à la carte: A field experiment on climate-friendly food choice

Jachimowicz, J., Gladstone, J., Berry, D., Kirkdale, C., Thonley, T., and Galinsky, A. (2021). Making medications stick: Improving medication adherence by highlighting the personal health costs of non-compliance. Behavioural Public Policy, 5(3), 396-416. Available at: Making medications stick: Improving medication adherence by highlighting the personal health costs of non-compliance

Persian, R., Prastuti, G., Adityawarman, Bogiatzis-Gibbons, D., Kurniawan, M., Subroto, G., Gandy, K. and Sutherland, A. (2023). Behavioural prompts to increase early filing of tax returns: A population-level randomised controlled trial of 11.2 million taxpayers in Indonesia. Behavioural Public Policy, 7(3), 701-720. Available at: Behavioural prompts to increase early filing of tax returns: A population-level randomised controlled trial of 11.2 million taxpayers in Indonesia

Query:

(“experiment” AND “design”)

[in: Search within full text; 2000-2023]

N.B. Cambridge Core’s search query doesn’t support nested Boolean connectors.

Hits:

116

Selected:

7

References:

Ackfeld, V., Rohloff, T., and Rzepka, S. (2021). Increasing personal data contributions for the greater public good: A field experiment on an online education platform. Behavioural Public Policy, 1-27. Available at: Increasing personal data contributions for the greater public good: A field experiment on an online education platform

Chandra, G. (2023). Non-monetary intervention to discourage consumption of single-use plastic bags. Behavioural Public Policy, 7(1), 143-156. Available at: Non-monetary intervention to discourage consumption of single-use plastic bags

Kantorowicz-Reznichenko, E., Kantorowicz, J., and Wells, L. (2022). Can vaccination intentions against COVID-19 be nudged? Behavioural Public Policy, 1-25. Available at: Can vaccination intentions against COVID-19 be nudged?

Kölle, F., Lane, T., Nosenzo, D., and Starmer, C. (2020). Promoting voter registration: The effects of low-cost interventions on behaviour and norms. Behavioural Public Policy, 4(1), 26-49. Available at: Promoting voter registration: The effects of low-cost interventions on behaviour and norms

Mol, J., Botzen, W., Blasch, J., Kranzler, E., and Kunreuther, H. (2021). All by myself? Testing descriptive social norm-nudges to increase flood preparedness among homeowners. Behavioural Public Policy, 1-33. Available at: All by myself? Testing descriptive social norm-nudges to increase flood preparedness among homeowners

Timmons, S., Barjaková, M., McEelvaney, T., and Lunn, P. (2022). Official advice improves mortgage-holders’ perceptions of switching: Experimental evidence. Behavioural Public Policy, 6(4), 625-653. Available at: Official advice improves mortgage-holders’ perceptions of switching: Experimental evidence

Van Roekel, H., Reinhard, J., and Grimmelikhuijsen, S. (2022). Improving hand hygiene in hospitals: Comparing the effect of a nudge and a boost on protocol compliance. Behavioural Public Policy, 6(1), 52-74. Available at: Improving hand hygiene in hospitals: Comparing the effect of a nudge and a boost on protocol compliance

Policy Design and Practice

Query:

(“randomised control trial” OR “experiment” OR “systematic review” OR “return on investment” OR “cost benefit” OR “counter-factual” OR “impact” OR “evidence”) AND (“service design” OR “social design” OR “co-design” OR “policy design” OR “design for policy” OR “public design”)

[in: Title, abstract or author-specified keywords; 2000-2023]

Hits:

95

Selected:

6

References:

Deserti, A., Rizzo, F., and Smallman, M. (2020). Experimenting with co-design in STI policy making. Policy Design and Practice, 3:2, 135-149. Available at: Experimenting with co-design in STI policy making

Gouache, C. (2022). Imagining the future with citizens: participatory foresight and democratic policy design in Marcoussis, France. Policy Design and Practice, 5:1, 66-85. Available at: Imagining the future with citizens: participatory foresight and democratic policy design in Marcoussis, France

Ferrarezi, E., Brandalise, I., and Lemos, J. (2021). Evaluating experimentation in the public sector: learning from a Brazilian innovation lab. Policy Design and Practice, 4:2, 292-308. Available at: Evaluating experimentation in the public sector: learning from a Brazilian innovation lab

Katsonis, M. (2019). Designing effective public engagement: the case study of Future Melbourne 2026. Policy Design and Practice, 2:2, 215-228. Available at: Designing effective public engagement: the case study of Future Melbourne 2026

Nogueira, A., and Schmidt, R. (2022). Participatory policy design: igniting systems change through prototyping. Policy Design and Practice, 5:1, 32-50. Available at: Participatory policy design: igniting systems change through prototyping

Perez, R. E., Ng, A. C. L., and Tiglao, N. C. C. (2022). Enhancing policy capacity through Co-design: the case of local public transportation in the Philippines. Policy Design and Practice, 5:1, 103-121. Available at: Enhancing policy capacity through Co-design: the case of local public transportation in the Philippines

Policy and Politics

Query:

(“randomised control trial” OR “experiment” OR “systematic review” OR “return on investment” OR “cost benefit” OR “counter-factual” OR “impact” OR “evidence”) AND (“service design” OR “social design” OR “co-design” OR “policy design” OR “design for policy” OR “public design”)

[in: Title, abstract or author-specified keywords; 2000-2023]

Hits:

27

Selected:

3

References:

Ball, S., and Head, B. W. (2021). Behavioural insights teams in practice: nudge missions and methods on trial. Policy and Politics, 49(1), 105-120. Available at: Behavioural insights teams in practice: nudge missions and methods on trial

Cotterill, S., John, P., and Moseley, A. (2013). Does mobilisation increase family engagement with an early childhood intervention programme? A randomised controlled trial. Policy and Politics, 41(2), 201-221. Available at: Does mobilisation increase family engagement with an early childhood intervention programme? A randomised controlled trial

Hermus, M., van Buuren, A., and Bekkers, V. (2020). Applying design in public administration: A literature review to explore the state of the art. Policy and Politics, 48(1), 21-48. Applying design in public administration: a literature review to explore the state of the art

Appendix 7. Bibliography – design journals, round 1

Alipour, L., Faizi, M., Moradi, A. M., and Akrami, G. (2017). The impact of designers’ goals on design-by-analogy. Design Studies, 51, 1-24. Available at: The impact of designers’ goals on design-by-analogy

Balters, S., Weinstein, T., Mayseless, N., Auernhammer, J., Hawthorne, G., Steinert, M., Meinel, C., Leifer, L. J. and Reiss, A. L. (2023). Design science and neuroscience: A systematic review of the emergent field of design neurocognition. Design Studies, 84, 101148. Available at: Design science and neuroscience: A systematic review of the emergent field of design neurocognition

Becattini, N., Borgianni, Y., Cascini, G., and Rotini, F. (2017). Surprise and design creativity: investigating the drivers of unexpectedness. International journal of design creativity and innovation, 5(1-2), 29-47. Available at: Surprise and design creativity: investigating the drivers of unexpectedness

Belmonte, M. V., Millán, E., Ruiz-Montiel, M., Badillo, R., Boned, J., Mandow, L., and Pérez-de-la-Cruz, J. L. (2014). Randomness and control in design processes: An empirical study with architecture students. Design Studies, 35(4), 392-411. Available at: Randomness and control in design processes: An empirical study with architecture students

Brun, J., Le Masson, P., and Weil, B. (2016). Designing with sketches: The generative effects of knowledge preordering. Design Science, 2, E13. Available at: Designing with sketches: The generative effects of knowledge preordering

Cash, P., Elias, E., Dekoninck, E., and Culley, S. (2012). Methodological insights from a rigorous small scale design experiment. Design Studies, 33(2), 208-235. Available at: Methodological insights from a rigorous small scale design experiment

Cash, P., Dekoninck, E., and Ahmed-Kristensen, S. (2020). Work with the beat: How dynamic patterns in team processes affect shared understanding. Design Studies, 69, 100943. Available at: Work with the beat: How dynamic patterns in team processes affect shared understanding

Choi, Y., Na, J. H., Walters, A., Lam, B., Boult, J., Jordan, P. W., and Green, S. (2018). Design for social value: using design to improve the impact of CSR. Journal of Design Research, 16(2), 155-174. Available at: Design for social value: using design to improve the impact of CSR

Cockbill, S. A., May, A., and Mitchell, V. (2019). The assessment of meaningful outcomes from co-design: a case study from the energy sector. She Ji: The Journal of Design, Economics, and Innovation, 5(3), 188-208. Available at: The assessment of meaningful outcomes from co-design: a case study from the energy sector

Coelho, D. A., and Vieira, F. L. (2018). The effect of previous group interaction on individual ideation novelty and variety. International Journal of Design Creativity and Innovation, 6(1-2), 80-92. Available at: The effect of previous group interaction on individual ideation novelty and variety

Collado-Ruiz, D., and Ostad-Ahmad-Ghorabi, H. (2010). Influence of environmental information on creativity. Design Studies, 31(5), 479-498. Available at: Influence of environmental information on creativity

Corazzo, J. (2019). Materialising the Studio. A systematic review of the role of the material space of the studio in Art, Design and Architecture Education. The Design Journal, 22 (supplement 1), 1249-1265. A systematic review of the role of the material space of the studio in Art, Design and Architecture Education

Dahl, D. W., Chattopadhyay, A., and Gorn, G. J. (2001). The importance of visualisation in concept design. Design Studies, 22(1), 5-26. Available at: The importance of visualisation in concept design

Dong, A., Lovallo, D., and Mounarath, R. (2015). The effect of abductive reasoning on concept selection decisions. Design studies, 37, 37-58. Available at: The effect of abductive reasoning on concept selection decisions

Flager, F., Gerber, D. J., and Kallman, B. (2014). Measuring the impact of scale and coupling on solution quality for building design problems. Design Studies, 35(2), 180-199. Available at: Measuring the impact of scale and coupling on solution quality for building design problems

Gergle, D., Kraut, R. E., and Fussell, S. R. (2013). Using visual information for grounding and awareness in collaborative tasks. Human–Computer Interaction, 28(1), 1-39. Available at: Using visual information for grounding and awareness in collaborative tasks

Hay, L., Duffy, A., McTeague, C., Pidgeon, L., Vuletic, T., and Grealy, M. (2017). A systematic review of protocol studies on conceptual design cognition: Design as search and exploration. Design Science, 3. Available at: A systematic review of protocol studies on conceptual design cognition: Design as search and exploration

Hernandez, N. V., Shah, J. J., and Smith, S. M. (2010). Understanding design ideation mechanisms through multilevel aligned empirical studies. Design studies, 31(4), 382-410. Available at: Understanding design ideation mechanisms through multilevel aligned empirical studies

Kokotovich, V., and Purcell, T. (2000). Mental synthesis and creativity in design: an experimental examination. Design studies, 21(5), 437-449. Available at: Mental synthesis and creativity in design: an experimental examination

Lee, J. H., and Ostwald, M. J. (2022). The relationship between divergent thinking and ideation in the conceptual design process. Design Studies, 79, 101089. Available at: The relationship between divergent thinking and ideation in the conceptual design process

Lievesley, M., and Yee, J. (2012). Valuing Service Design: Lessons from SROI (Social Return On Investment). In: Israsena, P., Tangsantikul, J. and Durling, D. (eds.). Research: Uncertainty Contradiction Value - DRS International Conference 2012, 1-4 July, Bangkok, Thailand. Available at: Valuing Service Design: Lessons from SROI (Social Return On Investment)

Manohar, A., Smith, M., and Calvo, M. (2016). Capturing the “How”: Showing the value of codesign through creative evaluation. In: Lloyd, P. and Bohemia, E. (eds.). Future Focused Thinking - DRS International Conference 2016, 27 - 30 June, Brighton, United Kingdom. Available at: Capturing the “How”: Showing the value of codesign through creative evaluation

Masaya, S., Fujii, K., Watanabe, K., Chen, C., Hai, L., Zhang, J., Ohkubo, T., Murakoshi, H., and Hirose, Y. (2023). An exploratory study for the impact of video-induced positive moods on ideation. International Journal of Design Creativity and Innovation, 11(1), 26-41. Available at: An exploratory study for the impact of video-induced positive moods on ideation

Nguyen, M., and Mougenot, C. (2022). A systematic review of empirical studies on multidisciplinary design collaboration: Findings, methods, and challenges. Design Studies, 81, 101120. Available at: A systematic review of empirical studies on multidisciplinary design collaboration: Findings, methods, and challenges

Ou, X., Goldschmidt, G., and Erez, M. (2023). The effect of disciplinary diversity on design idea generation in dyadic teams. Design Studies, 86, 101184. Available at: The effect of disciplinary diversity on design idea generation in dyadic teams

Owen, V., Ceyhan, P., Cruickshank, L., and Christou, E. (2022). Evaluating social innovations: How creative evaluation can help articulate their values and impacts. In: Lockton, D., Lenzi, S., Hekkert, P., Oak, A., Sádaba, J., Lloyd, P. (eds.). DRS2022: Bilbao, 25 June - 3 July, Bilbao, Spain. Available at: Evaluating social innovations: How creative evaluation can help articulate their values and impacts

Price, R. A., De Lille, C., and Bergema, K. (2019). Advancing industry through design: A longitudinal case study of the aviation industry. She Ji: The Journal of Design, Economics, and Innovation, 5(4), 304-326. Available at: Advancing industry through design: A longitudinal case study of the aviation industry

Sadowska, N., and Laffy, D. (2019). Measuring the impact of strategic design learning experience long after the classroom delivery. The Design Journal, 22 (supplement 1), 1305-1315. Available at: Measuring the impact of strategic design learning experience long after the classroom delivery

Tromp, N., and Hekkert, P. (2016). Assessing methods for effect-driven design: Evaluation of a social design method. Design Studies, 43, 24-47. Available at: Assessing methods for effect-driven design: Evaluation of a social design method

van Dam, S., Sleeswijk Visser, F., and Bakker, C. (2021). The impact of co-creation on the design of circular product-service systems: Learnings from a case study with washing machines. The Design Journal, 24(1), 25-45. Available at: The impact of co-creation on the design of circular product-service systems: Learnings from a case study with washing machines

Vasantha, G. V. A., Chakrabarti, A., Rout, B. K., and Corney, J. (2014). Influences of design tools on the original and redesign processes. International Journal of Design Creativity and Innovation, 2(1), 20-50. Available at: Influences of design tools on the original and redesign processes

Wang, Z., Jiang, T., Huang, J., Tai, Y., and Trapani, P.M. (2022). How might we evaluate co-design? A literature review on existing practices. In: Lockton, D., Lenzi, S., Hekkert, P., Oak, A., Sádaba, J., Lloyd, P. (eds.). DRS2022: Bilbao, 25 June - 3 July, Bilbao, Spain. Available at: How might we evaluate co-design? A literature review on existing practices

Worinkeng, E., Joshi, S., and Summers, J. D. (2015). An experimental study: analyzing requirement type influence on novelty and variety of generated solutions. International Journal of Design Creativity and Innovation, 3(2), 61-77. Available at: An experimental study: analyzing requirement type influence on novelty and variety of generated solutions

Other references

Cash, P., Elias, E., Dekoninck, E., and Culley, S. (2012). Methodological insights from a rigorous small scale design experiment. Design Studies, 33(2), 208-235. Available at: Methodological insights from a rigorous small scale design experiment

Cash, P., Daalhuizen, J., and Hekkert, P. (2023). Evaluating the efficacy and effectiveness of design methods: A systematic review and assessment framework. Design Studies, 88, 101204. Available at: Evaluating the efficacy and effectiveness of design methods: A systematic review and assessment framework

Corremans, J. A. (2011). Measuring the effectiveness of a design method to generate form alternatives: an experiment performed with freshmen students product development. Journal of Engineering Design, 22(4), 259-274. Available at: Measuring the effectiveness of a design method to generate form alternatives: an experiment performed with freshmen students product development

Lawson, B. (1979). Cognitive Strategies in Architectural Design. Ergonomics. 22. 59-68. Available at: Cognitive Strategies in Architectural Design

Nusem, E., Wrigley, C., and Matthews, J. (2017). Developing Design Capability in Nonprofit Organizations. Design Issues 2017; 33 (1): 61–75. Available at: Developing Design Capability in Nonprofit Organizations

Tripp, H.P (2021). Policy Instrumentation: The Object of Service Design in Policy Making. Design Issues 2021; 37 (3): 89–100. Available at: Policy Instrumentation: The Object of Service Design in Policy Making

Appendix 8. Bibliography – design journals, round 2

Allen, B., Tamindael, L. E., Bickerton, S. H., and Cho, W. (2020). Does citizen coproduction lead to better urban services in smart cities projects? An empirical study on e-participation in a mobile big data platform. Government Information Quarterly, 37(1), 101412. Available at: Does citizen coproduction lead to better urban services in smart cities projects? An empirical study on e-participation in a mobile big data platform

Balakrishnan, B. (2022). Exploring the impact of design thinking tool among design undergraduates: a study on creative skills and motivation to think creatively. International Journal of Technology and Design Education, 32(3), 1799-1812. Available at: Exploring the impact of design thinking tool among design undergraduates: a study on creative skills and motivation to think creatively

Bornet, C., and Brangier, E. (2016). The effects of personas on creative codesign of work equipment: an exploratory study in a real setting. CoDesign, 12(4), 243-256. Available at: The effects of personas on creative codesign of work equipment: an exploratory study in a real setting

Conradie, P. D., Van Acker, B. B., De Vos, E., and Saldien, J. (2021). Impact of user involvement on design students’ motivation and self-confidence. International Journal of Technology and Design Education, 31, 183-197. Available at: Impact of user involvement on design students’ motivation and self-confidence

Corcoran, R., Marshall, G., and Walsh, E. (2018). The psychological benefits of cooperative place-making: a mixed methods analyses of co-design workshops. CoDesign, 14(4), 314-328. Available at: The psychological benefits of cooperative place-making: a mixed methods analyses of co-design workshops

Fraser, K. (2009). Labour flexibility: impact of functional and localised strategies on team-based product manufacturing. CoDesign, 5(3), 143-158. Available at: Labour flexibility: impact of functional and localised strategies on team-based product manufacturing

Guaman-Quintanilla, S., Everaert, P., Chiluiza, K., and Valcke, M. (2023). Impact of design thinking in higher education: a multi-actor perspective on problem solving and creativity. International Journal of Technology and Design Education, 33(1), 217-240. Available at: Impact of design thinking in higher education: a multi-actor perspective on problem solving and creativity

Huffman, T. J., and Mentzer, N. (2021). The impact of modelling-eliciting activities on high school student design performance. International Journal of Technology and Design Education, 31, 255-280. Available at: The impact of modelling-eliciting activities on high school student design performance

Mitchell, V., Ross, T., May, A., Sims, R., and Parker, C. (2016). Empirical investigation of the impact of using co-design methods when generating proposals for sustainable travel solutions. CoDesign, 12(4), 205-220. Available at: Empirical investigation of the impact of using co-design methods when generating proposals for sustainable travel solutions

Perttula, M. K., Krause, C. M., and Sipilä, P. (2006). Does idea exchange promote productivity in design idea generation? CoDesign, 2(3), 125-138. Available at: Does idea exchange promote productivity in design idea generation?

Steen, M., Arendsen, J., Cremers, A., De Jong, A., De Jong, J., and De Koning, N. (2013). Using interactive model simulations in co-design: An experiment in urban design. CoDesign, 9(1), 2-16. Available at: Using interactive model simulations in co-design: An experiment in urban design

Sun, L., Xiang, W., Chai, C., Wang, C., and Liu, Z. (2013). Impact of text on idea generation: an electroencephalography study. International Journal of Technology and Design Education, 23, 1047-1062. Available at: Impact of text on idea generation: an electroencephalography study

Tang, C., Zhang, Y., and Reiter‐Palmon, R. (2020). Network centrality, knowledge searching and creativity: The role of domain. Creativity and Innovation Management, 29(1), 72-84. Available at: Network centrality, knowledge searching and creativity: The role of domain

Taoka, Y., Kagohashi, K., and Mougenot, C. (2018). A cross-cultural study of co-design: the impact of power distance on group dynamics in Japan. CoDesign. Available at: A cross-cultural study of co-design: the impact of power distance on group dynamics in Japan

Van Rijn, H., Sleeswijk Visser, F., Stappers, P. J., and Özakar, A. D. (2011). Achieving empathy with users: the effects of different sources of information. CoDesign, 7(2), 65-77. Available at: Achieving empathy with users: the effects of different sources of information

Villalba, C., Jaiprakash, A., Donovan, J., Roberts, J., and Crawford, R. (2019). Testing literature-based health experience insight cards in a healthcare service co-design workshop. CoDesign. Available at: Testing literature-based health experience insight cards in a healthcare service co-design workshop

Appendix 9. Bibliography – political science journals, round 1

Baxter, D., Dacre, N., Dong, H., and Ceylan, S. (2023). Institutional challenges in agile adoption: Evidence from a public sector IT project. Government Information Quarterly, 101858. Available at: Institutional challenges in agile adoption: Evidence from a public sector IT project

Bellé, N. (2014). Leading to Make a Difference: A Field Experiment on the Performance Effects of Transformational Leadership, Perceived Social Impact, and Public Service Motivation. Journal of Public Administration Research and Theory, 24(1), 109–136. Available at: Leading to Make a Difference: A Field Experiment on the Performance Effects of Transformational Leadership, Perceived Social Impact, and Public Service Motivation

Cole, L. and Hagen, P. (2023). Scaling deep through transformative learning in public sector innovation labs – experiences from Vancouver and Auckland. Public Management Review, 26(7), 2094-2121. Available at: Scaling deep through transformative learning in public sector innovation labs – experiences from Vancouver and Auckland

Criado, J.I., Alcaide-Muñoz, L. and Liarte, I. (2023). Two decades of public sector innovation: building an analytical framework from a systematic literature review of types, strategies, conditions, and results. Public Management Review, 27(3), 623-652. Available at: Two decades of public sector innovation: building an analytical framework from a systematic literature review of types, strategies, conditions, and results

Dudau, A., Glennon, R. and Verschuere, B. (2019). Following the yellow brick road? (Dis)enchantment with co-design, co-production and value co-creation in public services. Public Management Review, 21(11), 1577–1594. Available at: Following the yellow brick road? (Dis)enchantment with co-design, co-production and value co-creation in public services

Fledderus, J. (2015). Does User Co-Production of Public Service Delivery Increase Satisfaction and Trust? Evidence From a Vignette Experiment. International Journal of Public Administration, 38(9), 642–653. Available at: Does User Co-Production of Public Service Delivery Increase Satisfaction and Trust? Evidence From a Vignette Experiment

Jakobsen, M., Jacobsen, C.B. and Serritzlew, S. (2019). Managing the Behavior of Public Frontline Employees through Change-Oriented Training: Evidence from a Randomized Field Experiment. Journal of Public Administration Research and Theory, 29(4), 556–571. Available at: Managing the Behavior of Public Frontline Employees through Change-Oriented Training: Evidence from a Randomized Field Experiment

Kang, S. and Van Ryzin, G.G. (2019). Coproduction and trust in government: evidence from survey experiments. Public Management Review, 21(11), 1646–1664. Available at: Coproduction and trust in government: evidence from survey experiments

Kleinhans, R. (2017). False promises of co-production in neighbourhood regeneration: the case of Dutch community enterprises. Public Management Review, 19(10), 1500–1518. Available at: False promises of co-production in neighbourhood regeneration: the case of Dutch community enterprises

Liu, L.C. (2022). Policy innovation and public organizational performance: a moderated mediation analysis of Taiwan’s Bookstart program. Public Management Review, 24(11), 1824–1845. Available at: Policy innovation and public organizational performance: a moderated mediation analysis of Taiwan’s Bookstart program

Loeffler, E. and Bovaird, T. (2016). User and Community Co-Production of Public Services: What Does the Evidence Tell Us? International Journal of Public Administration, 39(13), 1006–1019. Available at: User and Community Co-Production of Public Services: What Does the Evidence Tell Us?

Mergel, I., Gong, Y. and Bertot, J. (2018). Agile government: Systematic literature review and future research. Government Information Quarterly, 35(2), 291–298. Available at: Agile government: Systematic literature review and future research

Sicilia, M., Sancio, A., Nabatchi, T., and Guarini, E. (2019). Facilitating co-production in public services: management implications from a systematic literature review. Public Money and Management, 39(4), 233–240. Available at: Facilitating co-production in public services: management implications from a systematic literature review

Soe, R.-M. and Drechsler, W. (2018). Agile local governments: Experimentation before implementation. Government Information Quarterly, 35(2), 323–335. Available at: Agile local governments: Experimentation before implementation

Thøgersen, D., Waldorff, S.B. and Steffensen, T. (2021). Public Value through Innovation: Danish Public Managers’ Views on Barriers and Boosters. International Journal of Public Administration, 44(14), 1264–1273. Available at: Public Value through Innovation: Danish Public Managers’ Views on Barriers and Boosters

Torfing, J., Cristofoli, D., Gloor, P. A., Meijer, A. J., and Trivellato, B. (2020). Taming the snake in paradise: combining institutional design and leadership to enhance collaborative innovation. Policy and Society, 39(4), 592–616. Available at: Taming the snake in paradise: combining institutional design and leadership to enhance collaborative innovation

Whicher, A. and Crick, T. (2019). Co-design, evaluation and the Northern Ireland Innovation Lab. Public Money and Management, 39(4), 290–299. Available at: Co-design, evaluation and the Northern Ireland Innovation Lab

Appendix 10. Bibliography – political science journals, round 2

Quantitative evidence (RCTs and field/quasi/natural experiments)

Ackfeld, V., Rohloff, T., and Rzepka, S. (2021). Increasing personal data contributions for the greater public good: A field experiment on an online education platform. Behavioural Public Policy, 8(3), 470-496. Available at: Increasing personal data contributions for the greater public good: A field experiment on an online education platform

Baggio, M., and Motterlini, M. (2022). Testing donation menus: On charitable giving for cancer research – evidence from a natural field experiment. Behavioural Public Policy, 6(3), 417-438. Available at: Testing donation menus: On charitable giving for cancer research – evidence from a natural field experiment

Bradt, J. (2022). Comparing the effects of behaviorally informed interventions on flood insurance demand: An experimental analysis of ‘boosts’ and ‘nudges’. Behavioural Public Policy, 6(3), 485-515. Available at: Comparing the effects of behaviorally informed interventions on flood insurance demand: An experimental analysis of ‘boosts’ and ‘nudges’

Brick, C., and Freeman, A. (2021). Communicating evidence in icons and summary formats for policymakers: What works? Behavioural Public Policy, 8(3),441-469. Available at: Communicating evidence in icons and summary formats for policymakers: What works?

Chan, F.K.Y., Thong, J.Y.L., Brown, S.A. and Venkatesh, V. (2021). Service Design and Citizen Satisfaction with E-Government Services: A Multidimensional Perspective. Public Administration Review, 81(5), 874-894. Available at: Service Design and Citizen Satisfaction with E-Government Services: A Multidimensional Perspective

Chandra, G. (2023). Non-monetary intervention to discourage consumption of single-use plastic bags. Behavioural Public Policy, 7(1), 143-156. Available at: Non-monetary intervention to discourage consumption of single-use plastic bags

Chen, N., Trump, K., Hall, S. and Le, Q. (2023). The effect of postcard reminders on vaccinations among the elderly: A block-randomized experiment. Behavioural Public Policy, 7(2), 240-265. Available at: The effect of postcard reminders on vaccinations among the elderly: A block-randomized experiment

Christensen, H.S. (2021). A conjoint experiment of how design features affect evaluations of participatory platforms. Government Information Quarterly, 38(1), 101538. Available at: A conjoint experiment of how design features affect evaluations of participatory platforms

Cotterill, S., John, P., and Moseley, A. (2013). Does mobilisation increase family engagement with an early childhood intervention programme? A randomised controlled trial. Policy and Politics, 41(2), 201-221. Available at: Does mobilisation increase family engagement with an early childhood intervention programme? A randomised controlled trial

De-loyde, K., Pilling, M., Thornton, A., Spencer, G., and Maynard, O. (2022). Promoting sustainable diets using eco-labelling and social nudges: A randomised online experiment’. Behavioural Public Policy, 1-17. Available at: Promoting sustainable diets using eco-labelling and social nudges: A randomised online experiment

Dolšak, N, Adolph, C, Prakash, A. (2020). Policy design and public support for carbon tax: Evidence from a 2018 US national online survey experiment. Public Administration. 98(4), 905–921. Available at: Policy design and public support for carbon tax: Evidence from a 2018 US national online survey experiment

Fesenfeld, L.P. (2022). The effects of policy design complexity on public support for climate policy. Behavioural Public Policy, 1-26. Available at: The effects of policy design complexity on public support for climate policy

Gravert, C., and Kurz, V. (2021). Nudging à la carte: A field experiment on climate-friendly food choice. Behavioural Public Policy, 5(3), 378-395. Available at: Nudging à la carte: A field experiment on climate-friendly food choice

Jachimowicz, J., Gladstone, J., Berry, D., Kirkdale, C., Thornley, T., and Galinksy, A. (2021). Making medications stick: Improving medication adherence by highlighting the personal health costs of non-compliance. Behavioural Public Policy, 5(3), 396-416. Available at: Making medications stick: Improving medication adherence by highlighting the personal health costs of non-compliance

Kantorowicz-Reznichenko, E., Kantorowicz, J., and Wells, L. (2022). Can vaccination intentions against COVID-19 be nudged? Behavioural Public Policy, 1-25. Available at: Can vaccination intentions against COVID-19 be nudged?

Keppeler, F., Sievert, M., and Jilke, S. (2022). Increasing COVID-19 vaccination intentions: A field experiment on psychological ownership. Behavioural Public Policy, 9(1) 16-35. Available at: Increasing COVID-19 vaccination intentions: A field experiment on psychological ownership

Kolle, F., Lane, T., Nosenzo, D. and Starmer, C. (2020). Promoting voter registration: The effects of low-cost interventions on behaviour and norms. Behavioural Public Policy, 4(1), 26-49. Available at: Promoting voter registration: The effects of low-cost interventions on behaviour and norms

Lee, H., Tsohou, A., and Choi, Y. (2017). Embedding persuasive features into policy issues: Implications to designing public participation processes. Government Information Quarterly, 34(4), 591-600. Available at: Embedding persuasive features into policy issues: Implications to designing public participation processes

Mol, J., Botzen, W., Blasch, J., Kranzler, E., and Kunreuther, H. (2021). All by myself? Testing descriptive social norm-nudges to increase flood preparedness among homeowners. Behavioural Public Policy, 1-33. Available at: All by myself? Testing descriptive social norm-nudges to increase flood preparedness among homeowners

Moore, R., Gan, K., Minnich, K., and Yokum, D. (2022). Anchor management: A field experiment to encourage families to meet critical programme deadlines. Journal of Public Policy, 42(4), 615-636. Available at: Anchor management: A field experiment to encourage families to meet critical programme deadlines

Persian, R., Prastuti, G., Adityawarman, Bogiatzis-Gibbons, D., Kurniawan, M., Subroto, G., Gandy, K. and Sutherland, A. (2023). Behavioural prompts to increase early filing of tax returns: A population-level randomised controlled trial of 11.2 million taxpayers in Indonesia. Behavioural Public Policy, 7(3), 701-720. Available at: Behavioural prompts to increase early filing of tax returns: A population-level randomised controlled trial of 11.2 million taxpayers in Indonesia

Renaud, K., and Zimmermann, V. (2019). Nudging folks towards stronger password choices: Providing certainty is the key. Behavioural Public Policy, 3(2), 228-258. Available at: Nudging folks towards stronger password choices: Providing certainty is the key

Reynolds, J., Hobson, A., Ventsel, M., Pilling, M., Marteau, T., and Hollands, G. (2022). Effect of visualising and re-expressing evidence of policy effectiveness on perceived effectiveness: A population-based survey experiment. Behavioural Public Policy, Advance Online Publication, 1-19. Available at: Effect of visualising and re-expressing evidence of policy effectiveness on perceived effectiveness: A population-based survey experiment

Timmons, S., Barjaková, M., McEelvaney, T., and Lunn, P. (2022). Official advice improves mortgage-holders’ perceptions of switching: Experimental evidence. Behavioural Public Policy, 6(4), 625-653. Available at: Official advice improves mortgage-holders’ perceptions of switching: Experimental evidence

Van Roekel, H., Reinhard, J., and Grimmelikhuijsen, S. (2022). Improving hand hygiene in hospitals: comparing the effect of a nudge and a boost on protocol compliance. Behavioural Public Policy, 6(1), 52-74. Available at: Improving hand hygiene in hospitals: comparing the effect of a nudge and a boost on protocol compliance

Qualitative evidence (case studies, interviews, expert surveys)

Ball, S., and Head, B. W. (2021). Behavioural insights teams in practice: nudge missions and methods on trial. Policy and Politics, 49(1), 105-120. Available at: Behavioural insights teams in practice: nudge missions and methods on trial

Bharosa, N., Janssen, M., van Wijk, R., de Winne, N., van der Voort, H., Hulstijn, J., and Tan, Y. (2013). Tapping into existing information flows: The transformation to compliance by design in business-to-government information exchange. Government Information Quarterly, 30(S1), S9-S18. Available at: Tapping into existing information flows: The transformation to compliance by design in business-to-government information exchange

Bason, C., and Austin, R. D. (2022). Design in the public sector: Toward a human centred model of public governance. Public Management Review, 24(11), 1727-1757. Available at: Design in the public sector: Toward a human centred model of public governance

Bloemen, P., Van Der Steen, M. and Van Der Wal, Z. (2019). Designing a century ahead: climate change adaptation in the Dutch Delta. Policy and Society, 38(1), 58–76. Available at: Designing a century ahead: climate change adaptation in the Dutch Delta

Capano, G., and Pavan, E. (2019). Designing anticipatory policies through the use of ICTs. Policy and Society, 38(1), 96–117. Available at: Designing anticipatory policies through the use of ICTs

Criado, I., Dias, T.F., Sano, H., Rojas-Martín, F., Silvan, A., and Filho, A.I. (2021). Public Innovation and Living Labs in Action: A Comparative Analysis in post-New Public Management Contexts. International Journal of Public Administration, 44(6): 451-464. Available at: Public Innovation and Living Labs in Action: A Comparative Analysis in post-New Public Management Contexts

Deserti, A., Rizzo, F. and Smallman, M. (2020). Experimenting with co-design in STI policy making. Policy Design and Practice, 3(2), 135-149. Available at: Experimenting with co-design in STI policy making

Ferrarezi, E., Brandalise, I. and Lemos, J. (2021). Evaluating experimentation in the public sector: learning from a Brazilian innovation lab. Policy Design and Practice, 4(2), 292-308. Available at: Evaluating experimentation in the public sector: learning from a Brazilian innovation lab

French, T. and Raman, S. (2021). Engaging People with Lived Experience in Co-design of Future Palliative Care Services. International Journal of Public Administration, 44(9), 778-789. Available at: Engaging People with Lived Experience in Co-design of Future Palliative Care Services

Gofen, A., Meza, O. and Moreno-Jaimes, C. (2023). Frontline organizations as experimental settings for policy change: why public management matters even more. Public Management Review. 26(7), 1827–1848. Available at: Frontline organizations as experimental settings for policy change: why public management matters even more

Gouache, C. (2022). Imagining the future with citizens: participatory foresight and democratic policy design in Marcoussis, France. Policy Design and Practice, 5(1), 66-85. Available at: Imagining the future with citizens: participatory foresight and democratic policy design in Marcoussis

Katsonis, M. (2019). Designing effective public engagement: the case study of Future Melbourne 2026. Policy Design and Practice, 2(2), 215-228. Available at: Designing effective public engagement: the case study of Future Melbourne 2026

Koski, C. and Siddiki, S. (2022). Linking policy design, change, and outputs: Policy responsiveness in American state electricity policy. Policy Studies Journal, 50(3), 553–574. Available at: Linking policy design, change, and outputs: Policy responsiveness in American state electricity policy

Leite, H. and Hodgkinson, I. R. (2023). Examining resilience across a service ecosystem under crisis. Public Management Review, 25(4), 690-709. Available at: Examining resilience across a service ecosystem under crisis

Montpetit, É. (2008). Policy design for legitimacy: expert knowledge, citizens, time and inclusion in the United Kingdom’s biotechnology sector. Public Administration, 86(1): 259-277. Available at: Policy design for legitimacy: expert knowledge, citizens, time and inclusion in the United Kingdom’s biotechnology sector

Mukherjee, I. and Mukherjee, N. (2018). Designing for sustainable outcomes: espousing behavioural change into co-production programmes. Policy and Society, 37(3), 326-346. Available at: Designing for sustainable outcomes: espousing behavioural change into co-production programmes

Nogueira, A. and Schmidt, R. (2022). Participatory policy design: igniting systems change through prototyping. Policy Design and Practice, 5(1), 32-50. Available at: Participatory policy design: igniting systems change through prototyping

Perez, R.E., Ng A.C.L. and Tiglao, N.C.C. (2022). Enhancing policy capacity through Co-design: the case of local public transportation in the Philippines. Policy Design and Practice, 5(1), 103-121. Available at: Enhancing policy capacity through Co-design: the case of local public transportation in the Philippines

van der Giessen, M. and Bayerl, P. S. (2022). Designing for successful online engagement: Understanding technological frames of citizen and police users of community policing platforms. Government Information Quarterly, 39(3). Available at: Designing for successful online engagement: Understanding technological frames of citizen and police users of community policing platforms

Reviews of evidence / Expert surveys

Hermus, M., van Buuren, A., and Bekkers, V. (2020). Applying design in public administration: a literature review to explore the state of the art. Policy and Politics, 48(1), 21-48. Available at: Applying design in public administration: a literature review to explore the state of the art

Van der Steen, M., and van Twist, M. (2018). Strategies for robustness: five perspectives on how policy design is done. Policy and Society, 37(4), 491–513. Available at: Strategies for robustness: five perspectives on how policy design is done

1. Public Design Evidence Review: Literature Review Paper 1 - Public Design (HTML) ↩

2. Public Design Evidence Review: Literature Review Paper 2 - Public Value (HTML) ↩