How universities and funders can support monitoring, evaluation and learning for policy engagement

Why support monitoring and evaluation of policy engagement?

Monitoring and evaluating policy engagement can enrich research and increase the possibility of impact. Institutions may be motivated to support the monitoring and evaluation of policy engagement by the growing emphasis on impact assessment, including in the context of the Research Excellence Framework (REF) and funders. Employing effective monitoring and evaluation may even be key to ensuring research integrity, a pressing challenge facing universities.

Our research reveals four principal benefits of monitoring, evaluation and learning activities for policy engagement for universities:

Enhancing engagement, enriching research

Engaging with policy can be beneficial to research. The changing landscape of public policy can reveal new research opportunities and spark fresh ideas for investigation. Policymakers may also be able to offer access to otherwise unavailable data and networks in the field.

For researchers engaging with policy, monitoring and evaluation creates a space for reflection on how to maximise engagement and research outcomes. The process of monitoring policy engagement also generates relevant material and evidence that helps to tell the story of impact for funders and REF.

As a result, monitoring, evaluation and learning enriches the educational value of others’ experience and so should be valued by any learning institution. For example, the sharing of policy engagement case studies helps to strengthen practice through ‘a culture of knowledge exchange’.

Enhancing REF Impact

For REF 2014, among the 6,679 impact case studies submitted, the most common category of impact was public policy and parliamentary debate. In just under half of all case studies, the word ‘policy’ occurred in the description of impact. More than half of Oxford’s impact case studies involved policy engagement.

Although influencing public policy is one of the primary impacts documented in REF case studies, it is notoriously difficult to measure and evidence. Gathering evidence of engagement along the way can make all the difference when it comes to measuring this kind of impact and may also prove to be useful in forming future REF impact case studies.

A monitoring and learning tool, such as a research diary, can enable researchers to record and reflect on events, and also use this data at a later date. Although some policymakers may not want to acknowledge engagement for one reason or another, collating this information along the way greatly reduces the need to retrospectively gather evidence (which may no longer be readily available) for a case study.

Ongoing learning enabled by monitoring and evaluation during a project also provides opportunities to improve policy engagement skills and activities. This tends to result in higher quality impact, something which may be appealing to universities in the context of calls for more boldness in impact for REF.

Strengthening the case for research funding

Thoughtful monitoring, evaluation and learning plans are increasingly important when it comes to securing funding, especially in relation to schemes focused on impact and co-design with external partners. The following are examples of schemes and funds with a greater focus on policy engagement, and associated monitoring and evaluation:

• Global Challenges Research Fund (GCRF);
• UKRI Future Leaders Fellowships;
• Government department funds, such as the National Institute for Health Research Public Health Research Programme, and the Arts and Humanities Research Council (AHRC) and Economic and Social Research Council – Foreign & Commonwealth Office Knowledge Exchange Fellowship scheme; and
• Horizon 2020 (and upcoming Horizon Europe).

Multi-million programmatic grants, such as the GCRF Interdisciplinary Research Hubs (2016), required an extensive theory of change, as well as detailed plans for monitoring and evaluation. Some research teams need additional support and expertise to develop these. Divisions and departments that set aside time and attract such support are likely to enjoy a greater success rate.

Not only do plans for tracking engagement enable researchers to win grants in the first instance, recording impact activities and successes during a project can lead to more research funding in the future.

Supporting researcher career trajectories and equality

Monitoring and evaluating policy engagement enables researchers not only to enhance the engagement itself, but also to provide more compelling evidence of their impact and breadth of their policy networks – which may be useful for promotion purposes.

Some universities reward and recognise policy engagement more than others. In those that do so more, impactful engagement, such as might be reflected in a REF case study, may result in promotion.

In common, no doubt, with other universities, Oxford has what one interviewee referred to as ‘policy engagement superstars’. In this context, it should be ensured that all researchers with policy impact potential are supported so they also feel confident to engage if they are approached by policymakers, for example. Monitoring and evaluation can be used to gauge what approaches are working for those less familiar with the process and therefore may help to further instil confidence in researchers.

How can institutions maximise these benefits?

There are a wide range of existing and highly relevant monitoring and evaluation resources that align with current research services infrastructure at the University of Oxford, a set-up which is likely also applicable to many institutions. Given the potential added value of monitoring, evaluation and learning activities on policy engagement, it makes sense to further incorporate available resources into the toolboxes of research facilitators.

Existing support infrastructure for researchers to undertake monitoring and evaluation for policy engagement includes:

• The Policy Engagement Team;
• Research evaluation and impact teams at departmental, divisional and central levels; and
• Other research facilitation teams at departmental, divisional and central levels.

How can support structures better encourage monitoring, evaluation and learning for policy engagement?

Support structures, resources and experts available at the University can advise on monitoring, evaluation and learning for policy engagement, and increase access to, and use of, the tools and guidance provided in this series of notes.

To do this more efficiently and effectively, it is first important to understand underlying motivations and challenges faced by individual researchers in this area and how practical support mechanisms can aim to usefully interact with these. In conversations with researchers across different disciplines and career stages the following, common motivations were identified.

Why might researchers undertake policy engagement and associated learning?

• Intrinsic motivation: potential for policy engagement to enhance research, and benefit of monitoring and evaluation for improving impact and making a change in the world
• Extrinsic motivation: REF impact case study evidence, funder requirements, institution placing value on activities through promotion structures

Why might researchers disengage with policy engagement and associated learning?

• Limited capacity or resource
• No access to relevant/discipline-specific resources or guidance
• Perceived or actual complexity of policy and engagement
• Limited understanding of the benefits of monitoring and evaluation for policy engagement
• Negative associations of monitoring and evaluation associated with REF or funder ‘tick-box’ exercises rather than learning
• Discouraged from pursuing policy engagement and associated monitoring and evaluation early in career (need to focus on publications, teaching)

There are three overarching areas of support that can be provided to academics who are monitoring and evaluating their policy engagement. This support can be designed and delivered in a way that balances generic guidance with more tailored advice, and that takes into account intrinsic and extrinsic motivations of researchers.

Provide bespoke learning events, advice and mentoring

This could take the form of case studies, or sharing internal expertise and common approaches that have led to policy impact.

Learning events and networks: The University of Oxford is home to numerous colleagues who are highly experienced in engaging with policymakers in the UK and across the globe. The Oxford Policy Engagement Network (OPEN) is a growing community of doctoral students, researchers and professional services staff, and as such a valuable means to connect those interested with the support they need. Utilising the great expertise of academics at Oxford and other HE institutions, and sharing experiences of policy engagement is a great learning process.

The Public Engagement with Research (PER) and the Research Evaluation, Impact and REF Teams at Oxford provide a suite of resources and events for monitoring and evaluation, and they emphasise that it is the range of learning opportunities on offer that makes it a success. Sharing learning and best practice between the Policy Engagement and PER Teams could further enrich these internal resources.

This support could be provided at Oxford by: the Policy Engagement Team or as part of events delivered by the Divisional Research Support Teams, including:

• Introduction to monitoring, evaluation and learning approaches for policy engagement;
• Policy Engagement/monitoring and evaluation drop-in sessions (e.g., a breakfast club); or
• Half-day workshops about a specific monitoring framework for an engagement activity.

Mentoring: Researchers reported that mentoring opportunities with colleagues who are experienced in public policy engagement and associated monitoring, evaluation and learning would be welcome. Linked to this, it is worth noting that organisations with champions for approaches to monitoring policy engagement are generally more successful in progressing these activities than those without them.

Mentoring on this aspect of engagement would be particularly useful for those new to policy engagement in general, or less experienced. Complemented by training and resources, such as these notes, mentoring could help save individuals having to learn this through costly trial and error.

Framing of monitoring and evaluation to maximise researcher engagement

Our research suggests that providing networking opportunities and space for peer reflection and exchange can be more effective than formal training.

We found that researchers often associate monitoring and evaluation with additional workload and retrospective reporting. Framing it as an opportunity for continuous learning and improvement – tapping into the intrinsic motivation of researchers to make a difference – is key.

You are likely to come across jargon when exploring monitoring and evaluation. Avoid acronyms and overly technical terms when providing guidance on this topic.

Resources on monitoring techniques and frameworks should be shared with an offer of bespoke follow-up support and discussion.

Why is monitoring and evaluating my policy engagements important? Why should I (a busy academic) spend precious time on this? Communicating some or all of the four principal benefits listed in this document may help facilitate discussions.

This could be facilitated at Oxford by the Policy Engagement Team, in consultation with the People and Organisational Development team, perhaps by matching OPEN members who indicate interest in becoming a mentee with more experienced members who have both an interest in mentoring and the capacity to take on such a role. Discrete professional development may be required to maximise the potential benefit of such a scheme

Share a range of guidance, resources and tools

Resources such as these could be used by research facilitators when helping design projects across different disciplines/funders, and supporting knowledge exchange activities and evidence collation for REF:

• Designing projects and applying for funding: Annex 1 (below) is a standalone resource that can be used by research facilitators supporting academics in the design phase of projects. The document lays out practical ways monitoring, evaluation and learning for policy engagement which can be planned for and appropriately resourced in a research project.
• Evidence gathering: Research facilitators and others can direct academics to chapters in Guidance Note 2 for approaches that support the collection of evidence for policy impact: ‘Thinking about relationships power and politics’ and ‘Tracking outcomes and impacts’. For example, keeping a research diary and maintaining an up-to-date stakeholder map are effective and user-friendly tools for researchers to collect policy evidence as they go.
• Sense-making: One of the most challenging aspects for researchers learning as they go is protecting the time needed to do this at appropriate moments. In Guidance Note 2, ‘Making space for learning and reflection’, we offer a list of reflective questions that can help. There are also other tools for different engagement needs that can be recommended to researchers. For example, contribution analysis breaks down how policy outcomes have been influenced by different factors and engagements.

How can institutions sustain a supportive environment for monitoring, learning and evaluation of policy engagement?

Increased interest in policy engagement is likely to see an increase in the need for support for monitoring and evaluation of these activities. Most academic institutions that devote resources to supporting policy engagement will want to understand the effects of this provision.

REF provides a universal assessment of research impact in UK Higher Education. For example, one of their findings is that impact can be more likely if researchers work across disciplinary boundaries. This is something that can be incentivised by appropriate institutional support structures.

So far this guidance note has focused on the contribution of individual researchers and support teams, often at departmental levels. The value of these efforts, however, depends on – and can be significantly increased by – support at the level of institution, and in the wider sector. Universities, funders and networks, such as the Universities Policy Engagement Network (UPEN), are in a position to make strategic choices that can better foster and sustain an environment in which these activities are supported – thereby maximising the return on investments.

The table below summarises some of the ways in which institutions and networks can contribute to and sustain a supportive monitoring, evaluation and learning environment.

Oxford and other UPEN members are looking to understand how best to incentivise and provide support for policy engagement activities. The more they are prepared to share their learning, the more they stand to reap benefits for research and public policy. This applies to approaches to monitoring, evaluation and learning, as much as engagement itself. UPEN provides an invaluable platform and opportunity to consider what strategic experimentation could look like. The following questions may be helpful to inform reflections on these issues:

• To what degree are we pooling our experience, expertise and evidence relating to monitoring and evaluation for policy engagement?
• How far are we creating space for collective experimentation and learning for policy engagement, and monitoring and evaluation techniques?
• Are we using different frameworks or tools to monitor and evaluate our policy engagement? If so, why? What the risks and opportunities of greater standardisation?

Policy engagement and monitoring and evaluation – a reflective guide for Universities

The following questions are intended to assist academic institutions, at central, department and school levels, to reflect on the current picture of support for policy engagement and associated monitoring and evaluation. They have been inspired by the popular EDGE tool, developed by the National Co-ordinating Centre for Public Engagement to help institutions assess how well their support for public engagement is working. We have not found a similar tool for policy engagement, but the EDGE tool could provide a useful starting point for developing one. This resource has been used by institutions, including departments at the University of Oxford, to inform a strategic approach to public engagement with research.

Levels of interest in the monitoring and evaluation of policy engagement vary across and between disciplines and higher education institutions, our questions provide a starting point for further conversations within and between institutions. Depending on how these go, a full assessment tool could perhaps be developed at a later stage.

• Do we have a shared understanding of the purpose and value of policy engagement, and of how associated monitoring and evaluation activities add further value to this activity? How widely is it shared with others?
• Who are our most experienced researchers and facilitators when it comes to monitoring and evaluation for policy engagement? Do we enable them to maximise their contribution?
• Who are our champions for monitoring and evaluation of policy engagement?
• Do we frame monitoring and evaluation of policy engagement as an opportunity for continuous learning and improvement, as well as being useful for reporting?
• Do we communicate the benefits of monitoring and evaluation of policy engagement consistently across our institution? Do we make time for discussion, and offer bespoke support?
• How do we encourage staff to develop their understanding of monitoring and evaluation of policy engagement, or related skills? Do these activities feed into workload plans and performance reviews?
• Do we provide the opportunity for researchers and support staff across all disciplines and career stages to contribute to or lead monitoring and evaluation of policy engagement formally and informally?

Annex 1: Monitoring and evaluation for policy engagement: applying for funding

Relevance: This resource is for research projects that plan on engaging with public policy. Every funder has different requirements regarding monitoring, evaluation and learning. The following content is generalised and the specific guidance notes of each scheme should be considered alongside these recommendations. Some schemes may not require a formal monitoring and evaluation plan at the application phase but it is strongly recommended that this it may increase the likelihood of proposals being accepted and potentially improve the chances of ultimate impact from policy-engaged research.

Include specific reference to policy engagement and uptake as part of your overall monitoring framework

If a funder is asking how the research team will monitor and evaluate activities in general, include monitoring and evaluation for policy engagement in your response. This demonstrates understanding of the complexity of policy impact, and will support you and any others in your team in more bespoke and effective methods of monitoring, evaluation and learning in this part of the project. For example:

For the policy engagement impact pathway, a set of bespoke activities will take place within the overarching monitoring and evaluation framework. This will enable the team to effectively demonstrate how the research is influencing public policy –which is especially challenging to monitor and evidence (Bornmann, 2013).

The main tools we will use to monitor, evaluate and learn whether the project is achieving its policy impact aims include: (1) a research diary, managed by the engagement-focused research fellow, recording and reflecting on interactions with policymakers and the impact process, and (2) a policy-focused advisory board, which will meet on a six-monthly basis to reflect on progress for this impact pathway.

Policy engagement will also be monitored as part of the overarching framework, and will be a standing agenda item in project review meetings. More guidance is available on suitable monitoring and evaluation tools for policy engagement activities in Guidance Note 2.

Include sufficient resources for monitoring and evaluating policy engagement

Engaging with policy can be unpredictable, and the associated monitoring of this process can be time-intensive as a result. Successful projects have built in appropriate infrastructure in the projects to support principal and co-investigators in undertaking this activity. In some contexts it is even recommended that 5-10% of the total budget is spent on monitoring and evaluation activities. Although this may not always be possible in the context of research funding and eligibility criteria, the following resources could be incorporated:

• A project manager, postdoctoral research assistant or communications manager to support policy engagement activities and associated monitoring and evaluation. For example, time could be allocated to one of these posts to help keep a policy engagement diary, organise events, and lead on learning and reflection activities. An experienced communications manager could support the writing of policy briefs, or track networks and stakeholder engagement.
• A PhD Student whose contribution could include studying the research-policy interface in the project, supporting the mapping of pathways to impact. This student could work closely with policymakers on a distinct portfolio of work as part of their research, or explore funded placement opportunities (for example with the Parliamentary Office for Science and Technology).
• A small amount of additional time for core research team members to account for monitoring and evaluation activities, as well as policy engagement. Consider if it makes sense for one team member who is experienced in policy work to focus on this.
• Administrative staff to assist with knowledge exchange events and evidence collection.
• Meeting, travel and subsistence costs that might be required, for example hosting team or advisory committee review meetings for continuous learning.

Utilise institutional networks and knowledge

Researchers do not need to reinvent the wheel every time they design a monitoring and evaluation plan for a research project. Speak to the Policy Engagement Team and explore the Oxford Policy Engagement Network, engage with a colleague at Oxford, or at another UPEN university, who shares your policy interest. They are likely to be able to provide advice and support, or to signpost to others with deep expertise.