UNDERSTANDING RESEARCH IMPACT OF CRIs: how can social sciences help?

By Sudesh Sharma and Suzanne Manning - ESR Social Systems Team

Crown research institutes (CRIs) are government-owned companies that do scientific research for the benefit of Aotearoa New Zealand. There are seven CRIs in Aotearoa, which were set up to lead research and science that address New Zealand’s pressing issues and achieve economic, social and environmental impact. For example, ESR is the CRI that plays a critical national role in health, food safety, groundwater, radiation and forensic sciences, and is a key contributor to public health, environment and biosecurity outcomes. 

While each CRI is focused on its core areas of science and research, the end goal for all of them is to create positive impact for society. A recent review called CRIs an indispensable part of the science system but said there were concerns that there is no mechanism to evaluate the research impact of CRIs. How can CRIs, the government and the public know whether CRIs are being successful if there is no rigorous process to assess the societal benefits? 
​
The Ministry for Business, Innovation and Employment (MBIE), the steward and major funder of CRIs, has recently shifted its focus to investing in research that has potential for social, economic and environmental impact. MBIE wants government funded science to have a clear and strong “line of sight to impact” and has been encouraging CRIs to use a logic model framework (Figure 1) to show this “line of sight”. Using consistent language in logic models helps to compare and evaluate across CRIs, and to focus on impact.

Figure 1. MBIE's definitions of key elements of a logic model framework for research impact

Of course, there are challenges to creating “line of sight to impact”. One challenge is limited concrete mechanisms for measuring impact. Most CRI monitoring and evaluation indicators have been output level measures (e.g. knowledge and skills) and sometimes outcome level measures (e.g. policy influence), but it is difficult to measure the impact of the science out in the community. It does not help that community and society impact of research tends to be incremental and over the long term, in direct contrast to funding mechanisms that have short time frames and pre-set deliverables. These funding mechanisms have arisen from neoliberal approaches that promote competition between CRIs, a “contract for services” approach, and a requirement for CRIs to generate commercial benefits from science.

A focus on impact would require change at both the MBIE/policy level – determining science priorities, creating longer time frames, flexible resourcing for developing and maintaining collaborative relationships, willingness to allow a research agenda to develop in consultation with communities – and at the CRI/science level – explicit mechanisms for measuring impact, focus on authentic and reciprocal relationships, commitment to allow a research agenda to develop in consultation with communities.  
​ 
Another challenge is the traditional approach to Western science that divides research projects and funding into separate disciplines. Yet increasingly it is being recognised that transdisciplinary and collaborative science research projects have the potential to yield greater impact, by addressing the bigger, more complex problems using a variety of different disciplines. This requires collaborative working not only within a like-minded research team, but between researchers with very different perspectives, and even completely different worldviews as when partnering with kaupapa Māori researchers. Most importantly, social science could be a bridge among multiple knowledge systems in an Aotearoa context.

There has also traditionally been a difference in status between science based on ‘things’ versus science based on ‘people’. Yet this is a false distinction, as there are important roles for social science (science with people) especially when considering the societal impact of research and innovation. Social science contributes not only to assessing impact of science on communities, but also ensuring beneficial impact of science for communities through input to the research agenda, collaboration while undertaking research, and feedback and sense-making of results.  

A question to be asked is: given it is relatively well established that complex problems require holistic, transdisciplinary approaches, why have we not focused our science system to operate in this way? One explanation can be found in the systems dynamics archetype of “Shifting the Burden”, as shown in Figure 2.

Figure 2. Shifting the Burden archetype. Solid lines with + signs indicate a causal effect in an increasing direction. Dotted lines with - signs indicate causal effect in a decreasing direction. Interruptions to the lines indicate a delayed effect

This causal loop diagram explains what happens when there are two competing options for fixing it: either apply a short-term fix that addresses the symptoms only or identify and apply a longer term fix to the underlying issue itself. In the diagram, the top balancing loop is the short-term fix, where mono-disciplinary ‘hard’ science solutions are sought to remove the symptoms, resulting in less symptoms but only temporarily.

​The bottom balancing loop addresses the problem in all its complexity, using transdisciplinary approaches and being informed by mātauranga Māori. However, it is the more difficult option and takes a longer time before the impact is seen. More resources get put into the piecemeal science approach because that is what gets the ‘quick wins’, and consequently undermines efforts to address the roots of the complex problem. As a result, a reinforcing addiction loop is formed because the burden of fundamental fix is too hard and business as usual continues. 

For example, one of the key performance indicators for CRIs is peer-reviewed publications. They are considered evidence of success and good publication records support scientists to get more research funding. However, publications are simply outputs, not impacts. There is no equivalent recognition given for outputs of relationships and transdisciplinary collaboration that can generate genuine societal impact. The Shifting the Burden archetype could also be called the “frustrating” archetype for those who are working in the science impact space.
​
The way out of this trap is for courageous and bold systems leadership across the system, where a plan for science, research and technology is developed that has an intergenerational view of impact, a transdisciplinary approach, and is underpinned by mātauranga Māori. Why do this? Because arguably the current approaches are not solving our complex problems. And if nothing changes – then nothing will change.