Case study: Capturing the impact of a challenge prize

Can a challenge prize improve farming in Nepal? The Data-Driven Farming Prize developed an impact framework to capture its effects on problem-solving, building capabilities and wider knowledge and awareness, writes Geoff Mugan, in the first of eight examples of different approaches to capturing impact.

The DDF Prize aimed to support solutions for smallholder farmers and agricultural stakeholders in Nepal, to help them make effective choices to enhance on-the-ground decision making, their productivity and market planning. The prize also aimed to recognise and promote localised data-driven solutions that could be accessible to and used by smallholder farmers to improve agricultural productivity in a sustainable manner.

Global concern regarding agricultural and food production has been growing. By 2050, over a billion people will be at risk of hunger, with arable land becoming increasingly scarce. Nepal’s agricultural sector contributes 26 per cent of the national GDP as of 2017, with 71 per cent of the population working in this field as of 2018. Nepal has the potential to be a food surplus country if smart, sustainable intensification that includes smallholder farmers can be realised.

The method

The impact methodology used in the Challenge Prize Centre is structured around the impact framework, which includes three key impact categories.

1. Innovation impact: Assessing to what extent the prize process had an impact on solving the specified problem or bringing about a significant advance in tackling a social issue.
2. Capabilities impact: Assessing if there are a greater number of innovators who are working on this prize and to what extent their networks or skills have been enhanced by participation.
3. Ecosystem impact: Assessing to what extent there is greater knowledge and awareness related to tackling the challenge, what is the role of policy, how much investment in solutions has increased and what changes occurred in the market.

The three categories reflect the types of impact that all prizes should have – prizes are meant to solve problems, improve capabilities, and lead to broader systemic changes. Using the same framework across all prizes ensures more consistency than creating a bespoke process each time. It also allows for comparison between different prizes, which leads to recommendations on how to improve prize methodology.

The DDF Prize evaluation was run throughout the prize process, enabling the collection of relevant data to inform a robust evaluation. The outcomes and key results from the prize are noted below.

The impact

DDF to inspire new solutions

On innovation impact, the DDF prize focused on:

• Attracting new approaches and tools to source, organise and translate data into actionable farming insights: 13 new solution prototypes and complete business plans.
• Improving opportunities for more effective and efficient agricultural decision making: positive feedback from testing solutions, recorded from overall 150 testing pools with a reach of more than 2,000 smallholder farmers in Nepal.

On capabilities impact, the DDF prize focused on:

• Mobilising new talents towards the Nepali agricultural market: 58 per cent of entrants from Nepal.
• Building new partnerships in the agricultural value chain in Nepal: over 70 per cent of finalists established new partnerships during the prize.
• Build context specific capacity and responsiveness to local user needs in innovators: all finalists have learned skills useful for their personal and business progression.

On ecosystem impact, the DDF prize focused on:

• Support scaling new products/services into the agricultural market in Nepal: eight out of 13 finalists likely to enter the market with their solution within six months of the prize ending, and two within the year.
• Leverage investments across stakeholders to support solutions entry into the market: the DDF prize received approximately $246,000 in-kind support from partners, equal to 17 per cent of USAID total investment.

Following this impact methodology allowed us to better map out the narrative on the impact that the prize would have. By understanding the benchmarks, baselines and impact metrics, the prize was able to build a strong evidence base. We used various evaluation methods including surveys, interviews, focus groups, desk and field research to collect key information.

The data collected at various stages of the prize helped inform certain prize decisions such as with early research into the ways in which farmers take decisions; this informed innovators on how best to frame their proposed solutions or the types of request for mentoring support, allowed us to allocate mentors who were most appropriate for the finalists.

We did not adjust the prize design based on any of the data we had collected during the prize, as the prize design was sufficiently robust. We were, however, able to provide innovators in the prize with greater access to farmers and extension workers based on this feedback.

What we learnt

Evaluation methods

We learnt that it is necessary to have a mix of evaluation methods at every stage in order to gain perspectives of innovators and partners. It is also important to have high response rates to surveys. This proved difficult in the DDF prize and involved having to chase the respondents. However, as part of another prize, we tried to gain the maximum responses to our surveys by ensuring that surveys were sent out alongside key milestones eg finalists pre co-creation event survey was sent with the seed funding contract.

Ecosystem impact

A key challenge was the difficulty in measuring the broader ecosystem given the magnitude of the problem that was being tackled in Nepal. Therefore some of the indicators for the DDF prize were either proxies and calculations, but in some instances we were unable to fully capture the ecosystem impact. For example, the number of jobs created as a result of the prize; though one of the finalists was able to hire a new employee, this was an indicator that plays out over longer timescales than we were able to track during and immediately after the prize.

To address this, the team will be conducting a post-prize assessment in Nepal (nine to 10 months after the prize). This work will be focused on three key areas:

1. The innovators approaches since the prize ended (their business and revenue models that ensure sustainability and scalability).
2. The prize ecosystem which includes the market (agricultural value chain), private sector entities (telecommunication providers and financial institutions) and government engagement that innovators had, as well as general awareness of ICT4Ag/AgTech in Nepal.
3. The prize as a stimulus – whether it creates attraction to an emerging market space (agriculture) where ICT and technology can accelerate change.

Prize

There were some key lessons that were learnt from the utilisation of a prize as a programme.

• An incentive tool for innovation: a meaningful and robust prize statement is essential to attract the right innovators and innovations. This means that, to be effective, a prize needs to factor the needed time for research and design of its elements. In particular, the DDF prize would have benefited from more time to run in-field research to further scope the prize statement.
• Attracting local solvers and generating impactful solutions: to attract local solvers it is essential to receive USAID mission support, and to run in-country events which can generate the needed momentum to create incentives for local solvers to participate.
• A tool to support human-centred design (HCD) approaches and co-creation: a prize is a staged process, during which activities such as HCD and co-creation add value to the whole journey innovators take during the prize to further develop their solutions. All finalists in the DDF prize benefit from testing insights and new partners as a result of the co-creation activities designed in the prize.
• A tool to generate partnership and engage with the private sector: this prize allowed for strong partnerships to form with expert agricultural and digital organisations. It was also particularly well received by the USAID country mission as a tool to engage with the private sector in supporting innovative solutions to tackle social problems.

What’s next?

The evaluation confirmed that this was a well-designed and well-implemented prize, serving as a model for the future. We are reviewing the impact framework, to further strengthen it.

This includes looking at:

• whether we are focusing on the correct metrics;
• whether we can more consistently capture impacts that potentially fall into more than one of the categories;
• whether we can prove causality; and
• ensuring consistency in following up in the years following the prize to identify lasting impact.

Overall, the impact of this prize was overwhelmingly positive, and the success of this prize was such that it inspired and informed the design of future prizes, including the Fall Armyworm Tech Prize.

The design elements that were integrated into this prize following the Data Driven Farming Prize were:

• Conducting in-country events through the field visits and co-creation event in Uganda, to not only increase awareness, but ensure local participation and engagement in the prize.
• Extended testing period for innovators to test their solutions and gain richer and deeper results and feedback, which can be integrated before the submissions of their development plan.
• Better informed evaluation processes to map out the core capacities of innovators prior to the first stage of assessment to build a stronger impact narrative around capabilities.

About the author: Geoff Mulgan has been chief executive of Nesta since 2011. Nesta is the UK’s innovation foundation and runs a wide range of activities in investment, practical innovation and research.

This article was first published by Nesta.

It is one of a series of eight examples pulled together by Nesta to highlight different approaches to capturing impact they have used in recent years.

Read Geoff Mulgan’s article here about why it’s essential to try and track what is being achieved.