Developmental Evaluation – Time to Act

It’s time for a different approach to evaluation if Australia is to tackle its most complex problems, writes collective impact specialist, Kerry Graham.

Complex problems are hard work. What makes them unnecessarily harder is the way we evaluate social policy and programs. If we are to make progress in complex problems, we need to rethink how we measure and evaluate social impact.

Nature of complex problems

It is now well accepted that collaboration across sectors is the most effective way to respond to complex problems. Problems such as child abuse and neglect, displacement of people, the impacts of climate change, disengagement from education, homelessness, crime and poverty.
Complex problems are hard for lots of reasons.

First, they are characterised by multiple layers of stakeholders, with different perspectives and disagreement about the causes of the problem and the best solutions. They require working across organisational boundaries, as they are beyond the capacity of any one organisation or sector to respond to.

They are also hard because they will not be solved by our current ways of thinking and working. To make this very tangible – complex problems require emergent strategies – ones borne from learning by doing. Yet this way of creating impact is largely at odds with the way funders and others manage risk through things like three-year strategic plans with KPIs and highly prescribed funding contracts.

Learning by doing requires permission and support to test and learn through “small bets”, possibly fail, and definitely adapt before placing “big bets”. It also requires access to real-time data, regular facilitated learning to derive insights from the data, and support in understanding what success in complexity looks like. Our traditional evaluation approaches provide none of these essential things.

How we currently evaluate

Traditionally evaluation has been associated with structured, linear style approaches that focus on identifying specific targeted outcomes and measuring delivery against those outcomes based on specific program based interventions. In this traditional approach, focus is placed on understanding the issue or target group in relation to which an intervention is directed, the nature and targeted outputs and outcomes associated with an activity and measuring delivery against those targets at a point in time or on a rolling basis using a consistent, repeatable methodology. (What a mouthful!)

The primary focus of the evaluation activity is on the assessment of what has been delivered or achieved at a specific point in time or over a particular period.

How we need to evaluate complex problems

Complex problems require a developmental approach to evaluation. While developmental evaluation still seeks to understand what is being achieved through a particular activity, it has a much stronger focus on learning. The purpose of developmental evaluation is not just to understand what is happening and what has been achieved to date, but also to provide a source of information through which those engaged in an initiative can reflect on and adapt to what they are learning and improve and innovate on a real time basis.

As such, developmental evaluation is both an assessment and a learning process. Because of that, developmental evaluation is a long-term process that is often delivered by either embedding an evaluator within an initiative or establishing collaborative coaching or critical friend relationships where evaluators work with those engaged in an initiative to support them to develop and apply evaluative thinking skills and to build reflective learning practices into their initiative.

The approaches and tools that are used to support developmental evaluations are by nature more fluid and iterative than those used to support more traditional forms of evaluation. They as they need to be able to adapt and change as an initiative evolves – focussing on different things and/or collecting data in different ways to suit different contexts or situations.

Calling all developmental evaluators!

As an organisation solely focused on supporting leaders, organisations and communities learn how to respond to complexity through effective collaboration, Collaboration for Impact has been searching the country for developmental evaluators. We haven’t found too many.

To draw them out, connect them with each other and develop them, we are offering a learning event called Complexity and Evaluation in an Uncertain World, featuring the best international and Australian developmental evaluators we can find. Those interested in this type of evaluation can go here.

About the author: Kerry Graham has worked in social change for more than 20 years. Over this time her work has broadened from advocacy for and with individuals to leadership of organisations, advising on policy and now – solving complex social challenges. She is a Director of Collaboration for Impact.