Progress is Slow in Measuring Social Impact

There is a disconnect between government and the social sector around measuring social impact, writes Australian Social Value Bank impact specialist Andrew Callaghan, who explains why both sides must commit to tackling fundamental issues in impact measurement.

The measurement of social impact has grown and matured over the last decade. There is a growing body of resources, practitioners and approaches to measuring impact. But while this growth can be seen as a success, there is no doubt that it has not reached its full potential to do good in that time and that we are still grappling with many of the same challenges today as we were 10 years ago.

There is agreement between the social sector, federal and state governments that an outcomes-based approach to planning and measuring the impact of social programs will produce greater benefits for society.

All of these stakeholders have invested large amounts of time over the last decade consulting, designing and publishing outcomes frameworks. There is general alignment around the key domains which need to be focused on at the government level. And the social sector organisations understand they need to align their activities to government outcome domains to receive funding and be commissioned to carry out services on government’s behalf.   

The disconnect between government and the social sector begins to appear when it comes to what indicators are being used to measure the impact of programs against the outcomes they are setup to achieve.

Government indicators in the majority are high level indicators which are highly relevant when reporting on their overall success of achieving an outcome. For example, they may be measuring the reduction of chronic absenteeism from school using Department of Education data which is published at a postcode level. The problem is that for a small not for profit which works with 20 children who attend eight different schools across multiple postcodes this indicator is not so relevant. The 20 students they work with are lost among the thousands of students in each postcode.

The second major issue is that there is no agreement on what methodologies organisations should be using when measuring their social impact. Organisations that have mapped their activities and outcomes to government frameworks have a plethora of approaches and tools available to measure their social impact. While each methodology has its merits and each evaluation may be showing positive outcomes, there are questions around how they choose one and if it will be recognised by their relevant government department.

In many cases, there is no transparency in how organisations have carried out their impact measurement work or if the outcomes they are reporting are a true reflection of their activities. This makes it difficult, if not impossible, for government to be able to compare one program’s effectiveness against another.

How can we scale successful programs if we cannot be confident which one will have the largest impact?

Government and social sector organisations are investing large amounts of financial and human resources into defining what outcomes they want to create and proving that they are having the desired impact. Even the most light-touch of evaluations involves planning, data collection, analysis and reporting, which isn’t cheap.

Right now, there is very little clarity on what cumulative benefit current impact measurement activities are bringing for government or social sector organisations in achieving better outcomes for society.

There is no easy solution, but it does need both sides to commit to tackling some fundamental issues around the way in which we measure social impact. There is also a major part to be played by social impact measurement practitioners, peak bodies and academics in the field of evaluation.

In 2010, DEMOS released a report in the UK about the gap between policy and practice in the measurement of social value. The researchers outlined three fundamental principles that need to be met to reduce this gap:

• Proportionality: So that the burden of evaluation is in line with the scale and nature of the organisation undertaking it.
• Comparability: So that even with a range of flexible frameworks, organisations can still produce outputs based on comparable principles and terms of reference.
• Standardisation: So that there are tools and data available to remove the need to evaluate outcomes from scratch and reduce the burden on organisations.

I had come up with my own list which were simplicity, clarity, consistency and transparency but to be honest I think the authors of the DEMOS report have put it far more eloquently.

In Australia, a CSI Bankwest Social Impact Series report in 2015, outlined the current state of impact measurement and included a look at what is happening internationally. It outlines that similar challenges still exist five years on from the DEMOS report.

Everyone within the social sector, and the state and federal governments, have a part to play in progressing the principles outlined above of proportionality, comparability and standardisation.

Let’s hope in another 10 years time we have made more progress than we have in the last decade on these fundamental issues slowing us down.

About the author: Andrew Callaghan is currently the ASVB impact specialist at the Australian Social Value Bank. Andrew has been measuring social impact for government, corporate and not-for-profit clients for almost 10 years. He writes about his experiences in measuring impact and social value.