Close Search
 
MEDIA, JOBS & RESOURCES for the COMMON GOOD
Analysis  | 

Progress is Slow in Measuring Social Impact


7 January 2019 at 1:25 pm
Andrew Callaghan
There is a disconnect between government and the social sector around measuring social impact, writes Australian Social Value Bank impact specialist Andrew Callaghan, who explains why both sides must commit to tackling fundamental issues in impact measurement.


Andrew Callaghan | 7 January 2019 at 1:25 pm


2 Comments


 Print
Progress is Slow in Measuring Social Impact
7 January 2019 at 1:25 pm

There is a disconnect between government and the social sector around measuring social impact, writes Australian Social Value Bank impact specialist Andrew Callaghan, who explains why both sides must commit to tackling fundamental issues in impact measurement.

The measurement of social impact has grown and matured over the last decade. There is a growing body of resources, practitioners and approaches to measuring impact. But while this growth can be seen as a success, there is no doubt that it has not reached its full potential to do good in that time and that we are still grappling with many of the same challenges today as we were 10 years ago.

There is agreement between the social sector, federal and state governments that an outcomes-based approach to planning and measuring the impact of social programs will produce greater benefits for society.

All of these stakeholders have invested large amounts of time over the last decade consulting, designing and publishing outcomes frameworks. There is general alignment around the key domains which need to be focused on at the government level. And the social sector organisations understand they need to align their activities to government outcome domains to receive funding and be commissioned to carry out services on government’s behalf.   

The disconnect between government and the social sector begins to appear when it comes to what indicators are being used to measure the impact of programs against the outcomes they are setup to achieve.

Government indicators in the majority are high level indicators which are highly relevant when reporting on their overall success of achieving an outcome. For example, they may be measuring the reduction of chronic absenteeism from school using Department of Education data which is published at a postcode level. The problem is that for a small not for profit which works with 20 children who attend eight different schools across multiple postcodes this indicator is not so relevant. The 20 students they work with are lost among the thousands of students in each postcode.

The second major issue is that there is no agreement on what methodologies organisations should be using when measuring their social impact. Organisations that have mapped their activities and outcomes to government frameworks have a plethora of approaches and tools available to measure their social impact. While each methodology has its merits and each evaluation may be showing positive outcomes, there are questions around how they choose one and if it will be recognised by their relevant government department.

In many cases, there is no transparency in how organisations have carried out their impact measurement work or if the outcomes they are reporting are a true reflection of their activities. This makes it difficult, if not impossible, for government to be able to compare one program’s effectiveness against another.

How can we scale successful programs if we cannot be confident which one will have the largest impact?

Government and social sector organisations are investing large amounts of financial and human resources into defining what outcomes they want to create and proving that they are having the desired impact. Even the most light-touch of evaluations involves planning, data collection, analysis and reporting, which isn’t cheap.

Right now, there is very little clarity on what cumulative benefit current impact measurement activities are bringing for government or social sector organisations in achieving better outcomes for society.

There is no easy solution, but it does need both sides to commit to tackling some fundamental issues around the way in which we measure social impact. There is also a major part to be played by social impact measurement practitioners, peak bodies and academics in the field of evaluation.

In 2010, DEMOS released a report in the UK about the gap between policy and practice in the measurement of social value. The researchers outlined three fundamental principles that need to be met to reduce this gap:

  • Proportionality: So that the burden of evaluation is in line with the scale and nature of the organisation undertaking it.
  • Comparability: So that even with a range of flexible frameworks, organisations can still produce outputs based on comparable principles and terms of reference.
  • Standardisation: So that there are tools and data available to remove the need to evaluate outcomes from scratch and reduce the burden on organisations.

I had come up with my own list which were simplicity, clarity, consistency and transparency but to be honest I think the authors of the DEMOS report have put it far more eloquently.

In Australia, a CSI Bankwest Social Impact Series report in 2015, outlined the current state of impact measurement and included a look at what is happening internationally. It outlines that similar challenges still exist five years on from the DEMOS report.

Everyone within the social sector, and the state and federal governments, have a part to play in progressing the principles outlined above of proportionality, comparability and standardisation.

Let’s hope in another 10 years time we have made more progress than we have in the last decade on these fundamental issues slowing us down.

About the author: Andrew Callaghan is currently the ASVB impact specialist at the Australian Social Value Bank. Andrew has been measuring social impact for government, corporate and not-for-profit clients for almost 10 years. He writes about his experiences in measuring impact and social value.


Andrew Callaghan  |  @ProBonoNews

Andrew Callaghan is ASVB impact specialist at the Australian Social Value Bank and has been measuring social impact for government, corporate and not-for-profit clients for almost 10 years.


Get more stories like this

FREE SOCIAL
SECTOR NEWS

2 comments

  • fiona doyle says:

    Great article Andrew – well balanced and very interesting read. Whilst, as you outline, some progress has been made – and much time invested in measuring outcomes – it seems there is a real need to bring the social sector and relevant government bodies closer together so they can agree tangible actions and co-create realistic solutions to address what needs to be done so that the impact of social programs, in creating benefits for the society around, can be properly measured, compared and where successful, rolled out further.
    Look forward to the next instalment!

    • May Lam says:

      This is a really good analysis of the core challenges for social impact definition, reporting and ways to assess whether the intended (or even unintended) value was actually created. One important question in this is who in government will lead the way, and how? We see a lot of programs and services contracted (or grant-funded by philanthropists) without reference to other programs or services, and though ‘collective impact’ is a good way to formulate and frame what should happen, there is still too little attention paid to which data should be used to understand, define and report success. Where is the debate about options for using real-time big data, acknowledging the politics and risks of that? Even having the debate might help to clarify what information could or should be joined up, even if anonymised.


YOU MAY ALSO LIKE

Following in one charity’s AI-powered footsteps

Ruby Kraner-Tucci

Wednesday, 29th March 2023 at 4:09 pm

Preparing the NFP sector for Privacy Act and cybersecurity reforms

David Spriggs

Monday, 27th March 2023 at 1:37 pm

There’s no need to fear, cryptocurrency is here

Ruby Kraner-Tucci

Wednesday, 22nd March 2023 at 1:20 pm

Recognising the sector’s excellence in technology

Contributor

Tuesday, 21st February 2023 at 9:37 am

pba inverse logo
Subscribe Twitter Facebook
×