Going deeper in social impact assessment: A focus on meaning rather than monetisation

Program outcomes need to be experienced as meaningful by participants if they are to result in longer-term social impact, writes Dr Gianni Zappalà, in this article highlighting a new approach to social impact assessment known as Meaningful Evaluation. 

Central to social impact is a focus on outcomes, the change that occurs in a person or program participant resulting from an intervention. Many social impact frameworks, however, encourage measuring outcomes to remain at a surface level, leading many social purpose organisations to question the quality of the outcomes measurement they undertake. 

Based on my experience as an educator, funder and evaluator in the social impact field over the past two decades, there are at least five factors that contribute to this lack of going deeper in understanding outcomes.

1. The first is an over reliance on the Impact Value Chain (Figure 1). The problem is that the “if-then” causal logic inherent in this chain oversimplifies and reinforces the view that change is linear, predictable, straightforward and mechanistic. The assumptions underlying the logic as to why a short-term outcome will lead to a medium-term and then a long-term outcome and eventually impact are often not clearly articulated, nor whether the change is sustainable.

Figure 1 The Impact Value Chain

Source: Clarke et.al. (2004) 

2. The second reason is the de-contextualisation of outcomes, with many social impact frameworks (e.g. social return on investment) describing, analysing and presenting outcomes as single line items along a spreadsheet or outcomes map, seen outside of their broader holistic context.

3. The third factor is the desire to monetise outcomes, which is seen as appropriate and something the sector should strive for, “…as it helps to translate impact assessment results into the common language: the dollar sign”. The trend to give outcomes a financial value gives incentive to focus on those outcomes that are more amenable to monetisation while ignoring the qualitative and relational aspects of outcomes.

4. The fourth reason is the “tyranny of the tangible”, the tendency to focus only on outcomes that are external, quicker and easier to measure and demonstrate, the so-called “hard” outcomes. The focus is on known outcomes rather than those that are unseen and not usually articulated, ignoring so-called “soft” or internal outcomes that are less tangible and embodied in processes rather than explicit in external outcomes.

5. The final factor is the continued dominance of a positivist approach to validity, that usually treats program participants as passive “objects” when collecting and measuring outcomes data rather than valuing the usefulness of the data collection tool for program staff and clients.

Meaningful Evaluation

I developed Meaningful Evaluation to address some of these issues as well as the desire to deal with complex social phenomena in a holistic and systemic manner. 

It is a “right-brain” perspective encouraging a more participative worldview compared to the dominant “left-brain” evaluation perspective which tends to be mechanistic and reductionist. The approach combines the Map of Meaning (MoM), a simple but profound framework that brings together the main dimensions that make work and life meaningful, with insights from next generation evaluation and complements rather than replaces traditional approaches to evaluation. 

Figure 2 The Meaningful Evaluation approach

Source: Zappalà (2021) 

Simply stated, Meaningful Evaluation puts in play a process that uncovers the extent to which participants in any given program experience the various activities and outcomes of a program as “meaningful” as defined by the MoM. It identifies and examines the main pathways along which meaning occurs (referred to in Meaningful Evaluation as: reflect; connect; respect; express, and summarised in the rectangles outside the circle in Figure 2) as well as any imbalances or barriers to meaning. By doing so, it enables the identification and assessment of the sustainability of any outcomes experienced, captures the inner process of change and any unintended consequences from the participation or program intervention. 

Key issues explored in Meaningful Evaluation include whether and how participation in a program: 

• leads to experiencing a sense of moral development, personal growth and authenticity (the reflect pathway);
• enables a sense of connection and unity via working together, sharing values with others and creating a sense of belonging (the connect pathway);
• provides opportunities to make a positive difference and assist others and/or the environment (the respect pathway); and
• enables the expression of participants’ full potential through creating, achieving, and influencing others (the express pathway).

It also examines people’s inspiration or motivation to participate in a program, and how their ideals are constrained or mediated by the reality or boundaries of the program (i.e. context).  

Five key assumptions underpin Meaningful Evaluation:

1. It is more likely that immediate outcomes lead to medium and long-term outcomes that are significant and sustainable, that is, achieve social impact, if the experience/s from participating in a program are “meaningful”.
2. Meaning is an important internal outcome, needed for longer-term external outcomes to occur.
3. Capturing meaningful unintended outcomes is just as important as capturing intended outcomes, especially where people did not experience those intended outcomes as meaningful.
4. For outcomes to be sustainable, people’s inspiration and motivations for participating in a program need to be contained within the contextual boundaries of the program.
5. The more balanced and extensive the experience of meaning across a program is, the more likely that any intended outcomes achieved are long lasting and lead to impact. 

Measuring meaning

Meaningful Evaluation has an open view with how data on meaning is collected. It does not prescribe any one method for measuring meaning but rather can be adapted to suit the principles of the inquiry paradigm adopted by the researcher. One can use a structured questionnaire to capture and measure the extent of meaning experienced along the four pathways as well as tensions in Figure 2. Alternatively, the Meaningful Evaluation framework can also inform interview and focus group protocols or guide most significant change analysis. 

Meaningful Evaluation also lends itself to participatory approaches that treat participants as active agents and validate their experience and perception of the change process. A person receiving a service is an active agent in their own life, inner change is key to achieving outer change, and data on change is collected collaboratively or co-produced by participant and program worker. The process of participating in an assessment can itself result in change through engagement and reflection. 

The process of Meaningful Evaluation identifies outcomes that are significant and sustainable, that is, achieve social impact by uncovering the “meaningfulness” from participating in a program. While a practitioner community is applying the MoM across a range of areas (e.g. leadership, change) its application to evaluation and social impact assessment is relatively recent and thus requires further testing and refinement. 

The framework has been applied in an evaluation of aspects of the Primary Ethics program in NSW. Its initial application has been promising as has the response from social impact practitioners. I invite others that share similar concerns with some of the directions of outcomes measurement practice to consider the Meaningful Evaluation approach.