Why NFPs need to ask powerful questions

Not for profits looking to measure outcomes must first ensure they are asking the right questions so they can get the right data, according to leading evaluation experts.

Pro Bono News spoke to Julia Coffman and Tanya Beer from Washington D.C’s Center for Evaluation Innovation about the importance of impact measurement for foundations and NFPs.

The duo are in Australia giving a series of talks on how organisations can use evaluative thinking to learn from their work, and bring timely data to the table for reflection and use.

The talks focus on how NFPs should embed a learning culture within their organisation through five key habits including: asking powerful questions, combating biases and making thinking “visible” by clearly articulating what you want specific actions to achieve.

When it comes to data and evaluation, Beer said effective learning was key to effective evaluation.

“Our core proposition is that foundations and non-profits often jump to collecting data and setting up measurement systems without thinking about their surrounding environment,” Beer said.

“As a result we see a lot of foundations and non-profits in the US have spent a lot of resources collecting data and doing evaluation that never gets used.

“So groups need to begin building the organisational capacity to use the data before collecting it or all that effort is for nothing.”

A key component of this approach is that organisations must ask the right questions they want evaluated so they can get the right data and use it successfully.

These questions must be action-focused and invite answers that would make a real difference to an organisation’s work.

A powerful question also must not be too vague or broad, such as “how can we make sure everyone has mental health care?”

Coffman and Beer said a better question would be: “How can we enroll the hardest-to-reach communities where we don’t have any existing relationships or partners?”

They said once you have the right questions, then you can identify the data you need to answer them.

There was some discussion during the Philanthropy Australia-hosted event around whether this process was equally valid for foundations and the NFPs they fund.

Beer said both types of organisation needed to apply the learnings in different ways.

“Our argument here is that for data to be useful for your strategy it has to be tied to what you think it means to be effective,” she said.

“How foundations think about what it means to be effective should be very different from how a non-profit thinks about it.

“This core set of learning habits can also really improve the relationship between non-profits and foundations, because it enables them to have deeper conversations about what’s challenging and creates a more mutually supportive way of interacting with each other.”

Recent research from the University of Western Australia’s Centre for Social Impact found there was a gulf between the expectations of NFPs to measure their outcomes and the funding available to do so.

The report said funders frequently expected NFPs to report on outcomes but were much less likely to provide funding for this.

Coffman said foundations needed to provide NFPs with the funds to evaluate complex areas.

She said research in the US showed less than 10 per cent of foundations funded evaluation of individual grants.

“In order to answer questions and get data from grantees you have to support the capacity to do it,” she said.

“This is very important because most problems that non-profits and philanthropy tries to solve are complex. And it is not possible to know at the beginning if the choices you’ve made are the right ones without testing it.”

But she warned there was a danger that organisations could spend too much time evaluating outcomes rather than actually taking action.

“We do not advocate for evaluating everything, we think that it makes the most sense to think about where organisations have the greatest uncertainty in their strategy and really focus on areas that will have the greatest learning value,” she said.

“It’s better to go deep on a few things than be shallow on a lot of things.”

Beer urged Australia to learn from the mistakes of the US philanthropic scene, which she said spent 20 years thinking the best way to make data more useful was just getting the right data.

“You need to place just as much emphasis on building the capacity for learning and using data as you do on gathering data,” she said.

“We had many years of wasted evaluation resources because of that mistake.”