In brief

Is it possible to be more relational in how we evaluate and determine impact? What organisational conditions help or hinder this? What’s working and what still needs work?

This month, with our Relational Funders cohort, we turned our attention to evaluation – with the challenge of examining current practices, the origins of these, and what it means in practice for those they work with, particularly grant holders.

Impact and evaluation are closely connected but they aren’t the same thing. One explores with neutrality and the other is laden with opinion. Impact examines a journey of change – what effect has this approach/project had? We might have set out with certain goals or ambitions in mind and we’re trying to determine whether we achieved the change we hoped for. And equally important, we should explore what, if any, unintended consequences, also emerged as part of this work.

But evaluation is an act of judgement, specifically (according to the Cambridge Dictionary):

“the process of judging or calculating the quality, importance, amount or value of something.”

Nesta’s 2020 Meaningful Measurement Report flags an overreliance on traditional methods in the public sector – that a focus on the accountability side of measurement has led to a neglect of learning. It aligns strongly with the learning that forms the foundations of IVAR’s 8 principles of open and trusting grantmaking – namely “be proportionate”, “don’t waste time” and “ask relevant questions.” On a scale of maturity in monitoring, evaluation and learning, there remains a continued weight towards compliance, accountability and performance – the features of a more ‘command and control’ style relationship. In this instance, the judgement that counts is that of the funder – they determine whether money has been spent appropriately, whether there’s been suitable delivery, and whether the work has made a difference.

But when you look to the other end of the scale, in relationships of trust and collaboration, we see learning, and specifically co-learning to build shared insights on what has been learnt together and what this means for the future. This is the space where evaluators, including the funders, aspire to. But how close are they now, and what barriers did we uncover?

Fundamental to any evaluation work, should be the question of why. Why are we evaluating, what’s the purpose and who is the audience? These are critical questions that should frame and determine the methodology and outputs. Here the group confronted some difficult truths – that grant reporting is still currently dictated by funders. That it, like applications and contracting discussed in previous sessions, firmly centres the funder, often to the detriment of the grant holder. They set the requirements, the structure and the deadlines – they also control the knowledge that emerges. This has inevitably contributed to the continued inequalities in the relationships between grant makers and holders. 

“Funders have had a tendency to think more is better – have I done a good enough job if I haven’t asked so many questions? We’ve got to move to a less is more approach.”

– Participant

The question of who holds the power takes us deeper into individual organisations – where the process is set by the auditor’s requests, the organisation’s culture or the Board’s resistance to change. We went back to earlier discussions on permissions and the need to interrogate whether these are real, imagined or unspoken conventions (see Changing Our Permissions blog). Is it an auditor’s preference for a certain approach, or an immovable need? What emerged was a realisation – that what’s been treated as fixed rules may in fact have far more flexibility than we imagine. It is well documented that grant holders can find it difficult to be honest with organisations they depend on for funding. But is there a second conspiracy of silence in grantmaking, those who know the current processes are inadequate but are cautious of the consequences of such honesty? How can we move from a collection of individual champions wanting to initiate change to building a powerful constellation of changemakers able to make it happen?

There are benefits and limitations to both approaches. What we heard from funders was a reliance on quantitative within grant reporting. Many felt there is safety and comfort in more traditional approaches that focus on counting rather than developing deep understanding. Questions bubbled around how easy it is to determine value from methods like storytelling, how well it demonstrates good use of funding, and how hard it is to shift organisational culture. We heard organisations recognise that complex reporting doesn’t work for grant holders, but it persists because it serves to reassure funders about their own performance. They highlighted how it’s the weight of risk-averse organisational cultures that restricts moves to a more trust-based approach:

“It’s habit. There’s some degree of comfort in doing what we’ve done before – it takes time to move away from historical pressure. We need to do more listening and less form filling, we gain much richer information through conversations.”

– Participant

Significant questions of how are being held – what methodologies are the right ones to use? Will these satisfy audit, finance, risk, senior, Board or public scrutiny? How can they bring others in their organisation on a journey of change to a better approach?

There was also recognition that this tendency to be risk-averse is shaped by the systems they operate in – the need to account for public money, the need to demonstrate credibility in the sector, the need to audit and demonstrate value to stakeholders. Although there is a growing shift towards more collaborative, flexible and relational approaches to evaluation, these practices remain the exception rather than the norm.

One of the strongest themes from the discussion was the gap between individual intent and organisational systems. Participants hold strong relational values, but they feel constrained by the processes, cultures and expectations that pull them back towards more transactional approaches. It highlights yet again the importance of recognising and working to build infrastructure that supports relational working, not just relying on the passion and skills of individuals.

We also heard concerns of whether the sector is seen by government as being light in evidence and light in data, and an undercurrent of concern that qualitative and learning-centric methods would not bolster credibility. How funders themselves may feel under pressure to maintain existing ways of working due to the systems they operate within, and those they need to be accountable to.

The pull to focusing on quantitative comes from its ability to simplify complexity. To focus on the count of outputs as the key marker of success, alongside accounting for where each pound has been spent. But at what cost? An overreliance on compliance and accountability locks organisations into prioritising measurement, to maintaining a paternalistic approach and crucially, applying a reductionist approach to the complexity of society. It risks assuming linear progress and simple causal links, it reduces people, communities and places to numbers – it doesn’t get to the heart of true impact and change. In his blog From Metrics to Meaning about the method Most Significant Change, Nick Andrews, of Swansea University, argues for the power of stories to unlock true meaning and learning. He says:

“Stories engage both the head and the heart, inspiring compassion, connection and meaningful learning. Talking and thinking together about stories helps us focus on what matters to people over process-driven care and support.”

It’s a challenge the group sees clearly. The grant holders who send through photos even when it’s not part of the reporting process. An act that creates such joy for funders, but almost wastefully, sits outside of standard process. Some funders recognised “that our grantees want a relationship, want a human element, but we’re not really set up to provide that.” This features particularly when funders are working at scale.

Many recognise the risks of reducing projects to reports. Not because metrics are inherently bad, or because stories are always better. But because relationships are fundamental, and they require recognition. “People want to be seen,” a participant asserted. Data may tell us what changed, but relationships help us understand why it mattered.

A further reflection comes when we take a step back from individual grantmaker processes and instead look at the collective impact on the sector. If we have funders admitting they collect more data than they use, that their processes mean they reward applicants that are good at reporting and not necessarily good at creating change and that there remains a tendency to trust metrics more than relationships – what has this meant for the voluntary and community sector?

A sector already under unbearable pressure, with many dependent on grant and foundation funding for survival. They have little choice but to engage – to reduce their work to neat boxes, to spend time completing reports, to account for money and to let others judge whether their version of impact has been achieved. As one funder noted, current practice feels “the very antithesis of relational.” The act of reducing the work also reducing the relationships.

So all this begs the question of what a better future would look like. What if evaluation was more of a collaborative venture between partners? What if the process and approach was agreed in partnership, what if the evaluation looked at both sides of the relationship, and what if its focus was on generating learning for all?

Our funder group recognise they are individuals often working in large organisations within a complex system. There is danger change feels out of reach. Certainly, there is no magic bullet to this challenge of how to shift the approach to a more relational one, instead it is an accumulation of adjustments, which together, have the potential to deliver a much more equitable future. For funders and grantmakers, reflecting on their findings of the Funding Experience Survey,  IVAR’s 8 principles of open and trusting grantmaking offer a great starting practical point and for local authorities, NESTA’s report on Meaningful Measurement provides insight, provocations and a wealth of practical experience. And our own page  on Measuring Relationships offers more resources, including introducing more relational methodologies.

To close, we share reflections from one of the participants, Jo from UK Community Foundations:

• Are we better at counting transactions than understanding relationships? 
• If relationships are the foundation of resilient communities, why do we invest so much effort measuring everything except relationships? 
• What would evaluation look like if it started with connection, trust and belonging rather than outputs? 
• What are we missing when we treat relationships as a means to an end rather than an outcome in themselves?

Changing evaluation is more than changing reporting forms. It takes us back to questions about rethinking power, building trust, and backing statements of what we value with action.

Learning Out Loud

Enjoyed this blog? Catch up with the rest of the series:

1. Six emerging needs
2. Power in partnerships
3. Endings and beginnings
4. Relational offsetting
5. Changing our permissions
6. The impact of evaluation

Please share any thoughts or questions in the comments, or contact katherine@relationshipsproject.org.