Aggregation of Impact: Does all this measurement add up?

How do you add impact data together to gain a good overall picture of how your organisation is producing impact, without losing the granularity and specificity of data which makes it useful for redesigning services to increase impact? This is a common problem across a wide spectrum of sectors and organisations, so we at Social Value UK have brought together a set of pieces from individuals who encounter and tackle this issue on a regular basis.

The full publication can be found on the Social Value UK website, but we have republished the blogs here on The Practitioner Hub as well. Below we introduce below why aggregation is important and pull together two broad themes from the set of different blogs.

Why do we need to aggregate impact?

The report explores some of the issues and approaches that are being faced by organisations that want to aggregate, or add up and summarise,  either their impact or the value they are helping to create. 

Any organisation that invests in a range of different activities will need some way of measuring performance so that the different impact of those activities can be compared and resources can be allocated to those activities which create the most impact. This applies to a social business with a range of products or services as much as to a social investor with a portfolio of investments. Comparison is easier with a common yardstick and measuring impacts with a common yardstick makes it possible to aggregate performance.  

Frameworks

Adding up impact is easier if there is a predetermined set of outcomes and indicators and all the activities use the same measures to measure impact. The relative importance of the different outcomes still needs to be determined, but at least the number of measures is manageable. 

For businesses that take this approach, aggregation doesn’t present too much of a problem. At one extreme it would become possible to simply report on ‘lives touched’ – a simple way to aggregate many different types of changes in people’s lives and impact.

However it is increasingly being realised that reporting performance and maximising performance are very different things. Organisations seeking to maximise impact will need to have a detailed understanding of the changes in people’s lives that result from their activities and, potentially, specific measures of those changes for different and specific groups of people. 

It is harder to add all this up, to summarise different outcomes and different indicators into a manageable list. This will require some form of taxonomy and also agreement on the relative importance of different impacts.

The example below shows two different options for supporting employment. As soon as issues around the quality of employment are recognised, it becomes more difficult to compare or to aggregate the total number of jobs created. This example recognises duration, quantity and value but not other factors, for example displacement, that could also be included  Some people will prefer the first project and some will prefer the second. 

Some form of taxonomy and weighting will create transparency and it will make it easier for a group of people to discuss the comparison when coming to a decision. 

More organisations are now confronting this challenge and developing solutions. Most recognise that they are on a journey, and for some it is clear that this is a journey towards maximising impact.

The responses to aggregation from these varying organisations show that this is a common challenge, affecting organisations across very different sectors and scales. However, two key themes emerge:

1.  There is a tension between the amount of detail necessary in determining outcomes, in order to maximise value created, and the ability to aggregate outcomes.

Information that  provides an understanding of the performance or progress of a project will be harder to aggregate.

One solution to this was GVE2.0’s creation of two service layers, which link a project or organisation’s specific, detailed outcomes with a funder’s summary level outcomes, aligned to the funder’s objectives, via a chain of events. This enables aggregation to occur at a high level, whilst maintaining the link to the project’s detailed and unique outcomes, which may include negative or unintended outcomes, and are crucial to an organisation’s ability to make decisions to maximise value.

Another solution is to develop shared measurement for organisations tackling similar issues or operating in a similar sector. A taxonomy, or an outcome matrix, may still be required but it will make aggregation easier. Big Society Capital’s Outcomes Matrix approach helps organisations to identify areas in which they are working which may be shared across sectors, whilst remaining flexible enough to allow incorporation of other outcomes.

2.  The level of accountability of those responsible for activities will influence the completeness of the information being collected, at one end of the spectrum only measuring objectives and, at the other end, measuring all the material outcomes of an activity.

In order for an organisation to be accountable and develop products and services in response, it will need to collect data on all outcomes of an activity (positive and negative, intended and unintended) with regard to a range of stakeholder groups. 

Information used for allocating resources needs to have the right level of detail, accuracy and completeness in order to minimise, as far as possible, the chance of making an incorrect decision. Transparency in the taxonomy used and in the weighting of different impacts will improve decision making.  

More organisations are exploring solutions to these challenges. We would like to hear the experience of other organisations and provide an update to this report.

If you would like to share your views and experience or have any reflections on this report please email: info@socialvalueuk.org.

Download the full report Aggregation of Impact here.

Social Value UK is the UK's national membership network for those interested in social value and social impact. Through our network, we promote the Principles of Social Value, a set of Principles which can help people to measure and, importantly, to manage and increase the social value and impact that they're creating.

If you're interested in learning more about Social Value UK's approach and how to improve your impact measurement and increase your impact, why not take a look at our training? In collaboration with EVPA, we'll be running an Impact Management workshop: the fundamentals on 5th October 2016, aimed at anyone and everyone who wants to improve their impact skills.