Foto: E. Rudzitis

Unlike the private and public sectors, the voluntary sector lacks any other means of measuring its performance, like share prices, profits and election results, hence the need for some kind of performance measurement system tailored to the sector.

Introduction

With an annual income of around £16bn, the UK voluntary sector is increasingly perceived as ‘big business’, and arguably it is. But there is a key difference between the voluntary and private sectors: in the absence of share prices or profit figures, how do we know what we are achieving? In other words, how do we know what our impact is?

This is a difficult question for voluntary organisations. A wealth of literature and research over the last 20 years has gone some way towards providing some of the answers, but there is still a long way to go. The diversity of the sector means that there can be no “one size fits all” approach for measuring impact. And, although a range of tools is available for organisations that want to measure some aspect of their impact, some of these are resource intensive and difficult to implement. Here, we explore some of the reasons why organisations choose to measure their impact, as well as some of the obstacles that get in their way.

Why measure impact?

Motivation for measuring the voluntary sector’s impact can come from a number of sources. However, while few would question the benefits of measuring an organisation’s impact, the Government’s recent emphasis on “what works” and the voluntary sector’s involvement in the delivery of public services, means that the sector has more reason than ever to demonstrate its worth and this has led to added interest in impact measurement.

Government initiatives and public service delivery: Government initiatives in the UK including Best Value, and the recent Treasury review of the delivery of public services, mean that voluntary organisations are keener than ever to demonstrate their ability to deliver public services.

Voluntary organisations are competing not only with each other for the public’s donations, but with other public, voluntary and private sector organisations for public service delivery contracts. They need to be able to demonstrate their added value; that is, any particular qualities or strengths that the voluntary sector has in comparison with the other sectors. These might include the ability to engage the community, to contribute to social capital and to access hard-to-reach groups. It is generally accepted that with these distinct qualities, the sector needs a distinct type of performance measurement. The danger is that if the voluntary sector is judged on the same criteria as the other sectors, it will not be demonstrating the characteristics that make it unique. Also, unlike the private and public sectors, the voluntary sector lacks any other means of measuring its performance, like share prices, profits and election results, hence the need for some kind of performance measurement system tailored to the sector.[1]

Funders: Additional external pressure to measure may come from non-government funders. While some funders may ask for evidence that a funded organisation or programme has met its outcomes, some will require proof of its past performance as a prerequisite to awarding funding. Or, a funding organisation may itself want to carry out an evaluation of its funded projects in order to establish the impact of that funding.

Donors: Motivation for measuring impact may also come from donors. With an increasing number of organisations to choose from, donors are becoming more discerning in their choice of who to give to. Not only will an organisation want to demonstrate to its existing donors that it is using funds in an efficient and effective way, but it will also want to appeal to potential donors. Furthermore, the public not only funds voluntary organisations directly through donations, but indirectly through taxes, and has an additional stake in the sector as users and beneficiaries: all reasons why voluntary organisations are keen to be publicly accountable.

Internal motivation: Equally, motivation to measure impact may come from within an organisation. It may want to collect information on its own performance in order to help it to improve efficiency and effectiveness. This information may be financial, like expenditure, or it may be outcome information based on organisational objectives. There are also benefits to an organisation arising from the actual process of impact measurement. The process of defining aims and objectives is in itself useful for encouraging staff to think about and clarify their organisation’s goals. Also, measuring and demonstrating impact can help to motivate staff by providing them with concrete evidence that their work is making a difference.

Soft outcomes: Finally, there has been a lot of discussion in recent years about “soft outcomes”: intermediate outcomes that are achieved along the route to attaining the ultimate goal of an activity or organisation. It is possible to think of the progression towards a final goal as a journey, and even if the end result is not achieved, some progress will almost certainly have been made towards this goal. The amount that has been achieved in the process is sometimes referred to as “distance travelled” and usually consists of “soft outcomes”. These tend to be fairly intangible in nature and may be seen as stepping-stones, or necessary achievements en route to the final, hard outcome. In the case of getting an unemployed person into work, they may include increased confidence or a change in attitude. The danger is that funders can be primarily concerned with the final, hard outcome, and will see anything short of this as a failure. Organisations therefore want to be able to demonstrate that something worthwhile has been achieved – that some distance has been travelled.

Obstacles to impact measurement

Impact measurement is rarely straightforward. There are numerous challenges to be faced by any organisation measuring its impact, and as a rule the more intangible or long-term the factors being measured, the more difficult the process. Furthermore, even if no problems are encountered along the way, there are limits to the usefulness of different types of impact measurement. This section explores some of these problems and limitations, helping organisations to be realistic about what can be achieved, and to be prepared for the possible pitfalls.

Prevention activities: Organisations whose main purpose is the prevention of something, like teenage pregnancies or drug use, may find it hard to demonstrate positive impacts, partly because success for them is something not happening, and also because it is often difficult to identify exactly who the beneficiaries are. For example, a particular drug use prevention advertising campaign may result in five percent fewer people using illegal drugs in a particular year. However, identifying those people who would have taken drugs if it were not for the programme is likely to prove impossible.

Baseline data: Some types of measurement require baseline data representing the position before the programme or activity begins, which can be used for comparison after the event. This assumes that you know what the types of outcomes will be in advance in order to be able to collect the relevant baseline data. While this is a common way of measuring impact, it is somewhat restrictive as it limits the measurement process to outcomes that were anticipated beforehand. It can result in a rather blinkered view of impact, and makes it unlikely that any unanticipated impacts will be identified during the process.

Impact of the measurement process: Another problem is the impact of the measurement process itself. In many cases this may be an advantage: the process of measuring impact may make staff think about their objectives and activities in new ways. However there will be occasions when any effect resulting from the intervention of an assessor will not be desirable as it will affect the extent to which impact can be attributed to the activity or programme. Similarly, there may be ethical issues to take into account when assessing the impact of an activity that deals with sensitive issues or vulnerable clients.

Attitudes to measurement: Attitudes to measurement often prove to be a barrier, and these can include a mistrust of measurement in general. Managers may perceive it as a waste of resources that should be spent on delivering results rather than measuring them. They may see impact measurement simply as more unnecessary paperwork, taking up time that could be better used elsewhere. Members of staff may feel they are being examined or the quality of their work scrutinised. Funders may feel unhappy about a proportion of their grant being used for measurement, and in turn, organisations bidding for funding may not feel able to justify including impact measurement in their funding application.

Diversity of the sector: The heterogeneity of the voluntary sector poses a number of problems for impact measurement. No single methodology is applicable to all voluntary organisations. Clearly there is the need for a wide range of methodologies tailored to the requirements of different types of organisation, depending on their type, size, activities, objectives, the aspects of their impact that they want to measure and their reasons for measuring their impact. Furthermore, the diversity of the sector means that developing a method of measuring the impact of the sector as a whole is likely to be impossible.

Cost: Perhaps the most fundamental of all the objections to impact measurement is that of resources. Measurement of any type requires resources, no matter how much an organisation manages to integrate the measurement process into its everyday routine. Some performance measurement systems have been developed especially for small organisations (eg PQASSO[2] and Quality First). While not measuring impact as a whole, they at least provide small organisations with a means to demonstrate and improve some aspects of their impact.

Conclusion

Despite these obstacles to measuring and demonstrating impact, organisations have begun embarking on impact measurement of one type or another with some degree of success. The lack of a one-size-fits-all approach for this most diverse of sectors means that organisations have to chose an approach that works for them – either an off-the-shelf system for their particular size or function, or perhaps a tailor made, in-house approach.

Measuring impact is a learning experience for us all, and although we are doing a pretty good job of identifying the problems, we are a long way from finding all the solutions. One thing is for certain though – impact measurement is here to stay, and this is no bad thing, particularly in a world where we place so much importance on organisational transparency and accountability.
______________________________________

[1] Kendall and Knapp, 1999

[2] Practical Quality Assurance System for Small Organisations. http://www.connexions.gov.uk/partnerships/index.cfm?CategoryID=4&ContentID=115