Developing an economic case: why data collection and quality of records is important
Evaluation should be built into programmes from the outset and not tacked on at the end and staff should welcome the chance to challenge and change their views on what the agency delivers and why.
Undertaking a cost benefit analysis (CBA)
“Different business cases can be made for commissioners and professionals to get a better understanding of the needs of different service users, leading to better business outcomes.”
Manchester City Council
First and foremost, the agency should be committed to monitoring and evaluating their activities. The Treasury’s Magenta Book (https://www.gov.uk/government/publications/the-magenta-book) is a good starting point for understanding the role of evaluation.
Second a voluntary agency should only undertake CBA if they need a financial/return on investment perspective on their work. If commissioners are happy with qualitative and less detailed quantitative evidence then just do case studies, surveys, interviews etc.
If a voluntary agency decides to undertake CBA, then ideally the following pre-requisites will be in place:
Understanding of client base
Agencies should be able to describe the number and types of clients they will be working with, give evidence of the types of issues these clients face (i.e. background data on levels of unemployment, poor mental health, debt etc).
National/official datasets from ONS (www.nomisweb.co.uk) and government departments (www.data.gov.uk) can often provide this context but also good to look for local survey evidence – why is the issue particularly pressing locally?
Agreement on desired outcomes
There should be identification of one or more measurable outcomes that the project is aiming to deliver. It should be clear how these outcomes link to what activities are being funded and when outcomes will occur and (ideally) how long they will last.
A trickier thing to work out is how much an outcome is ‘worth’ – if a charity is commissioned on a ‘per unit’ or payment by results basis then this is quite easy. If the outcome is wider than that (say, reduced number of A&E visits) then the New Economy’s cost database (this is available on password protected section of the website) and similar databases produced by agencies such as the Department of Health can help.
Good quality data
Charities should keep accurate project management/client records – anonymised where possible – which show when somebody became involved with a project, how long they stayed involved, give objective evidence (i.e. scale or score based) demonstration of the outcomes they achieved – tools such as Outcomes Stars and Survey can be used to good effect very cheaply.
If charities are able to track their clients beyond the intervention, that is even better.
It’s pointless having good quality data if it cannot be shared – informed consent and structured agreements on how the data will be stored, used and shared are quite easy to develop – see www.informationsharing.co.uk or www.ico.org.uk as a starting point. Note that charities have an exemption to share personal data under the Data Protection Act.
Charities should look at their programme and identify when costs are incurred (i.e. some money is spent that would not have been spent), by whom, at what stage in a client’s ‘journey’ – they should pick up costs to all agencies, not just themselves - the principles of Activity Based Costing can help to build this picture.
Knowledge of alternative/past approaches
Has the charity got evaluation reports for its previous/similar work or are they aware of similar schemes that have been reviewed? Developments such as the Dartington Social Research Unit and the What Works Centres are starting to create such databases.
You can still undertake a CBA if some of these pre-requisites are not in place but it means your analysis is more based on assumptions.
Undertaking a Cost Benefit Analysis
External – if the above tasks are beyond the agency then any reputable research consultancy should be able to undertake a CBA on behalf of the charity – if it’s just a CBA of a relatively small project then £5,000-10,000 would cover the cost. The actual modelling will probably take a week or so but there are always meetings to arrange around this, data to source, reports to write etc – so the average CBA takes 4 months but they can be quicker. If it’s a wider evaluation is needed, with more qualitative research then costs might be nearer £15,000-£20,000. NCVO has a directory of approved consultants.
Internal – New Economy run training on how to apply their CBA model and alternatively, resources on the NCVO and New Economics Foundation website in regard to impact and social return on investment are handy. The Treasury’s Green Book (https://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-governent) and publications by BIS (http://www.bis.gov.uk/assets/biscore/better-regulation/docs/i/11-1112-impact-assessment-toolkit.pdf) are more technical documents but a bright member of staff with some knowledge of numbers and excel could probably get their head round the approaches they advocate in a few days.
Using the outputs – findings from a CBA should inform decisions but not be the arbiter of decisions – a voluntary organisation should still collect softer/qualitative evidence of their work and their impact and present these alongside quantitative/return on investment evidence. There’s a risk with CBA that it creates perverse incentives – e.g. don’t work with those groups of people or people who live in that neighbourhood because they provide a lower return on investment to our funder.
Is your organisation able to undertake its own Cost Benefit Analysis or does it need to look at commissioning one? What can you do yourselves internally and what external expertise might be needed?
Review own evaluation and record keeping methods. Consider all points from ‘understanding of client base’ through to ‘knowledge of past approaches’. How far along this pathway are you?
Check out the Treasury’s Magenta book. Does it offer any additional guidance on models/processes that night be useful for collecting data in your organisation?
Know what commissioners are looking for – a CBA might not be needed depending on the service outcomes you deliver. If it is an option – focus on services that have a history rather than newer initiatives as there will be more data to provide a more accurate picture.