The second annual progress report from Ecorys - the research agency undertaking the  meta-evaluation of Creative People and Places - has just been published. 

As before, it is a lengthy read, with 77 pages in the full report including appendices, but it is well worth looking through – particularly if you focus on the areas most relevant to your interests.

It contains a wealth of data and is well-structured. The Executive Summary clearly outlines the key findings, and there is more in depth explanation on all areas. As an evaluation geek I certainly gained a lot from the comparison of outcomes from local evaluations, one of the benefits of a meta-evaluation.

What is a meta-evaluation and what is the point of doing one?

There were two key choices made in designing the national evaluation of CPP: the decision to use a meta-evaluation approach, and the decision to appoint from outside the arts research sector. These are valid decisions to make as there has been a tendency in arts research to talk to ourselves, in our own language, while decisions are made by others in settings where this language doesn't carry much weight. In my view, reflective and context-specific action learning approaches to evaluation, are one of the  best ways of supporting change in practice. However these can be seen as too involved and 'subjective' by those outside the sector.

The structured meta-evaluation approach taken by Ecorys for the core part of the work – the collation of quarterly monitoring and the secondary analysis of data from the local evaluations – is a good way of overcoming this. Meta-evaluation is basically “evaluation of evaluation”. You do a structured overview of the local evaluation data, and then evaluate this both in terms of outputs and outcomes identified, but you also look at to  what extent different data can be seen to be comparable and to what extent it should be treated as accurate. This is a neat solution to the challenge of an action learning programme that also needs a final assessment of whether or not it is working.

However, all the findings do need to be presented within the context in which they occur. In this second annual progress report, Ecorys shows a much stronger grasp of the sector and challenges – beginning to be able to identify what is happening that is interesting, rather than simply what is happening.

What is happening?

Two years of the meta-evaluation, and three years of the Creative People and Places programme, find some key outcomes emerging:

• More people are attending the arts – CPP reached its millionth ‘engagement’ earlier this year, and the majority of these engagements are from those least likely to attend.
• There is increased capacity and capability in arts provision and locally an increased understanding of the arts and the confidence to make informed choices.
• There is increased excellence and innovation in the arts (including understanding what works well and less well) and excellence in engaging and empowering communities.
• In some places there is increased revenue generation.

Lessons to be learnt are mainly around how much time is needed to do excellent work: building delivery partnerships and engaging local people in the commissioning and delivery process have been very effective elements of many CPP Places approaches, but take longer than was expected.

Future learning is expected over the effect of place specific differences and why there's no one CPP model that fits all. Ecorys points out that this could be where some of the most interesting learning emerges from, and this is something a good meta-evaluation can definitely help with.

So is it any good?
The meta-evaluation now feels fully underway, and is beginning to bring interesting things to the fore. By its very nature it is dense – bringing together an amazing amount of data from different sources – but there is a path through this and some clear conclusions are emerging.

CPP is doing new and valuable work in bringing new audiences into the arts; in developing practice and raising ambition to provide excellent arts engagement; and in providing a space for reflection and development of learning around what works. 

Ruth Melville is a researcher in cultural policy and a 'critical friend' for two CPP places; Transported and Market Place.

www.ruthmelville.net

You can read the Ecorys Year 2 Report: Progress and Outcomes here