This document gives a background to work on identifying best practices for gathering evidence of the impact of networked services and how the development of metrics can by valuable in ensuring there is consistent approach developed across the sector.

There is a need for publicly-funded organisations, such as higher education institutions, to provide evidence of the value of the services they provide. Such accountability has always been required, but at a time of economic concerns the need to gather, analyse and publicise evidence of such value is even more pressing.

Unlike commercial organisations it is not normally possible to make use of financial evidence (e.g. profits, turnover, etc) in public sector organisations. There is therefore a need to develop other approaches which can support evidence-based accounts of the value of our services.

UKOLN’s Evidence, Impact, Metrics work developed a methodology which can be used within the sector to demonstrate the value and impact of a variety of online services. Regular blog posts have been published on the UK Web Focus blog [1].

Evidence can take the form of anecdotes and case studies. However, whilst such qualitative approaches are valuable approach, there is also a need to complement such approaches with quantitative measures.

Institutions will need to have processes in place to measure, for example, usage of their services for capacity planning purposes (will new hardware or bandwidth be required to cope with growth) and to identify both successful services and those for which usage may be dropping in order to decide on  the best ways of allocating scarce resources.

Carrying out such measurements over time can help to identify trends, which can help in long term planning. The analysis of such trends across a sector, such as higher education institutions, can be valuable in helping to detect ‘weak signals’ which may not be easily spotted from trends within an individual institution.

As an example the accompanying image compares the number of ‘fans’ for popular UK University Facebook pages in 2008 and 2010 [2]. This survey work began in 2007 following suggestions that “Something is going on with Facebook” [3] which led to capturing evidence of initial usage patterns in order to benchmark future trends.

Institutions should benefit from knowledge of early indications of such trends as this can help to inform policy decisions within the institution.

Note that snapshots of use of iTunes U [4], YouTube Edu [5] and institutional use of Twitter [6] have been published recently in order to provide a benchmark and to help inform institutional policy-making decisions on use of such services.

There are dangers that such evidence-gathering approaches will fail to reflect the diversity to be found across UK HEIs, which are very different in size, levels of funding, institutional priorities, organisation culture, etc. Although such diversity should not be of a concern in depicting trends across the sector there is a danger that numerical evidence of institutional activities will be reduced to league tables.

It would be dangerous if such concerns resulted in a failure to provide such evidence; there is an expectation that universities should be transparent in how their funding is being used and in any case, FOI requests are being used to obtain such evidence [7].

UKOLN work across the sector seeks to gather feedback on best practices for collecting and using evidence which will ensure that benefits of such approaches are gained whilst associated risks are minimised.

1. Evidence category, UK Web Focus Blog, <http://ukwebfocus/wordpress.com/category/evidence/>
2. Planet Facebook Becomes Less of a Walled Garden, UK Web Focus blog, 8 Oct 2010, <http://ukwebfocus.wordpress.com/2010/10/08/planet-facebook-becomes-less-of-a-walled-garden/>
3. Something IS going on with Facebook, UK Web Focus blog, 29 May 2007, <http://ukwebfocus.wordpress.com/2007/05/29/something-is-going-on-with-facebook/>
4. What Are UK Universities Doing With iTunesU?, UK Web Focus blog, 11 Oct 2010,  <http://ukwebfocus.wordpress.com/2010/10/11/what-are-uk-universities-doing-with-itunesu/>
5. How is the UK HE Sector Using YouTube?, UK Web Focus blog, 18 Oct 2010, <http://ukwebfocus.wordpress.com/2010/10/18/how-is-the-uk-he-sector-using-youtube/>
6.  Social Analytics for Russell Group University Twitter Accounts, UK Web Focus blog, 6 Jun 2011, <http://ukwebfocus.wordpress.com/2011/06/28/social-analytics-for-russell-group-university-twitter-accounts/>
7. University Web Sites Cost Money!, UK Web Focus blog, 16 Nov 2010, <http://ukwebfocus.wordpress.com/2010/11/16/university-web-sites-cost-money/>