FAQ

What are metrics?

“A performance metric is a measure of an organization’s activities and performance. Performance metrics should support a range of stakeholder needs from customers, shareholders to employees. While traditionally many metrics are financed based, inwardly focusing on the performance of the organization, metrics may also focus on the performance against customer requirements and value. In project management, performance metrics are used to assess the health of the project and consist of the measuring of six criteria: time, cost, resources, scope, quality, and actions.” [1]

Which should I bother with metrics?

Metrics can provide quantitative evidence of the value of aspects of  project work. Metrics which indicate the success of a project can be useful in promoting the value of the work. Metrics can also be useful in helping to identify failures and limitations which may help to inform decisions on continued work in the area addressed by the metrics.

What are the benefits for funders?

In addition to providing supporting evidence of the benefits of successful projects funders can also benefit by obtaining quantitative evidence from a range of projects which can be used to help identify emerging patterns of usage.

What are the benefits for projects?

Metrics can inform project development work by helping to identify deviations from expected behaviours of usage patterns and inform decision-making processes.

What are the risks in using metrics?

Metrics only give a partial understand and need to be interpreted careful. Metrics could lead to the publication of league tables, with risks that projects seek to maximise their metrics rather than treating metrics as a proxy indicator of value.

What are the resource implications of using metrics?

There are dangers that gathering, interpreting and visualising metrics can be resource-intensive. Note that increasing numbers of social media services, such as Slideshare, provide hosting services for free but have a licensed service which provides additional functionality included access to detailed usage statistics.

How can projects maximise the benefits of metrics whilst minimising the risks?

Projects may benefit from open publication of metrics and their interpretations of the metrics which can help encourage discussions. Note that the risks of using metrics should be counter-balanced by assesing the risks of failing to use metrics.

Which sources of metrics are valuable?  Which are not and why?

Examples of sources of metrics for online services include usage, feedback and links. Usage statistics can provide evidence of use of a service, channel or deliverable; feedback statistics can provide evidence of interest as can statistics on links to a service. Also note the value of comparisons with peers and trend analyses, rather than using figures in isolation.

However metrics need to be carefully interpreted. Note that if benchmark figures are valued too highly projects may be tempted to ‘game the system’. Remember that high usage statistics may not be relevant for projects which have a niche audience.

What is the status of use of metrics in institutions at the moment?

In some areas use of metrics are well-established. This includes metrics collated for the Sunday Times and Time Higher Education annual University guides. Metrics are also collated by SCONUL for evidence of library usage and underpin citation analyses which are used to help identify the value of research papers.

Are there licensing or political issues to be aware of?

It can be helpful to publish metrics with a Creative Commons licence (possibly CC-0) to enable others to freely reuse your data.  In addition government moves towards greater openness and transparency for public sector services will encourage increased moves towards openness.

What additional information sources are available?

The Oxford Internet Institute (OII) published a JISC-funded report on “Splashes and Ripples: Synthesizing the Evidence on the Impacts of Digital Resources” [2]. The OII have also published a “Toolkit for the Impact of Digitised Scholarly Resources” [3].

What should JISC projects and services be doing to keep up with good practice?

Blog posts about evidence and metrics are published under the Evidence category on the UK Web Focus blog [4]. Comments and feedback on the posts are welcomed, especially from those involved in JISC-funded activities.

References

1. Performance Metric, Wikipedia, <http://en.wikipedia.org/wiki/Performance_metric>
2. Splashes and Ripples: Synthesizing the Evidence on the Impacts of Digital Resources, Eric Meyers, Oxford Internet Institute, <http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1846535>
3. Toolkit for the Impact of Digitised Scholarly Resources (TIDSR), Oxford Internet Institute, <http://microsites.oii.ox.ac.uk/tidsr/>
4. Evidence category, UK Web Focus blog, <http://ukwebfocus.wordpress.com/category/evidence/>