UKOLN organised a series of workshops on Evidence, Impact and metrics during 2010/11. A report on the final workshop, was written by Kirsty Pitkin, is given below. The report is available at <http://ukwebfocus.wordpress.com/2011/07/18/event-report-metrics-and-social-web-services-workshop/>.

Introduction

In introducing the final workshop event, Brian Kelly, UKOLN, emphasised that the aims were to explore ways of gathering evidence that can demonstrate the impact of services and to devise appropriate metrics to support the needs of the higher and further eduction sector.

Many people argue that you cannot reduce education to mere numbers, as it is really about the quality of the experience. However, Kelly argued that numbers do matter, citing the recent JISC-funded Impact Report, which found that the public and the media are influenced by metrics. As we have to engage with this wider community, metrics are going to become more relevant.

Why Impact, ROI and Marketing are No Longer Dirty Words

Amber Thomas, JISC, mapped out the current landscape, drawing on her own experiences and those of colleagues working in other areas at JISC. She observed a dominant culture of resistance to measurement within education for a number of reasons, including the concern that caring about metrics will mean that only highly cited people or resources will be valued. She noted that the search for an effective impact model is taking place on shifting sands, as issues associated with the value, ownership and control of media channels are being contested, as is the fundamental role of the university within British society.

In discussing impact, Thomas noted that it would be tempting to use the language of markets – with education as a “product” – but stressed that this not how we see ourselves in the education sector. One of the challenges we face is how to represent the accepted narrative of the sector as a nurturer and broker of knowledge, through the use of metrics.

Thomas went on to describe some of the dirty words in this space and the measurements that are associated with them. However, she noted that these measurements can be used for good, as they can help to instigate change. To support this, she provided a model for the role of metrics in decision making, with metrics being one form of evidence, and evidence being only one form of influence on the decision maker.

She concluded by outlining our options for responding to the impact debate: we could deny the impact agenda is important, or we could deepen our understanding and improve our metrics so they work for us and are fit for purpose. The possible directions we could take include developing business intelligence approaches, improving data visualisation techniques and looking for better tools to give us deeper understanding of the metrics. She also stressed that we need to look more closely at the use and expectations of social media in the commercial sector, as we might find we are expecting too much of ourselves.

“I don’t think we can ignore the debate on impact and metrics… what we need to do is engage with the impact debate and use the sort of language that is expected of us to defend the values of the sector a we wish to defend them.”

Surveying our Landscape from Top to Bottom

Brian Kelly provided an overview of the surveys he has been carrying out using a variety of analytics tools.

He began with a personal view: discussing the picture of his own Twitter usage provided by the Tweetstats tool, and how this differs from his own memory. He noted that the data did not always correspond with other evidence, emphasising that we cannot always trust the data associated with such tools.

“You need to be a bit skeptical when looking at this data… you can’t always trust all the data that you have.”

From an institutional perspective, he asked: “What can commercial analytics tools tell us about institutional use of Twitter?” He compared the Klout scores of Oxford and Cambridge Universities’ Twitter accounts, showing how visualisations of the numbers can give a much better understanding of what those numbers really mean than the numbers themselves do in isolation.

He continued in this vein by demonstrating Peer Index, which he used to analyse participants of the workshop. He noted that the top seven people are all people he knows and has had a drink with, so asked whether this shows that the gathering is really a self-referential circle? Kelly also noted how easy it can be to gain extra points and questioned whether it is ethical to boost your score in this way. However, he observed that research funding is determined by flawed metrics and gaming the system is nothing new. So will universities head hunt researchers with valuable social media scores?

Next he looked at Slideshare statistics, using a presentation by Steve Wheeler as a case study. Wheeler made a presentation to 15 people, but his slides were viewed by over 15,000 people on Slideshare. Kelly asked us to consider the relationship between the number of views and the value of this resource. He also examined statistics from the collection of IWMW slides, observing that the commercial speakers had higher view rates, and that the most popular slides were not in corporate look and feel. This evidence could be used to challenge standard marketing perspectives.

Finally, Kelly compared Technorati and Wikio results to demonstrate that four people in the room were in the top 67 English language technology blogs. He pondered whether they should they share their success strategies, or how we could tell the story of this data in different ways.

To conclude, Brian emphasised that he believes this kind of analysis can inform decision making, so it is important to gather the data. However, the data can be flawed, so it is important to question it thoroughly.

Learning From Institutional Approaches

Ranjit Sidhu, Statistics into Decison, focussed primarily on the role of pound signs in communicating particular messages and connecting social media metrics to reality in a powerful way.

He began by observing that the data is often vague. The analytics institutions receive look exactly the same as the analytics used by commercial organisations, despite the fact that their needs and objectives differ widely. He attributed this to the dominance of the technology, which has taken control over the information that gets delivered, thus ensuring everyone gets data that is easy to deliver, rather than data that is meaningful to them. Sidhu also observed that universities often fail to break down their data into relevant slices, instead viewing it at such a high level that it cannot usefully be interpreted in financial terms.

In a self-confessed rant, Sidhu emphasised that you have a chance to tell the narrative of your data. Most social media data is openly available, so if you don’t, someone else will and you will no longer have control over that narrative.

“You need to be proactive with your data. If you’re proactive, people don’t sack you.”

Sidhu went on to demonstrate the type of analytics dashboard he creates for universities, discussing the importance design as well as the analysis itself. His dashboard features nine groups of data and only three key themes, which fit onto one A4 sheet and are arranged in an attractive way. He also discussed his methodology when creating these dashboards, which involves finding out what people want to know first, then finding the data to match those requirements. This is the reverse of common practice, where people take the data that is readily available and try to fit that to their requirements.

He explained the need to match up offline experience with online experience to help to generate projections and quantify the savings produced by online tools and social media. He exemplified this by talking us through one of the most powerful statistics he creates: a calculation demonstrating the amount saved by online downloads of prospectuses compared to sending printed versions. This is usually around £500 per month. This takes the online data, combines it with existing data from the comparable offline process and creates a tangible value.

He extended this to show other types of story we could tell with such data, including the potential value of a website visit from a specific country. Once you have this, you can more effectively demonstrate the monetary value of social media by using referrer strings to show how a visitor from that country reached your site, and therefore make better decisions about how you attract those visitors.

“You have to justify your spend. Your justification has to be based on what you are trying to do at that particular time.”

Identity, Scholarship and Metrics

Martin Weller, Open University, posed many questions and points to ponder, focussing on how academic identity is changing now we are online.

He observed that identity is now distributed across different tools, with a greater tendency to intersect with the personal. There are more layers to consider: where once you had your discipline norms and your institutional norms, now there are more social media norms to observe to create cultural stickiness. You end up with a set of alternative representations of yourself, so your business card is now a much messier thing.

Weller went on to define impact as a change in behaviour, but emphasised that telling the story of impact online is actually very difficult. Your impact may be more about long term presence than an individual post. The metrics we currently use do not necessarily correspond to our traditional notions of academic impact: after all, what do views mean? What do links mean? What do embeds mean? How do they compare to citations?

He put forward the accepted view that blogging and tweeting provide you with an online identity, which drives attention to more traditional outputs. He placed this in the context of a digital academic footprint, which helps tell the story of the impact you are having within your community. Whilst metrics can be useful for this, he warned that they could also be dangerous, with official recognition leading to a gameable system.

He concluded by illustrating a sandwich model explaining why metrics will be increasingly important to what academics do: with top-down pressure from above to demonstrate impact when applying for funding, and bottom-up pressure from individuals asking why their impact via social media doesn’t count. Once you’ve got those two pressures, you have an inevitable situation.

Impact of Open Media at the OU

Andrew Law, Open University, discussed the activities of the Open University when monitoring the various media channels used to disseminate content and how these metrics have led to real, significant funding decisions.

He observed that several of their online media channels did not necessarily have a very clear strategic remit. However, they found that the data was increasingly asking the question: “What is the purpose of all this activity?” Deeper analysis of this data led to the development of clearer stategies for these channels, based on their core institutional aims.

Law emphasised the importance of having all of the information about the different channels in one place to help dispel the myths that can grow up around particular tools. He used the example of iTunes U, which gets huge amounts of internal PR on campus, whilst channels like OpenLearn and YouTube sit very quietly in the background. However, the reality is very different and he observed that one of the challenges they face is ensuring that the broad story about the performance of all of these channels is well understood by the main stakeholders.

Law expanded on this, noting that whilst the iTunes U download statistics provide a positive story, it does not actually perform well against their KPIs compared to other channels, despite little or no investment in those other channels. He observed that their pedagogical approach to iTunes U – which includes offering multiple, small downloads, with transcripts and audio downloaded separately – can inflate the numbers. He compared this to their YouTube channel, which has received very little investment, but is performing very effectively. He also discussed the OpenLearn story, which has been quietly outstripping other channels against their KPIs – particularly in terms of conversions, because it has a lot of discoverable content. He emphasised that this is a very positive story for the university, which needs to be told and built upon.

By demonstrating these realities, the data has demanded of management a much clearer sense of purpose and strategy. This has led to real investment. The OU has massively increased the amount of money spent on YouTube and OpenLearn, representing a significant change in strategy.

In conclusion, Law did note that, so far, the data has only helped the university, not the end user, so their next steps include mapping journeys between these channels to identify the traffic blockages and better tune the service delivered across the board.

The Script Kiddie’s Perspective

Tony Hirst, Open University, provided a set of observations and reflections, which ranged from ethical issues about the use of statistics through to practical demonstrations of visualised data.

He began by observing that social media are co-opting channels that were private and making them public, so there is nothing inherently new going on. He quoted Goodhart’s Law, emphasising that, whilst measuring things can be good, once measures are adopted as targets they distort what you are measuring and create systems open to corruption.

Hirst went on to discuss the perils of summary statistics and sampling bias. He emphasised that the way you frame your expectations about the data and the information that can be lost in the processing of that data are both vital considerations if you are to accurately tell the story of that data.

Hirst discussed the role of citations as a traditional measure of scholarly impact and the ways your content can be discovered, and thereby influence through citation. He highlighted three layers of discovery: the media layer, the social layer and the search engine layer, each of which enables your material to be discovered and therefore influence behaviour. He noted that if links come through to your own domain, you can already track how they are reaching your content. What is difficult to track is when there is lots of social media activity, but none of it is coming back to your domain.

Hirst demonstrated some approaches to tracking this type of activity, including the Open University’s Course Profiles Facebook app; Google search results, which are including more personalisation; and social media statistics gleaned through APIs, many of which can be accessed via an authentication route using OAuth.

Hirst concluded by discussing some visualisations of Twitter communities to show how these can provide insight into external perspectives and how we are defined by others in our community.

Conclusions

The workshop brought forward a number of concerns, that were often less about the tools and technologies involved, but more about the ethics and pitfalls of formalising the measurement of social media activity. The main concern seemed to be the potential for creating a gameable system, or metrics do not reflect reality in a useful way. Ensuring that the metrics we use are fit for purpose will not be an easy challenge, but the discussions held within this workshop helped to identify some potential routes to improving the value and integrity of social media data.