What’s the difference between ‘game-playing’ and ‘strategizing’? More on the REF.

What’s the difference between ‘game-playing’ and ‘strategizing’? My sense, when it comes to the REF, is that strategizing is what one’s own university does, and game-playing is what other universities do. Especially if those other universities did rather well.

But whatever we call it, success in REF2014 depended to an uncomfortable degree on decisions made by research managers, as opposed to work done by researchers. Here, in my third of too many blog-posts on the REF (others here and here), I want to look at three aspects of gaming. Maybe this is just a way of yoking together a few random things I haven’t said in my previous blogs. Maybe, though, there’s a serious point lurking underneath: that the REF is at its least credible when it looks most like a game.

1. Double-weighting

Double-weighting is one of the great under-told stories of the REF, having a significant influence in the subjects I know best. This was the first time that universities were responsible for requesting double-weighting, so nobody really knew how high the bar would be set. Some places assumed the standard would be high, and so submitted relatively few requests. Others (including Exeter) took the more logical approach: since there was nothing to lose, why not propose anything that looked substantial (including just about every monograph)?

And what happened? The English panel received 506 requests, and turned down a mighty total of one. Indeed perhaps the great question of REF2014, for me, is: what the hell was that one? I really hope it wasn’t my monograph.

As a result, though, places that were conservative in their requests are now counting the cost. The English panel’s report makes interesting reading:

In a significant number of cases, double-weighting was not requested for outputs where this would clearly have been appropriate. In other cases, the option was used inconsistently, or very sparingly. Some institutions chose not to request double-weighting for any of their outputs. This had an appreciable effect on the outputs sub-profile of a number of institutions.

As I read those words, they’re not happy. The REF, on this matter, became too game-like. Next time, to state the blindingly obvious, the panels surely need to be given more authority to make double-weighting judgements themselves.

2. The impact case-study

There’s been a lot of analysis of the impact of impact, and there’s bound to be a lot more. One study by Seb Oliver, reported in last week’s THE cover-story, demonstrates the disproportionate effect that impact had on overall results. As the THE summarizes the argument: ‘because the scores for impact (and, indeed, environment) typically show a wider variation than for output, they in effect count for more than their nominal weighting in determining the overall scores’. For the English panel, Oliver estimates, impact thus effectively amounted for 28.6% of the overall result.

But the bigger revelation in the THE piece, for me, was quite how many universities employed consultants to write their case-studies. Now, all the precise prose in the world won’t manufacture impact, but one wonders how many places under-sold themselves on account of trying to muddle through the process without the best advice. How about Newcastle? One of my favourite REF statistics is that Newcastle’s English submission achieved 100% four-star for its impact case-studies, whereas Newcastle’s History submission managed 100% of impact at two-star. Might Newcastle History have been able to sell itself a bit better if, say, it had taken advice from colleagues in English?

The goal is surely that decisions on impact are driven as much as possible by objectively verifiable data rather than professional prose. The reality is perhaps that next time we will all rely on consultants, whether internally employed or externally contracted, thereby driving up the overall administrative cost of the REF and handing more ammunition to its critics.

3. Environment

And will we also be employing consultants to write our environment statements next time?

One unanswered question from REF2014 is: to what extent were the panels’ assessments of environment dependent on metrics (grant income, PGRs), and to what extent were they dependent upon the (REF 5) templates? I think we were all expecting to be able to trace correlations between the metrics and the overall scores; however, in initial reviews these patterns are far from obvious.

The logical conclusion is that departments, at least to some degree, have been rewarded on account of directors of research presenting information in a convincing manner. Others may have suffered: not necessarily because there were no positive things to be said, but because the authors of the statements didn’t quite get what was important. The effects may have been marginal, but I’m left wondering whether we might reasonably expect something more transparent next time: maybe something with clearer, more explicit directions, making absolutely clear to those preparing the statements the criteria on which they would be judged.

—————————————————–

So what does this all mean? Time after time, looking through the results, we might find ourselves saying: ‘Woah, X screwed up on Y?’ And the person who led the process at X might, of course, be feeling rather queasy. I guess my point is that those responses are not only tough on individuals and their departments – generally good people making honest efforts – but also on the process itself. Ideally, it should look rather less like a game.