REF-cycling

One of the characteristics of the REF/RAE is that we always spend two or three years playing a game with no goalposts. It is now over two years since the deadline for REF 2014 submissions. It will be close to three years by the time we know what form REF 2020-something will take.

This hiatus poses a question: to what extent should we plan when we don’t know the rules? Responses might be positioned on a spectrum. I’ve heard of departments where academics are told to keep calm and carry on doing good research. There’s a simple decency to that. And there are others where plans, drafts and publications are being reviewed, monitored and graded, more assiduously than ever.

My university is somewhere towards the latter end of the spectrum. That’s just the way we are. And I feel myself, at the level of a department, that it would be irresponsible not to ensure that colleagues are carefully assessing the possible consequences of decisions about research plans. Research time is precious – and REF contributions, let’s face it, have a huge bearing on career progression in the UK – so these decisions will matter.

That’s why we’ve been spending some time recently, in my department, listening to REF 2014 panellists and thinking about our planning. I’ll outline here – in a post that is admittedly humanities-heavy and UK-centric – some of what’s been on our minds.

 

Originality, significance, rigour

The REF criteria are simple enough, attuned with what most of us value about academic work. (For some helpful expansion, see the Panel D criteria, p. 88.) Originality, significance, originality: and ‘significance’, it’s worth noting, should not be confused with ‘influential’, since a publication may be significant, in the eyes of a REF panel, even in a small area and with a small readership. (That’s one way metrics could change things.)

In my department, though, one of the lessons we’ve been learning is that we’ve relied too much in the past on proxies of quality. Most of all, we’ve been overly swayed by length. In our REF 2014 planning, we made easy assumptions that monographs would be graded four-star. Now we’re accepting that there were plenty of three-star monographs submitted to the English panel in 2014, and even some two-star ones. Meanwhile, there were plenty of articles graded at four-star: something we were equally reluctant to predict. (Equally, plenty of editions, collections of essays, creative works: there are some myths about what ‘matters’ in the REF that are well worth exploding.)

What lessons might we draw from this? Firstly, it seems to me there are risks of over-reacting, or of seeing the positive message (‘my articles could be four-star’) and missing the negatives (‘my monographs might not be’). Secondly, there are some really interesting mentoring challenges, because if I have colleagues devoting a lot of time to writing monographs, I want to do all I can to help them ensure those monographs are as good as they can be. The alternative might be deciding, in a future mock-REF process, that some monographs don’t cut it. And that would be a tough, potentially divisive message.

Thirdly, it perhaps changes the mind-set, quite common in the humanities, that goes along the lines: ‘focus on the monograph and sort out some other items along the way’. If an exceptional article may rate better than an unexceptional monograph, we need to be thinking very carefully about all our items.

Fourthly – and this isn’t strictly relevant, but I like to say it whenever I can – we need to think about what not to do. There’s no point showing up for a REF with a pile of two-star publications and a case of work-related stress.

 

Game-playing or dodgy rules?

There was a lot of talk about ‘game-playing’ after REF 2014, much of which was sour-grapes silliness. But it’s becoming pretty clear that some of the rules created the effect of game-playing, regardless of the players’ intentions.

Exhibit A is ‘double-weighting’. Asking units of assessment to decide whether or not to request double-weighting for an item distorted the playing field, hurting those who interpreted the requirements more conservatively than others. The solution next time must be to give panels more authority to determine what is in the best interests of the submission.

Then there’s ‘repetition’: the submission of work that had previously been published in another form. We were appalled to be criticized for this, since we had been very careful; however, it now seems rather a lot of English departments received the same feedback. The application of the rules, let’s say, had a punctilious vigour. I’d hope we can do better next time – but in the meantime, it pays to know.

 

The impact industry

Allow me to quote again my favourite sentence from the Green Paper: ‘We must also address the “industries” that some institutions create around the REF and the people who promote and encourage these behaviours.’ And hence to impact.

The impact industry is building. There are impact gurus who will help us maximize our impact, case-study writers to support us as we prepare for a REF, and training programmes to help us understand it all. My impression is that the level of expertise around impact in the humanities is rather less reliable than it is for the sciences and social sciences. Although I’ve made this point before, it therefore remains crucial to nurture and respect experience – knowledge of what works, and what was recognized in 2014 – within our disciplinary communities.

Thinking ethically, one wonders whether all the money devoted to the impact industry might be better spent. But thinking rationally, this was surely bound to happen. I mean, what did they expect? There was indeed a touch of utopianism to that Green Paper sentence.

 

Might the rules all change? I wouldn’t bet on it. I’m predicting a REF 2021 that will look rather like REF 2014: with maybe a few of the creases ironed out, and probably a few more ironed in. But I might be wrong.

Advertisements

2 thoughts on “REF-cycling

    • There once was an English Head,
      Who muttered out “metrics” in bed,
      When crossing the road,
      Or saluting a toad,
      “Metrics, more metrics!”, he said.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s