How was the REF for you?

I had very little to do with REF preparation this time around, being more absorbed with the (very high, by the way) quality of education across our humanities units. But I’m in it today! Exeter English managed to increase its grade-point average in REF2014, as compared with RAE2008, and yet slip a full 25 places in the table on this measure. If we adjust that for levels of intensivity (i.e. the percentage of people submitted) we rise to seventeenth. If we use the Times Higher’s ‘research power’ measure, which is the best predictor of QR dividends, we rise to fourth; on the Research Fortnight power rating, we’re seventh. So we’ll do ok on the financial side. But still, what looks to all the world like a fall, feeds into most league tables as a fall, and feels for all the world like a fall, just might well be a fall. We made an awful fuss when we did better than expected in 2008; this result leaves us with different stories to tell.

So what lessons might we learn from this REF? Here are some of my initial thoughts.

  • Grade inflation is the big story. And don’t we know it? My first view of our figures left me with a warm glow – but only until I saw the tables. There are six English departments with grade-point averages over 3.4, which is almost unbelievable. I certainly didn’t see this coming.
  • Don’t do the twos. Don’t believe anything you hear to the contrary: two-star work is counter-productive; one-star is poison. We’ve been very tolerant of two-star work, along the lines that everyone needs to do some of that sort of thing some of the time. But the real winners from this REF had ruthlessly minimized their sub-threes. Warwick English had 95% of work at three-star and four-star. Enough said.
  • Impact is a good news story for the humanities. We were worried about this, weren’t we? But across the country, the English panel graded 83% of impact case-studies at three-star or four-star. So did History. Classics came in even higher, at 88%. So is that settled, or have these grading practices created a credibility gap? I predict some lively debate about this.
  • But did impact really have any impact on results? Actually, impact didn’t have the effect that some people expected. There are some interesting winners and losers on impact in the English list: Newcastle, Swansea, Kingston and Bedfordshire at the top (putting to bed, in the process, the hypothesis that impact would favour the metropolitan universities); Cambridge and Bristol quite a long way down. But I don’t think anyone could claim that it’s had a hugely distorting effect. And for us, impact wasn’t the problem: we had 100% of impact at three-star or four-star. In terms of commitment of resources as we move forward, might there even be a risk of over-egging this pudding?
  • A youth-first policy has its risks. Some of our rivals have a reputation for buying fully-fledged stars, at cost. We’ve done the opposite: we’ve bulked up (only Oxford and Cambridge entered more staff than us), but done so almost exclusively from the bottom up. That has always seemed to me an ethical approach, since we’re developing young careers. To be honest, also, we haven’t had an awful lot of choice, since we have needed bums on office-chairs to deal with the bums on the lecture-theatre benches. Other places haven’t had quite the same pressures, and have grown in different ways. At Queen Mary, one of the big winners, roughly 38% members of academic staff in English are professors – a figure significantly higher than the equivalent at Exeter. So maybe it’s time to admit that our approach, for all the efficiency of our wage-bill and all the excitement of developing young researchers, has its limitations. I’m profoundly sorry about that.
  • The jury’s still out on creative writing. Warwick has invested in creative writing, and comes out on top of the table. But with 85% intensivity, we can’t absolutely say that creative writing is now (against popular opinion within our discipline) a REF winner. Ditto UEA: bloody good result, but obtained at a mere 63% intensivity.
  • The AHRC really is prepared to plough a different furrow. Some of the big losers in the doctoral training partnership competition (e.g. Warwick, Queen Mary – Swansea, even) have been big winners in the REF. And vice versa. This debate has legs.
  • Bedfordshire: good on ’em. They will get some headlines, outperforming Cambridge. But their intensivity rate is respectable, at 82%; they just shine out on this table as a very strong, albeit very small unit.
  • How big is beautiful? Setting aside some anomalies – Oxbridge at one end, Bedfordshire at the other end – a well-formed English department looks from these results like one with 30-45 people. A number of the bigger ones (e.g. King’s, Birmingham, Leeds, Sheffield) are notable for having been tough on exclusion. There may be some lessons there.
  • What happened to double-weighting? It’s impossible to tell from the data released to date, but this is a very interesting question. The most likely explanation for our blow-out (against internal predictions) in two-star grades is that we were overly optimistic about double-weighting applications. If some of these were rejected, some weaker work would inevitably have been pulled into the process. There will be calls for clearer, and earlier, guidance on this next time.
  • How useful was the internal monitoring? We all did it, didn’t we? The internal grading, the external ‘experts’, the ‘mock-REFs’. And the cost, in terms of time and money, was huge. Somewhere – let’s say a university somewhat east of Exeter – even paid me for my opinions. But an awful lot of questions will be asked, in an awful lot of departments, in retrospect. Some departments might even be asking external advisors for their money back.

There is, doubtless, much more to be said, but that’ll do for a start.


4 thoughts on “How was the REF for you?

  1. To me it seems that double weighted stuff was included but suffered as a result. This is a big problem for English as it’ll undoubtedly incentivise journal articles over books – the latter far more likely to get 4* but taking massive amounts more energy and at least 10x the work compared to a journal article.

  2. Woah – was I wrong about double weighting?? In fact, the English panel accepted a whopping 99.8% of requests for double-weighting. Makes you wonder about the other 0.2%, doesn’t it?

  3. On Bedfordshire – I’m really glad to see a post-92 so high.

    But there are caveats – having looked at their Dept staff list, and with due respect to some really outstanding people who do ‘pure’ Eng Lit, it seems they have benefitted from non-subject specialism, if that makes sense. One English lecturer is as much Art History as English, and a lot of them are similar crossovers, most notably with regards to sports science and dance, as well as education. Surely the panel who were assessing the English UoA have limits in terms of their expertise that might not match these, though they’re clearly submittable therein? Maybe a lesson there is that it pays to embrace the porosity of boundaries between UoA’s?

    Also, since I teach at a post-92, it’s depressing that size still matters so much. 30 is more than twice the size of eligible people in my Dept, and that includes non-research-active types.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s